Benchmarking Discussion Group

1. Benchmarking Discussion Group Telluride Cognitive Neuromorphic Engineering Workshop 2014

2. Major outcomes • We need NE-specific benchmarking to: • Improve the performance of NE systems with apples-to-apples comparison • Convince investors and industry that our systems have high performance • Benchmarks should be multi-variate, i.e. not just measuring accuracy but also any or all of the following: • Biological realism • Necessity for tuning • Power consumption or performance per operation or task • Latency / speed • Noise robustness • Area / technology node / resources • Learning speed • Adaptability • Robustness in real-world problems • Multi-sensory problems • Usability • Existing Benchmarks • There are a number of NE datasets already, such as MNISTDVSSpikes. • There are a number of good datasets from cognate fields e.g. robot tasks, and spike sorting.

3. New Benchmarks • Potential for NE-specific benchmarks encompassing all variables of interest • Potential for benchmarks with predetermine sample statistics, characteristics etc. • Annual (at Telluride) hardware benchmarking with common input (scenes, audio etc.) • Hosting of Datasets • Giacomo is keen to host on INE website, with mirror sites at e.g. other Universities. UWS is available to host immediately. • Data will be available under licence – we suggest Open Data Commons Attribution ODC-BY 1.0. • Datasets should include readme files or ascii headers, and any code necessary to translate e.g. jAER to MATLAB. • Dissemination • Giacomo has suggested the possibility of two papers in FNE - a general overview of the problem, and a specific paper on benchmarking of spatio-temporal pattern recognition systems (both hardware and software). These may form part of a special issue under Michael’s leadership. • Anyone who wants to participate in these papers, please email Jon Tapson, jtapson@gmail.com • Thanks to everyone who participated, esp. Danny who wrote up the meeting notes.

4. Hard Problems in Neuromorphic Engineering • We should come up with a list of Neuromorphic Challenges that we think only neuromorphic engineering can solve • These challenges can raise awareness and drive the field, as the Hilbert Problems did a century ago • There should be an annual meeting to update and evaluate the latest progress • This list may be taken as a definition of Neuromorphic Engineering, so we should be careful how we construct it • Example challenges: • Face recognition within energy and time constraints • Speaker recognition within noise, energy, and time constraints • Adaptive Motor Control • Operant classical conditioning • Limited memory / Limited time response