Instructions Parameters Commands Conditions. Finite State Machine Blocks Summary 20 DA sistemare FSM SM1 SM2 SM16 FSM ODR INT1 1 OUTS1 2 INT2 OUTS2 INT16 OUTS16 FIXED DATA VARIABLE DATA INSTRUCTIONS FIXED DATA VARIABLE DATA INSTRUCTIONS FIXED DATA VARIABLE DATA INSTRUCTIONS 16 LONG COUNTER Machine Learning Core Filters
A Breakthrough Innovation in MEMS Sensors Introducing LSM6DSOX, iNEMO 6DoF Inertial Measurement Unit (IMU), with Machine Learning Core Edoardo Gallizio Sr. Product Marketing Manager Agenda 2 · Artificial Intelligence in IoT · ST Smart Sensor: 6-axis IMU with Machine Learning Core · Why new sensors solutions are attractive? · Solutions and Comparisons · Tools · Conclusions Artificial Intelligence in IoT 3 Logic Programming Decision Tree Machine Learning A.I. Deep Learning Search Algorithms Neural Networks Data Mining Artificial Intelligence in IoT 4 Analysis of vast amount of information Artificial Intelligence in IoT 5 Centralized A.I. Architecture Artificial Intelligence in IoT 6 Edge A.I. Architecture Artificial Intelligence in IoT 7 Edge A.I. Architecture Artificial Intelligence in IoT 8 Edge A.I. Architecture LSM6DSOX Artificial Intelligence in IoT 9 Artificial Intelligence in IoT 10 LSM6DSOX Data processing inside the sensor: + Local processing + Real time analysis + Reduced cost of bandwidth + Ultra Low Power + Intrinsic Security = Configurable Logic - Simple computation (Dec. Tree) Data processing inside the STM32: + Local processing + Real time analysis + Reduced cost of bandwidth = Low Power = Improved security = Configurable Logic = Std Computation (Neural Net.) Data processing inside the Cloud: + Advanced Computation + Availability of Wide Amount of Data + Continuous Algo Improvement = Remote processing - Data Transfer Latency - High cost of bandwidth - Very High current consumption LSM6DSOX Smart Sensor 11 The First IMU Sensor with embedded Machine Learning Core Sensing Full activity monitoring processing Power optimization at system level Sensing & pre- processing High-level processing LSM6DSOX Unique Performance 12 Improved Accuracy, Optimized System Power High Accuracy Noise: Gyro 3.8 mdps/Hz Accelerometer 70 µg/Hz Low Current Consumption 0.55mA HP combo -15% vs. LSM6DSL/M New ultra low power Axl: 14uA @100Hz ODR LSM6DSOX 2.5x3x0.86 mm Sensor HUB & compressed 9kB FIFO Embedded Pedometer 2.x WeChat Compliant OIS Stabilization Core New Standard Protocol I3C / I2C / SPI Finite State Machine & Machine Learning Core From Low Power Sensor 13 to Low Power System Best in class IMU Power Consumption (0.55 mA combo High Perf. Mode, 9uA Accel. only) 10 to 1000 time energy saving by running Machine Learning on Sensor HW (vs. AP) Simplified And Lean System Concept based on 2 mainstreams: 1. Configurable power mode and high speed communication 2. Flexible HW solution running accurate algorithms LSM6DSOX Smart Sensor 14 The first Smart Sensor Embedding: Finite State Machine Start Sensing RESET NEXT RESET NEXT COMMFuAll NacDtivity monitoring PARAMpEroTceEsRsinSg RESET NEXT Machine Learning Core Power optimization at system level Sensing & pre- processing Start node node High-levneol de processing node node node CONT FSM & MLC allows sensors to process data with reduced help of a host MCU Deductive vs. Inductive algorithms 15 Inertial Algorithms Overview DEDUCTIVE INDUCTIVE Hypothesis Observation Rules i.e. Wake Up Finite State Machine Model i.e. Activity Rec Machine Learning Processing Finite State Machines overview 16 LSM6DSO/X - Finite State Machine FSM is composed of a series of states, configurable parameters/resources and variables Possibility to implement up to 16 independent FSM FSM are executed simultaneously or sequentially FSM can use data from accelerometer, gyro or an external sensor FSM outputs: - Interrupt - Source information Finite State Machine structure 17 In LSM6DSOX Sensors Data Data Section Instructions Accelerometer Gyroscope External Sensor MLC Filtered Data Fixed Data Variable Data Conditions Commands Parameters Finite State Machine structure 18 In LSM6DSOX There are 2 Data Sections in each FSM program: · The Fixed Data SeScteionnsoof trhse program is used to specify the resources examples: that are neeDdeadtain the instructions section. Resources · Thresholds · Timers · Masks · Hysteresis · Decimation factor Data Section Fixed Data · The Variable Data Section is used to define all the values for the used resources. Variable Data Finite State Machine structure 19 In LSM6DSOX In the Instruction Section the user defines the algorithm by using available commands Sanedncosnodirtisons: · Commands can be foDlloawtead by parameters. Once executed the program pointer is set to the next command/condition line that is immediately evaluated. · Parameter are application masks, threshold and timers. · Conditions can be: · Reset. If true pointer set to last Reset Point · Next. If true pointer set to Next Line · None interpreter keeps evaluating the same line when next sample occurs Instructions Conditions Commands Parameters Finite State Machine Blocks 20 Summary FSM FSM ODR LONG COUNTER Sensors Data DA sistemare Machine Learning Core Filters INPUT MUX16 INPUT MUX2 INPUT MUX1 SM1 SM2 FIXED DATA VARIABLE DATA INSTRUCTIONS FIXED DATA VARIABLE DATA INSTRUCTIONS INT1 OUTS1 INT2 OUTS2 SM16 FIXED DATA VARIABLE DATA INSTRUCTIONS INT16 OUTS16 Finite State Machine 21 Examples of Libraries Exploiting Finite State Machine capabilities · High Efficiency (Computation Resources, Power) Wrist Tilt · High Customization level · Library of example gestures available: Free-Fall Motion-Stationary Pickup Flip Up/Down Motion-Stationary Wakeup Glance 4D Shake Machine Learning Core Overview 22 LSM6DSOX - Machine Learning Core Start node node node node node node MLC is a programmable logic. LSM6DSOX embeds a Decision Tree Logic composed by a series of configurable nodes. Each node is characterized by one "ifthen-else" condition. Up to 8 decision trees can be configured to run simultaneously in LMS6DSOX. Sense Machine Learning approach 23 Machine Learning Core LSM6DSOX · User defines Classes to be recognized · Collects multiple Logs for each class · Defines Features that better characterized the defined classes Envisioning Self Learning Approach Log continuous acquisition enabling Decision Tree fine tuning · Based on Logs and Features, the Decision Tree can be generated using machine learning tools · Configure the device and run the application Machine Learning Core (MLC) 24 In LSM6DSOX Sensors Data Computation Block Decision Tree Accelerometer Gyroscope External Sensor Filters Digital Pre-defined Features Meta-classifier Node Machine Learning Core (MLC) 25 In LSM6DSOX · Wide set of Inputs available: · Accelerometer [ax ay az ], av , a2v · Gyroscope [gx gy gz ], [gv] , g2v · External sensor [mx my mz ], [mv], m2v (. . ) · Magnitude = X2 + Y2 + Z2 Sensors Data Accelerometer Gyroscope External Sensor Machine Learning Core (MLC) 26 In LSM6DSOX · Sensors data can be filtered with a 2nd order IIR Filter. · Features are statistical parameter calculated from: · Input Data · Filtered Input Data Computation Block · Examples of features are: Mean, Variance, Energy, Peak to Peak, ... Filters Digital Pre-defined Features Machine Learning Core (MLC) 27 In LSM6DSOX · The Decision Tree is a predictive model built from training data. The outputs of the computation blocks are the inputs of the decision tree. · Each Node is characterized by one «if-then-else» condition. Some examples of conditions: · Mean on Acc_X < 0.5 g · Variance on Gyro_Z < 200 dps Decision Tree · Decision tree can either generate a result at every sample or filter the results with a Meta-classifier, to have a more robust output. Meta-classifier Results Input Condition Node True Path False Path Machine Learning Core 28 Examples of Libraries Exploiting Machine Learning Core capabilities · High Efficiency (Computation Resources, Power) Activity Recognition (for Mobile) · High Customization level Activity Recognition · Higher Complexity than FSM (for Wrist) Motion Intensity Gym Activity Recognition · Example Libraries available: Wrist Tilt Motion-Stationary Hand washing detection Head motions recognition Pickup Vibration Monitoring 29 Comparison of Solutions LSM6DSOX Smart Sensor 30 The First IMU Sensor with embedded Machine Learning Core Industrial Robots Personal Electronics Sensors Data Computation Block Decision Tree LSM6DSOX Power Consumption Application Case examples 31 LSM6DSOX Lowest Current Consumption FSM and MLC are incredibly efficient in current consumption needs: + 3 µA for each Finite State Machine + 1-15 µA Machine Learning Core as low power features to recognize customizable conditions and generate interrupts. System Level Benefits using LSMDSOX 32 Activity Recognition in SW Activity Recognition in DSOX DSOX SoC/uC Real Time Data Parameter DSOX current MIPS uC/SoC Traffic Battery usage DSOX SoC/uC Interrupt Benefit in using MLC Similar DSOX consumption Offload SoC / uC Less traffic on Bus Reduction of System current Computational Current Consumption 33 Applications has been developed on Machine Learning Core Algorithm Sensors used ODR Vibration A Monitoring Motion A Intensity 6D position A recognition Activity A Recognition for mobile 26 Hz 12.5 Hz 26 Hz 26 Hz Number of decision trees 1 Number of nodes 2 MLC additional Current consumption 1 uA 1 7 1 uA 1 8 2 uA 1 126 4 uA Application Case 34 Same application developed on low power microcontroller and on Motion Learning Processing Activity Recognition Algorithm Classes Additional Current Consumption Required to Run the Application · Stationary · Walking · Fast Walking · Jogging ... MLP Start node node Running on: Cortex-M3 STM32L152RE@32MHz [µA] 240 MLC on LSM6DSOX 7 System Level Benefits using LSMDSOX 35 Activity Recognition in SW Activity Recognition in DSOX DSOX SoC/uC DSOX SoC/uC Real Time Data Interrupt Same performance on Confusion Matrix on SW Vs. algo running in DSOX Detected as -> Stationary Walking Fast Walking Jogging Stationary 99.1% 0.9% 0.0% 0.0% Walking 0.0% 99.4% 0.2% 0.0% Fast Walking 0.0% 3.7% 95.9% 0.2% Jogging 0.0% 0.6% 0.7% 98.5% 50x less current for same performance 36 Tools ST MEMS Sensor evaluation kit 37 Professional Unico software Tool and ProfiMEMS · High level tool for device configuration, data logging and programming · Support FSM & Machine Learning Core Configuration Tool Sensor Adapter · FSM / Decision Tree outputs visualization and Logging · Different configuration of Power Supply (Vdd & Vdd_IO separately) · Device power consumption measurement (with configurable power mode by SW) STEVAL-MKI196V1 STEVAL-MKI109V3 HW board Device Configuration Runtime verification CONFIGURATION .UCF file Software tool ST Unico GUI software 38 · Unico Lite GUI (or STSW-MEMS034) is an user interface that assists users to graphically and easily: · configure registers, · see the sensor's behavior on a chart, · export settings in a header file to include in the source code of an application. 1 Registers configuration 2 Data Collection 3 Pattern analysis 4 FSM Implementation 5 MLC Configuration FSM examples 39 · ST provides FSM examples to start experimenting and test. Fine Tuning 40 It is possible to debug in real time the FSM behavior playing back previously recorded data. FSM can detect 41 when the user picks up a product · Option 1: Programmers can load the example, write it onto the sensor, and start experimenting with it right away by changing threshold values or timers, for instance. · Option 2: You can replay a series of events to see if the FSM would respond appropriately and throw an interrupt at the right moment. FSM Algorithm GNTH1 Threshold1 Threshold2 START RESET POINT LNTH1 TI4 NOP LNTH2 TI4 LRTH1 - Threshold1 NOP TI3 Timer GRTH1 TI4 CONT XL Z-axis Start Working with ProfiMEMS Board 42 · The best way to start experimenting and prototyping with the LSM6DSO/X is: · STEVAL-MKI196V1 daughter board. · STEVAL-MKI109V3 motherboard. Sensor Adapter STEVAL-MKI196V1 STEVAL-MKI109V3 · Users need to plug the sensor board into the main board and connect it to a PC using the onboard USB port. · Unico v8.0 supports FSM & MLC on Windows, Linux, Mac Build the Decision Tree through ML tools 43 External tools for decision tree generation · The decision tree can be generated by a dedicated Machine Learning tools: WEKA · Attributes selection (from .ARFF file) · Data filtering · Decision tree generation · Decision tree performance evaluation (e.g. number of nodes, accuracy, confusion matrix, etc...) · Or other alternative tools: · RapidMiner Example available in AN5259 rev2 · Matlab Scripts available at https://github.com/STMicroelectronics/STMems_Machine_Learning_Core/tree/master/tools/matlab · Python Scripts available at https://github.com/STMicroelectronics/STMems_Machine_Learning_Core/tree/master/tools/python · LSM6DSOX ST datasheet · LSM6DSOX: LSM6DSOX.html · Application Notes: · AN5272 (LSM6DSOX): DM00571818.pdf · AN5273 (LSM6DSOX Finite State Machine): DM00572971.pdf · AN5259 (LSM6DSOX Machine Learning Core): DM00563460.pdf · Video Tutorial · Getting Started with LSM6DSOX: LSM6DSOX.html · Boards: · Professional MEMS tool board - STEVAL-MKI109V3: mems-motion-sensor-eval-boards/steval-mki109v3.html LSM6DSOX 44 Reference & Support · Unico GUI package v8.0 - SW package for boards · STSW-MKI109W for Windows OS: evaluation-toolsoftware/stsw-mki109w.html · STSW-MKI109L for Linux OS: evaluation-toolsoftware/stsw-mki109l.html · STSW-MKI109M for Mac OS: evaluation-tool-software/stswmki109m.html · MEMS Community · Community: community.st.com/s/group/CollaborationGroup · Q&A: mems-and-sensors · LSM6DSOX Adapter board - STEVAL-MKI197V1: memsmotion-sensor-eval-boards/steval-mki197v1.html Takeaways 45 · LSM6DSOX completely transforms what a sensor can process without help of a host microcontroller (MCU), thanks to its unique Machine Learning Core. · Exceptional advantage in terms of Power consumption and Flexibility. · ST is supporting your projects by refence design, dedicated tools and example libaries. Thank You! 46Microsoft PowerPoint 2016 Microsoft PowerPoint 2016