July 25, 2020

2300 words 11 mins read

Can a piece of drywall be smart? Bringing machine learning to everyday objects with TinyML

Can a piece of drywall be smart? Bringing machine learning to everyday objects with TinyML

Can a piece of drywall be smart? Bringing machine learning to everyday objects with TinyML Kurt Marko Mon, 11/09/2020 - 23:23

So-called smart devices like Amazon Echo and Google Nest made early headway into our homes. But will devices as small as a vibration sensor soon outsmart an Echo? Here’s a look under the hood of “TinyML.”

Since the HAL9000 and Star Trek’s M-5 Multitronic, the power and capabilities of AI have always been oversold by both Hollywood and Silicon Valley. Although we’re still waiting on machines that can carry on an intelligent conversation, AI has been creeping into many objects in our everyday lives behind the scenes, making them more useful and proactive. People are most familiar with the intelligent assistants built into devices like the Amazon Echo, Google Nest Hub and Apple HomePod, but as I wrote more than three years ago, these rely on cloud backend services for most of their smarts, using local hardware primarily to recognize their wake word and listen for follow-up questions.  Soon, devices as small as a vibration sensor will outsmart an Echo due to significant advances in the performance of low-power hardware and more efficient AI algorithms. The combination allows surprisingly sophisticated deep and machine learning models to run on embedded systems. Until recently, shoehorning AI software into a battery-powered device has required data scientists skilled in working with the constraints of an embedded SoC, but recent advances in AI development and automation frameworks, categorically termed TinyML, greatly expands the realm of smart devices. From phones to sensors, AI permeating the environment AI has significantly reshaped and improved everyday objects in ways that few people recognize. For example, most phone users don’t realize that when they press the shutter button to take a snapshot, it unleashes a complicated process causing the camera to rapidly take multiple images using different exposure settings, analyzes them for features and then pixel-by-pixel using embedded deep learning models before combining them into a single picture. Apple calls this feature Deep Fusion, while Google uses similar computational photography techniques for its Night Sight, Astrophotography and HDR+ shooting modes. Here’s what the process looks like when Pixel phones take a low-light shot. Apple’s most recent iPhone 12 Pro and iPad Pro models go even further by combining data from both the camera and LIDAR (laser rangefinder) sensors. The stunning results are often impossible to recreate with a conventional camera and tripod. 

(via Google)

Source: Google Research paper; Handheld Mobile Photography in Very Low Light While sensors and other low-power devices can’t run algorithms of the same sophistication, TinyML and associated development tools promise to give AI smarts to an immense range of battery-powered devices. TinyML is the moniker for both a movement and a developer community. The movement is galvanized by the idea of making ML work on sensors that can be powered by a watch battery or energy harvesting to turn raw data into useful information. As two Google engineers put it in their how-to on TinyML development (emphasis added) This is where the idea of TinyML comes in. Long conversations with colleagues across industry and academia have led to the rough consensus that if you can run a neural network model at an energy cost of below 1 mW, it makes a lot of entirely new applications possible. This might seem like a somewhat arbitrary number, but if you translate it into concrete terms, it means a device running on a coin battery has a lifetime of a year. That results in a product that’s small enough to fit into any environment and able to run for a useful amount of time without any human intervention.

For context, a phone SoC like the Qualcomm Snapdragon 865 uses up to 5W, or about 1000-times the power of some TinyML devices.  Cost is another aspect that differentiates TinyML devices from mobile or ultra-portable processors. For example, the cheapest Raspberry Pi, the Pi Zero, which uses a Broadcom SoC with an older Arm 32-bit core, runs about $5 in volume. The same model with embedded Bluetooth and Wi-Fi is double the price at $10. In contrast, many 32-bit microcontrollers used in embedded systems, like those using the popular Arm Cortex M0+, only cost $1. At that price, the ubiquity of microcontrollers in everyday objects isn’t surprising, with sales expected to hit 38 billion devices in 2023. The ability to run machine learning algorithms on such quotidian hardware opens up a slew of new applications.  Making TinyML easy with AutoML TinyML the developer community has been kindled by the TinyML Foundation, a group of like-minded researchers and developers seeking to promote information exchange about innovative ML implementations on ultra-low power devices “at the very edge of the physical and digital world.” In promoting the idea of TinyML services, Ericsson offers a useful graphical depiction of where TinyML fits in relation to other computing paradigms, where it sees the movement at the intersection of IoT devices, edge computing and machine learning data analysis. 

(via Ericsson)

Source: Ericsson; TinyML as-a-Service: What is it and what does it mean for the IoT Edge? TinyML has been the inspiration for several tools and services designed to accelerate and simplify the development and deployment of ML software on embedded systems. One of the first was TensorFlow Lite, a variant of the popular AI development framework targeting mobile and embedded devices. As a presentation by one of its chief developers illustrates, training  a TF Lite model merely requires passing standard TensorFlow models through a converter before feeding it with sensor data. Similarly, inference works by running data through a preprocessor and the TF Lite interpreter. TF Lite works in most TinyML scenarios using 32-bit microcontrollers and has been extensively tested with Arm Cortex-M devices. The TF Lite runtime takes only 16 KB. A simple speech recognition app like wake word detection takes only 22 KB, while person detection in a grayscale image feed can run in only 250 KB.  TF Lite is perfect for developers already fluent in the TensorFlow framework and with an understanding of the limitations of embedded hardware, however, these requirements set a high bar for the millions of embedded developers. AutoML, a new development platform from Qeexo, is designed to lower these technical barriers by automating the data processing, model development, tuning and hardware provisioning for embedded developers.  Like ML automation cloud services or server software such as AWS SageMaker, Google Cloud AutoML, Auger and Sigopt (which I highlighted back in 2017), Qeexo AutoML:

Supports a variety of popular ML techniques Simplifies data preparation, labeling, validation and visualization via a management UI Provides no-code automation of most of the typical ML workflow  Supports most types of mobile sensors, including:

Motion: accelerometer, magnetometer, radar, gyroscope Acoustic: microphone, ultrasonic, vibrometer Environmental: temperature, humidity, air pressure, illumination, IR Image: photo/video, thermal Touchscreen: capacitive and IR Biometric: fingerprint, heart rate

And, builds memory-efficient models for Arm Cortex-M0 to M4 microcontrollers.

There are several alternatives to Qeexo’s system for embedded ML, including Cartesian NanoEdge, Edge Impulse, NeuroPilot Micro and OctoML.

(via Qeexo slide deck)

  Myriad applications The overriding impetus behind moving ML to the far edge is so-called sensor fusion in which increasingly capable edge devices combine, correlate and analyze data from multiple sensors to detect anomalies, objects and their relative positions and make predictions using ML that are far more accurate that simple trend extrapolation techniques. Applications span many industries and usage scenarios, including:

Industrial predictive maintenance   Cybersecurity  Smart city and home   Mobile and wearable devices (gesture detection, computational photography, medical health) Automotive  (ADAS, hands-free assistants)

These environments require rapid results since the type of streaming data they generate is fleeting, with exponentially-decaying value over time. Thus, locally performing the ML without sending it the cloud and back is critical to achieving near-real time low-latency.

(via Qeexo slide deck)

My take We remain in the twilight hours of TinyML as the capabilities of microcontrollers and sophistication of ML optimization have reached a point where incredibly useful applications can now run on near-invisible devices. Systems like TF Lite, AutoML and others will unleash the creativity of millions of embedded developers to infuse intelligence, interactivity and uncanny features in almost every physical object we interact with.  Qeexo has several examples that illustrate the way TinyML will reshape everyday products, including:

A touch-sensitive interactive wall An environmentally aware countertop Environmentally aware shipping boxes

From cars that tell you when an engine bearing is about to fail to kitchen faucets that warn of harmful chemicals in the water, embedded intelligence is set to revolutionize our interactions with everyday objects.

Image credit - Feature image - Intelligent car, intelligent vehicle and smart cars concept, by @jirsak, from Shutterstock.com. Screen shots credited above.

Read more on:
IoT robotics and AI Machine intelligence and AI

Author: Kurt Marko

Date: 2020-11-10

URL: https://diginomica.com/can-piece-drywall-be-smart-bringing-machine-learning-everyday-objects-tinyml


We benchmark, you score productivity, they surveil - the good, bad and ugly of teamwork analytics (2020-11-27) We benchmark you score productivity they surveil - the good bad and ugly of teamwork analytics Phil Wainewright Fri 11/27/2020 - 09:10 Summary: Microsoft has conjured up the ugly face of teamwork analytics with its Productivity Score but the data can also be used in a good way princerko - Fotoliacom It turns out that analyzing teamwork data is subject to one of those irregular verbs celebrated in..
Can social distancing with IoT contribute to safer workplaces? Learning from Software AG’s customers (2020-12-02) Can social distancing with IoT contribute to safer workplaces? Learning from Software AGs customers Jon Reed Wed 12/02/2020 - 04:22 Summary: Making workplaces safe during COVID-19 is no small undertaking This year Software AG and its customers have learned plenty about how IoT devices for smart social distancing can help At conXion 2020 we got a closer view into the field lessons so far Bakery Gbe.. Can social distancing with IoT contribute to safer workplaces? Learning from Software AG’s customers
Sainsbury’s job losses and store closures signal rapid digital change (2020-11-05) Sainsburys job losses and store closures signal rapid digital change Derek du Preez Thu 11/05/2020 - 03:40 Summary: UK retailer Sainsburys announced today that it would be slashing 3500 jobs and the closure of a number of its Argos stores Image sourced via Sainsburys website UK supermarket retailer Sainsburys today announced that it would be cutting approximately 3500 jobs and that it would also b..
conXion 2020 - In process transformation, Tesco finds that every little helps (2020-11-10) conXion 2020 - In process transformation Tesco finds that every little helps Phil Wainewright Tue 11/10/2020 - 00:28 Summary: A session at Software AGs global conXion 2020 event gives insight into the impact of process mining at grocery retail giant Tesco Tescos Jason Dietz Screengrab from conXion 2020 session Lean margins are a fact of life in the highly competitive retail grocery market Profitab..
How Salesforce’s acquisition of Slack enables the Frictionless Enterprise (2020-12-02) How Salesforces acquisition of Slack enables the Frictionless Enterprise Den Howlett Tue 12/01/2020 - 18:17 Summary: An analysis of how Salesforces acqiusition of Slack changes the way in which we can look at Frictionless Enterprise via Kusser In its press release announcing the Slack acquisition Salesforce said: Combination of #1 CRM platform with the most innovative enterprise communications pla..
Video meetings killed the messaging star (2020-12-01) Video meetings killed the messaging star Phil Wainewright Tue 12/01/2020 - 14:24 Summary: Video meetings became the surprise star of digital teamwork this year and suddenly messaging alone doesnt have what it takes Composite of Zoom and Slack images Back at the beginning of 2020 video meetings were interruptions to the flow of work People went to video for interviews to make or receive sales calls.. Video meetings killed the messaging star
Digital growth slips at Macy’s as CEO asks what’s the difference between ‘essential’ and ‘non-essential’ retail? (2020-11-20) Digital growth slips at Macys as CEO asks whats the difference between essential and non-essential retail? Stuart Lauchlan Fri 11/20/2020 - 04:05 Summary: Dont shut my stores! Macys CEO Jeff Gennettes plea around essential and non-essential retail as digital growth slows down in the third quarter PIxabay A theme that has emerged over the years in diginomicas coverage of the retail sector is the i..
Creating a culture of inclusivity at Natwest through better collaboration (2020-11-12) Creating a culture of inclusivity at Natwest through better collaboration Mark Samuels Thu 11/12/2020 - 00:33 Summary: NatWests head of Unix engineering says agile principles and employee-led networks can help boost employee experiences Image sourced via the Natwest website Business leaders must ensure employees know that their opinions count and should use a range of techniques to help people com..
conXion 2020 - Nordex on their push towards real world IoT at scale, a Software AG use case (2020-11-04) conXion 2020 - Nordex on their push towards real world IoT at scale a Software AG use case Jon Reed Tue 11/03/2020 - 20:03 Summary: Software AGs global conXion event is underway this week I seized the chance to dig into a customer story about real-world IoT Thats where we find out the true impact of the so-called Industrial IoT and the challenges companies face along the way via conXion 2020 Reade.. conXion 2020 - Nordex on their push towards real world IoT at scale, a Software AG use case
GoodData founder and CEO Roman Stanek on DataOps, radical openness, and how Snowflake changed the data value chain (2020-11-03) GoodData founder and CEO Roman Stanek on DataOps radical openness and how Snowflake changed the data value chain Jerry Bowles Tue 11/03/2020 - 04:29 Summary: Big things are happening in the world of data analytics as the data for everybody movement picks up pace Roman Stanek CEO GoodData via GoodData Roman Stanek is a man on a mission A veteran entrepreneur he founded GoodData an analytics company..