Why machine vision matters for the Internet of Things

Connecting machine vision systems to the IoT creates a powerful network capability. Being able to identify objects from cameras allows the local node to be more intelligent and have greater autonomy, thus reducing the processing load on central servers and allowing a more distributed control architecture. This is turn provides more efficient operation that requires much less external input.

by Mark Patrick, Mouser Electronics

Machine vision has developed in great strides over the last decade. State-of-the-art algorithms capable of detecting edges and movement within video frames, alongside advances in silicon technology relating to image sensors, programmable logic, microcontrollers and graphics processing units (GPUs), have helped bring it into a wide range of embedded applications. More sophisticated designs that can be downloaded to an FPGA are being used in conjunction with new development environments, such as OpenCV, to make machine vision much more accessible to embedded system designers.

This growing proliferation of machine vision is converging with the trend of linking up industrial systems to the Internet of Things (IoT). As sensors become increasingly intelligent, driven in part by the supporting computer vision algorithms, so the data produced is offering valuable insights into the operation of industrial systems. This in turn is opening up new ways of monitoring equipment, with autonomous robotic systems such (as drones) being connected to IoT infrastructure.

Part of the move to machine vision is driven by bandwidth considerations, while the other major motivation is the prospect of automating more parts of an industrial operation. One of the key applications for machine vision is in inspection systems. High performance camera systems with CMOS image sensors have fallen in price considerably over the last ten years, allowing higher resolution examination of boards and systems during manufacturing. These camera modules are combined with FPGAs to add more processing and decision making. This allows the camera to respond accordingly to the received data, reducing the need to send video over the network and enhancing overall operational efficiency.

Robotic machine vision system

Connecting the machine vision elements of inspection equipment to the IoT provides more data for the enterprise systems that are undertaking analysis on the performance of the factory. Rather than raw data, machine vision can provide information at a level of abstraction that is suitable for such enterprise systems. This markedly reduces the bandwidth overhead both for the servers and the network as a whole - the enterprise systems are handling millions of data points coming from the IoT, so any reduction in the load on the servers will help to make more timely decisions.

The rise in machine vision uptake has also opened up the market for robot guidance systems in automated factories. Inspection machines can thus bypass the central enterprise servers and communicate directly with other equipment in the factory, based on the results derived from the machine vision systems. This increases efficiency and again reduces the load on the network and servers. Machine vision is also being used to control automated equipment, particularly in materials handling. This encompasses everything from the control systems for autonomous robots transferring material around a plant through to the automated picking machines in warehouses identifying products.

For autonomous materials handling robots, machine vision can be as simple as determining a line on the floor to follow from one location to the next. However, it can also be used to detect people or obstacles in the way, allowing factory operatives and robots to work together safely and more efficiently. As already mentioned, product picking is now also employing machine vision - with items being identified via their barcode and then a robotic gripper being aligned to capture a particular item and place it into a basket. The camera and local processing to accompany this are both essential, and the pickers/autonomous robots are also constantly monitored as part of the wider IoT.

This has even extended to the air, with Unmanned Aerial Vehicles (UAVs) increasingly relying on machine vision. UAVs have proven to be a highly effective method for carrying out inspections in hard to reach areas, such as oil pipelines and gas installations. As well as allowing the UAV to identify a particular target area and approach it, so that it can be examined more closely, machine vision is being utilised for anti-collision purposes - avoiding fixed obstacles and even other UAVs by linking the camera system to the on-board flight controller.

Robotic machine vision system

Then there is the surveillance market. The increasing use of machine vision has tremendous implications here. Instead of feeding back Megabits of video data every second for an operator to examine, video can be processed locally and alarms triggered without needing any human intervention. The machine vision algorithms running on FPGAs are becoming increasing accurate. As a consequence they are much better able to differentiate between the movement of an intruder, an animal or leaves on a tree (for example), allowing an operator to simultaneously support a larger number of surveillance nodes. Furthermore, surveillance cameras can themselves instruct other machines to respond to an alarm. The combining of autonomous ground and air vehicles, such as UAVs, potentially changes the whole way in which surveillance operates. Rather than fixed cameras that can be avoided, imaging systems are instead mounted on airborne craft that are constantly monitoring the area while in flight. These UAVs then return to a charging base as their batteries run down, and other drones are sent out to replace them. This means constant surveillance cover can be provided. More advanced machine vision algorithms are able identify potential threats and then summon other air and ground craft to the area to further monitor the situation - all without the involvement of an operator. The same type of scenario applies equally well to agricultural applications, where the machine vision algorithms on an airborne craft can monitor the condition of crops and direct an operator (or an autonomous tractor) to the target area if an issue arises that it deems needs some form of responsive action.

The applications outlined here have been enabled by progress made in the underlying hardware and software technologies. The structure of machine vision algorithms is increasingly sophisticated and they can be downloaded to the latest FPGAs and GPUs. These devices can handle 8 or 16 channels at a time, supporting rates of 60 frame per second. They can also be coupled with high level software such as OpenCV.

Originally focussed mainly on research and prototyping, in recent years OpenCV has increasingly been used in deployed products on a wide range of platforms - from cloud to mobile. The latest version, OpenCV 3.1, has just been released. The previous version, 3.0, was a major overhaul, bringing OpenCV up to modern C++ standards and incorporating expanded support for 3D vision and augmented reality. The new 3.1 release introduces improved algorithms for important functions such as calibration, optical flow, image filtering, segmentation and feature detection.

Future possibilities

Machine learning is the obvious next stage after machine vision. Computer vison algorithms are able to provide deterministic analysis of still images and video content, but machine learning is applying other neural network approaches to ‘teach’ a system what to look for. The latest version of OpenCV, for example, now supports for deep neural networks for machine learning.

The increased performance of FPGAs and GPUs is opening up new opportunities for machine learning. This relies on a training phase, where the neural network is shown many different images that are tagged with the objects of interest and is usually handled by a large server system in a lab or in the cloud. It produces a series of weights and bias data that is then applied to the same network implemented in the embedded design. This ‘inference engine’ uses those weights to assess whether new data it is seeing contains those objects. For example, the latest surveillance cameras are using neural network machine learning algorithms to go beyond traditional functions (like monitoring and recording) and offer additional video analysis features (such as crowd density monitoring, stereoscopic vision, facial recognition, people counting and behaviour analysis). This local processing can then be delivered into the IoT and thereby integrated into broader analysis software within the cloud.

 


Related


The rise of edge computing

Edge computing in IoT is opening new opportunities for embedded designers. FPGAs can be used to aggregate data, and once in place can also process that data and deliver real time analytics. Coupled wi...

Secure Software Licensing for Industry 4.0

The future of the industry is digital and intelligent. At the core of the new infrastructure, we find Cyber Security 4.0 and Digital Business 4.0, both facilitated by CodeMeter Embedded 2.0. The b...

Building Secure & Reliable IoT Devices

Network security will be one of the defining issues for embedded developers connecting to the IoT. Mass connectivity will present new risks as companies transmit and store data to create much rich...

Making things come alive in a secure way

Considering the security of one single business application is not enough anymore. With the IoT, multiple applications across multiple industries can share and exchange data across different types...

Monitoring and Securing the Smart Grid

Smart grids use phasor measurement units (PMUs) to assess and manage independently operated grid systems. National Instruments, Hewlett Packard Enterprise, and OSIsoft created a "best of bree...

IoT Kit Pairs Gateway with Cloud Platform

Few companies have expertise in both embedded design and Big Data—or know how to combine these domains. The Advantech IoT Gateway Starter Kit simplifies matters. Find out how this gateway co...

Vending Machine IoT Makeovers

Operators of large vending machine fleets need ways to update these machines to profit from Internet of Things advantages. Explore a retrofit solution engineered by Bsquare and AAEON that uses a I...

Protecting IP in the IoT with Trusted Computing

In this on-demand webinar Steve Hanna, Senior Principal at Infineon Technologies, and Guenther Fischer, Senior Consultant Licensing and Protection at Wibu-Systems, present a broad picture that hig...

LTE-A for Data Intensive IoT Applications

LTE-Advanced (LTE-A) is the latest enhancement to LTE. As its name implies, LTE-A is well suited to applications that need the next level of performance. Find out how to improve practical data rates a...

The Rise of the IoT Analytics Gateway

Collecting large amounts of edge data and sending it to the cloud can slow network performance and increase storage costs. Learn how equipping IoT gateways with dual-core processors enables analyz...

Creating embedded systems enhanced with IoT

Engineering a bridge between the physical world and the online universe Engineers have gained some design flexibility with the advent of the Internet of Things (IoT) and the many options for produc...

Turning Healthcare Data into Value

Everyone wants to reduce healthcare costs and improve care. Tapping Big Data helps accomplish both. Discover how an IoT health analytics platform monitors patient populations and helps improve wel...

Six Hidden Costs in a 99 Cent Wireless SoC

When adding wireless functionality, the two approaches generally available include using a system-on-a-chip (SoC) or a module. The SoC is lower cost, but comes with various unknowns of how to desi...

What’s the right security for IoT?

As our world transforms into a highly connected architecture, security has become a strategic concern. Impacts extend beyond financial losses into the physical realm. This video will give you a quick ...

Licensing and Security for the Internet of Things

The IoT needs much stronger protection than the closed systems of the past. Safeguards against reverse engineering and manipulation are required along the entire design chain – from the hard...

M2M Communications and the Internet-of-Things

Machine-to-Machine (M2M) Communications used to be a niche topic. But today it is regarded as the key technology of a networked, mobile future. On this microsite we present a collection of White Paper...

Planning a scalable long-term wireless strategy

For companies with global deployments, there is no single right answer for planning a scalable long-term wireless strategy. The best option, focus on solutions that allow for reuse of development, tes...

IoT makes Smart Foundation for vacation resort

Vacation resorts are looking for ways to help conserve water and energy while improving guest comfort and enjoyment. With the rapid growth of the Internet of Things and the increasing availability...

Connect building systems to the IoT

In this article, we will explain how smart buildings can connect to the cloud, and show how building owners, equipment providers, and service companies can benefit from this connectivity. Specific...

IoT takes mass transit to a new level

The Internet of Things is critical to Bus Rapid Transit (BTR). It enables real-time collection and transmission of data, helping improve fleet management, scheduling, ticketing, safety, and advert...


Making your device secure

The internet of things is faced with a major security challenge. Compared to traditional, often unconnected embedded systems, the nature of IoT devices radically increases the risk of attack not just ...

Home Technology — From Connected to Proactive

While Connected Home adoption continues to expand, we are looking ahead to the next phase, developing the technology that will guide the transition to homes that grow more proactive and automated than...

 


Dirk Giesen describes the Parasoft tool suite for Embedded Software Development

Are you responsible for embedded software development in your organization? Your goal should be to create safe, secure, and reliable software. To make sure your device will work properly, deploy Paras...


Ross Sabolik of Silicon Labs talks about advanced Power over Ethernet

In this video Ross Sabolik of Silicon Labs talks about smart  Power over Ethernet systems with Alix Paultre at their APEC exhibit in San ANtonio, Texas. As PoE migrates to higher power levels and...


Dialog Semi walks through their latest IC solutions for battery chargers

In this video an engineer from Dialog Semiconductor walks us through their latest ICs for battery chargers at APEC 2018. Dialog's Qualcomm Quick Charge adapter solutions offer high efficiency to e...


Steve Allen of pSemi explains their latest LED driver solution

Steve Allen of pSemi explains their latest LED boost product based on Arctic Sand's two-stage architecture. Their PE23300 has a charge-pump, switched-capacitor architecture that offloads most of t...


Teledyne describes their latest 12-bit Wavepro HD oscilloscope

In this video Teledyne LeCroy describes their latest Wavepro HD oscilloscope to Alix Paultre of Power Electronics News at the company's launch event. The WavePro HD high-definition oscilloscope de...


Dialog Semi walks through their latest IC solutions for battery chargers

In this video an engineer from Dialog Semiconductor walks us through their latest ICs for battery chargers at APEC 2018. Dialog's Qualcomm Quick Charge adapter solutions offer high efficiency to e...


ROHM explains their latest wireless battery charger solution kit

In this video an engineer from ROHM goes over their latest wireless power development kit, co-developed with Würth for embedded development. The kit provides a complete wireless power transfer sy...


Tektronix describes their latest mixed-signal oscilloscope

In this video Tektronix explains the features in their latest 5 Series MSO Mixed Signal Oscilloscope. Features include an innovative pinch-swipe-zoom touchscreen user interface, a large high-definitio...


AVX shows a supercapacitor demonstrator at APEC

In this video Eric from AVX explains their supercapacitor demonstrator box at APEC 2018 in San Antonio, Texas. The box shows how a 5V 2.5-farad supercapacitor can quickly charge up using harvested ene...


OnSemi explains their latest passive smart wireless sensor for industrial applications

In this video On Semiconductor explains their latest wireless sensor for hazardous environments at APEC in San Antonio, Texas. Intended for applications like high-voltage power cabinets and other plac...


TI demonstrates an improved gaming power system at Embedded World

In this video Texas Instruments' explains Significant reduction in ripple, which results in improved reliability and increased design margins, among other advantages. Another benefit that improve...


Infineon explains their latest motor drive technology at APEC 2018

In this video Infineon demonstrates new gate drivers using their LS-SOI technology at APEC 2018. In the demo Victorus, an Infineon application engineer, shows in real time how much better thermal the ...


STMicro goes over their latest wireless-enabled microcontroller for the IoT

In this video STMicroelectronics goes over their latest wireless-enabled STM32WB microcontroller for the IoT and intelligent devices in several live connectivity demonstrations at Embedded World 2018....