Machine Learning For IoT

Machine Learning for IoT

This article scans the edge of the IoT ecosystem discussing the pace at which intelligent devices are emerging. The overview of protocols for device communication is done with a detailed explanation of federated learning with its challenges and usage in various situations.

Explosion of IoT Market

McKinsey reported that the IoT would be a $11.1 Trillion market value by 2025. According to Bosch, multinational engineering and technology based in Germany, there will be over 14 billion connected devices and as per Cisco, there would be 5 billion connected devices. $309 Billion of IoT supplier revenue would be generated with a $1.9 Trillion Economic value addition in mean time as per Gartner. Moreover, the IoT solutions revenue Is likely to reach $7.1 Trillion according to the IDC.

Here is a report from MIT Technology Review, which shows the growth of IoT devices in the past 10 years from 2010 to 2021. These are connected devices that are bound to exceed the PCs, tablets, and smartphones numbers in time.

Courtesy: MIT Technology Review  

Implications 

There is no denying that the IoT storm is here and likely to stay and even grow in years to come. We will have to manage billions of interconnected devices. A huge volume of data was observed in 2020. Daily traffic ranged to approximately 1.5 GB from an average internet user with 4000 GB per day from autonomous vehicles and 3000 GB from Smart Hospitals on a daily basis. Radars and Sonars produce over 10-100 Kbps with GPS over 50Kbps and Cameras creating 20Mbps. The connected aircraft are responsible for creating 40,000 GB per day with connected factories creating over 1,000,000 GB on a daily basis.

The connected devices and intelligent systems are not new. The PLC, SCADA, Automated factories have been there since the 1980s with electrical and magnetic sensors connected to logic controls. Most dumb sensors connect to intelligent hubs and control hubs got more distributed with & miniaturized with micro-controllers. The biggest push came after cheap ARMs/ SOCs emerged which reduced the cost of intelligent nodes and changing computing scheme. With the newer IoT devices in operation and usage now, the global market has risen exponentially. 

IoT Devices Today 

The IoT devices of today have local CPU, memory, NIC, WIFI, GPIO, UART, GPS, GSM, LTE in a small package. Due to the small packaging and with limited costing, CPUs are not very powerful however sufficient enough with low memory that ranges from a few MB to a few GB at most. They also have slow lossy links and the batteries are operated most often at remote locations. Some of the widely used IoT devices today are Arduino, Beagle board, Raspberry Pi, ESP8266, ESP32, and more. These IoT devices are used in numerous sub-sectors today.

Protocols

Protocols are specific guidelines of rules which are followed for any procedures. The IoT devices follow certain groups of Protocols. Because the IoT endpoints are intelligent as it has CPU, we will need a communication protocol over low power & lossy channels. Most protocols are modeled after the web style such as HTTP(S)/ WebSocket with proxies or direct or use message queues. The scale is much smaller due to restrictions on packet sizes and transfer rates. There are numerous communication IoT Protocols in use. Some of the major ones are listed below.

MQTT- Message Queue Telemetry Transport 

It is lightweight with a small footprint publish-subscribe protocols which transport the messages through the network between devices. It mainly runs over TCP/IP and requires a message broker which resembles CORBA. It is also data agnostic and the message is delivered with or without confirmation or guarantee but many deliver and guarantee delivery once only.

CoAP - Constrained Application Protocol 

It is a specialized protocol which is a lightweight fast HTTP used specifically for constrained nodes with limited resources using RESTful interface. Many contact devices to cloud or via proxy to cloud, or device to device uses CoAP.

V2X Protocol

The Vehicle-to-everything protocol is used to connect vehicles with the other parts of the traffic system around them. This helps the connection of cars with other cars, transport infrastructures, data centers, and pedestrians thus underpinning its name as Vehicle-to-Everything.

There are few other protocols too which include cloud and numerous communication mediums. 

Converging

In Machine Learning, the data from multiple nodes are sent to a central server for modeling, training, and testing. The model training specifically is performed in the central server and it is where all decision making is happening. But, a similar approach to send all the data from nodes to the central server could lead to numerous issues from data breaches and privacy concerns to huge overheads. Understanding deeper into this problem to replicate this for IoT without creating communication overhead will help solve this issue.

Privacy

Privacy is a huge factor when sensitive data are being used. With IoT devices, a lot of data are sensitive in manner and it is integral to make sure no breaches be possible leaving out vulnerabilities to be exploited without having the expensive on the process to provide the main goals or services, for instances like autonomous vehicles need functioning in real-time otherwise, serious other problems could arise while only focusing on privacy too.

To learn more about Federated Learning, watch this video:

Solution 

Development of Intelligent Systems

Each system can make decisions based on a variety of stimulus. Distributed decision making would be helpful as we cannot traverse long distances with so many devices involved. The intelligence is driven by data which is easily possible as the IoT enables data collection and hence in-place decision making can be easily done.

Edge Computing - Distributed Machine Learning

Edge computing refers to the distributed architecture that features a decentralized system enabling computing in the devices itself rather than the servers and having to transmit data to the data centre thus mitigating the cause of overheads in the process, faster processing, and transmission less solutions. Here, mobile phones or IoT devices can do this data processing which exponentially helps for improved response.

Federated Learning

Federated learning is a special case of distributed machine learning which focuses on enabling devices to learn from each other with the goal to train models over a distributed channels of large number of individual devices functioning as clients. This helps break down the privacy challenges as raw data need not be sent to the server itself. Some of the widely used federated learning applications can be found in Google Assistant.

The steps in which federated learning can be done with the server and multiple nodes are as follows,

• The server contains the untrained model 
• The server sends a copy of that model to its nodes 
• Each of the nodes now also have the untrained model 
• The nodes now contain the data on which the model can be trained
• Each nodes trains the model to fit the data they contain 
• Each nodes sends a copy of its trained model back to the server 
• An average is taken from all the combined models in the server
• The server now has a model that captures the patterns in the training data on all the nodes 
• The server now sends a copy of the updated model to each of its node 

Federated Averaging

The Federated Averaging (FedAvg) is an algorithm that enables distributed training for a large number of clients through generic neural network model which uses gradient descent and sends the trained weights back to the server. The average from all the updates is taken to return the final weights by the server. Introduced by Google in 2017, for personalized keyboard experience, federated averaging is a relatively new approach in technology. In 2016, Federated Averaging was introduced in a paper, Communication-Efficient Learning of Deep Networks from Decentralized Data.

Federated learning should be used when privacy is a requirement and the bandwidth are a concern simultaneously with power consumption. It is of high value when the data transfer cost for the application is huge, thus making the system cost-efficient when federated learning is used.

However, federated learning wouldn’t be a viable option, when more data necessarily won’t improve the model. This can be tested through the construct of a learning curve. Also, when additional data is uncorrelated to better performance or the performance is through the roof already, the cost-benefit advantage of Federated Learning won’t be huge.

There are a few considerations to take before beginning with Federated Learning. Always test the Benchmark performance initially. Check if the performance improves with more data. This would mean more data would better the model in time if found positive. Thereafter, the Models can be meaningfully combined. Also, it helps to realize that the Nodes can train models and not only predict. And thus, it is empirical to determine how the transferring of the models will be executed.

Tools of Trade

OpenMined

OpenMined is a community for open-source which focuses on developing, researching, collaboration, and elevating the tools to create more secure, artificial intelligent, and privacy preserving tools. PySyft is one such tool. 

PySyft

PySyft is a library that is open-source that has enabled secure, private computation in deep learning which is combined with federated learning and differential privacy using one programming model integrated into multiple deep learning frameworks for an instance, Tensorflow, Keras, and PyTorch.

TensorFlow Federated

Tensorflow Federated is an open-source framework designed to cater to machine learning computations for decentralized data enabling global models to be trained in local clients.

Challenges

Dropped Connections

When some nodes are off the grid or internet, the connections will be broken hence the transfer of data or model training would be interrupted.

Privacy

Privacy is a huge concern as though the data might be off the system but the model updates will be critical to be made leaving out vulnerabilities. Thus, encryption based might be used to preserve privacy but could create an overhead due to the time complexity to solve some time critical problems. 

System Issues

Power Consumption is a huge issue as the edge devices can be drained off its battery for complex work of model training using federated learnings.

Stragglers

Stragglers refer to the devices which failed to respond or train the models during the time window.

Thus, in this article, we had an overview of IoT and learnt about Machine Learning for IoT devices. We learnt about the current scenario of IoT comparing to its past 10 years history, its implications, and evolution. Moreover, the protocols for IoT were discussed with the concerns, problems, and the ways to solve them. Then we dived deep into Federated Learning and the tools of trade that had made Federated Learning a possibility today. Finally, we discussed the challenges with Federated Learning as it is an open problem as research is ongoing as it is a newer technology and is a work in progress. The scope of Federated Learning is huge and IoT is here to grow. This is just a bare minimum introduction to Machine Learning to IoT as we have just scratched the surface as of today.

References

• https://arxiv.org/abs/1602.05629 
• https://www.openmined.org/ 
• https://blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-methods-and-future-directions/