Wednesday, March 16, 2016

Watson IoT Platform and data storage

In December 2015, IBM opened new unit Watson Internet of Things, with its global headquarter in Munich, Germany. The core of the the new portfolio is Watson IoT Platform, formerly known as IBM IOT Foundation (IOTF).

Cloud data management model
Watson IoT Platform is handy way to connect IoT devices into the cloud. It provides device connectivity with MQTT - the standardized IoT protocol ISO/IEC 20922. Device management, token authentication, data storage and integration interfaces for further cloud applications.

Here comes the catch: Even if IOTF provides data storage, it should not considered as primary storage of your device data in production use. There are two main reasons:
  • Storing excess amount of data in IOTF is expensive
  • IOTF database does not provide QoS, data may be lost if delivered at high rate
So, what instead? Better way is to use IOTF only as an interface to cloud for your connected devices, and store your data in some No-SQL database like Mongo, Couch, Cloudant or Cassandra. This is much more robust solution as IOTF does provide QoS for data delivered.

Majority of device data generated in the world is deprecated just few seconds after it's created. Data aggregation may significantly reduce the amount of data that really needs to be stored for later use. However, if you wish to store all raw device data for later use like analytics not invented yet, it's better to use some long term data storage solution like Object storage, as data over there costs only a fraction of what it would cost to store it in the active No-SQL database.

This creates a three-tier data management model where Watson IOT Platform (IOTF) is the front-line to receive the data from devices. No-SQL database is the place where active application data is processed, and finally a Object storage or similar place is the attic for historical data perhaps never needed, but nobody dares to throw away.

Bluemix and Watson IoT Platform are tightly integrated and it's very easy to access the data in Bluemix application. Here is an example of simplest possible glue-logic to integrate IOTF and No-SQL database with each other ,Cloudant in this case. The example is implemented with Node-RED running in Bluemix Node.js application.

Integration of IOTF and Cloudant with Node-RED app in Bluemix.
The left-hand side IOTF node receives and data arriving in IOTF that meets the filtering criteria defined inside the IOTF configuration. The right-hand side cloudant node puts any data it receives into the defined cloudant database. If you need to do any filtering or data aggregation prior putting data into the database, just add a function node in between. If both IOTF and cloudant service instances are created inside the same Bluemix app, there is no need to configure any credentials here, those services are readily integrated.

In addition to being more robust than using IOTF database, this may be more economic as well. Free plan of IOTF provides storage space of 1GB for device data, where as Cloudant offers 20GB free quota per instance. Fetching data from Non-SQL database is also more comfortable with efficient query algorithms. Actually the REST API of IOTF only provides max 200 items at a time.

Reading data from Cloudant is as simple as writing it in, in Node-RED application. Here is an example of simplest possible implementation of REST API to fetch data from the database.

Implementation of REST API to fetch data from Cloudant in Node-RED.
With Watson IOT Platform, Bluemix, and Cloudant it's possible to build very quickly a robust, efficient, and cost effective IOT-solutions in cloud. The integration of IOTF and Cloudant overcomes all the pitfalls the current Watson IOT Platform has. 

8 comments:

  1. Great post Jaakko (as always). This "3 tier storage" architecture is exactly what I see in use with our customers. If serious analytics are required, the Cloudant DB is often replaced with DashDB or even a spark-based service.

    ReplyDelete
  2. rfid tags Hey what a brilliant post I have come a cross and believe me I have been searching out for this similar kind of post for past a week and hardly came across this.

    ReplyDelete
  3. After reading your post. I am able to define this information in the dissertation writing services data storage is the place where data is held in an electromagnetic or optical form for access by a computer processor. I will do more research on the types of data storage.

    ReplyDelete
  4. An author must have a vast knowledge of vocabulary. The dictionary of a writer must be full of new english vocabulary to make their work more attractive. Use of new words makes their work more valuable and graceful.QNAP TS ECx80 R2 Series

    ReplyDelete