Telemanagement & Big Data: A global perspective from IRIS and Hadoop

 

Introduction

With the legal requirements of replacing analog power consumption meters by electronic meters (smart meters), many spanish companies have to handle huge amounts of information. We are not only talking about direct consumption of a home user but a lot of information:
  • Reactive information. 
  • Reactive energies.
  • Maximeters.
  • Service interruptions.
  • Events related to power quality.
  • All information regarding changes and remote power cuts.
  • Meter PLC states.
  • Conectivity with gateways.
  • Etc.
All this amount of information, when a company have to work with thousands of gateways and meters must be properly and efficiently managed. This document presents a technological perspective that pretends to indicate the role of a telemanagement system of remote devices such as IRIS regarding with exploitation technologies and the analysis of large volumes of data.

 

Telemanagement of PRIME devices

We can define as Telemanagement system that lets you manage, operate, and take full advantage and performance of all field devices installed. IRIS is a PRIME Telemanagement system amply proven, mature and in operation long ago which, among other things, allows:
  • Easily manage thousands of devices.
  • Correct operation of the administration commands sent to devices.
  • Single platform for complete device management.
  • Maximize the devices performance (highest number of billings obtained) using smart strategies.
  • Multi-user, scalable hardware and the implementation in cloud.
  • Etc.
According to the information managed by IRIS (devices, PLC states, orders, tasks, historical connectivity, extensions, users, etc.), clearly the data repository must be structured and based on SQL, like "could not be otherwise".
IRIS uses a pattern repository that allows to use the platform with different database engines; in its current implementation is used with great success and scalability SQL Server 2012; well you could use Oracle, MySQL or any other database manager.
The nature of the information stored in IRIS is related to the need for a Telemanagement system and that properly operate measure devices for:
  • Send orders correctly and keep a trace of them.
  • Read avalanches of daily billing in an high performance.
  • Massive sending of firmware changes.
  • Export to the billing system, ticketing system or MDM (for example), the necessary information for the subsequent decision-making.
  • Etc.
These are the capacities that should roughly incorporate to Telemanagement system of an electric utility.
Now, what volume of information we are talking about?

 

Volume of data managed by IRIS

The IRIS database is extremely optimized and designed for efficient use from the point of view of a remote management platform.
Our Azure deployment with virtual technology gateways, made for the simulation of one million meters, has allowed us to draw some valuable conclusions, including: 
  • In a real scenario, the system manages files between 40k and 60k a day.
  • The avalanche of daily billings are processed in less than an hour (with the current scaling of demonstration).
  • For a million meters, and based on our experience with electric companies using IRIS, the system manages between 5k and 15k daily orders.
  • The database grows at a rate of 400Mb per day (with a 1M of meters).
  • The operation and use of the user interface for an operator is light, fast and efficient. More data does not imply slower in daily operation because the database is sufficiently well designed and indexed.
In conclusion, the performance of SQL Azure (SQL Server 2012) is very high and it follows that for a database of 4M meters, equally acceptable.
But, what about data locks that would accumulate over time?
We understand that here is the crux of the matter: for a system of 4M meters in operation for one year, we concluded that the IRIS database would not reach 2TB of storage space, far from its limit.
While Telemanagement platform must manage the base of installed devices, a common misconception is to think that this system must also:
  • Acting like a historical data repository.
  • Provide a high-level treatment of information for decision making.
Both characteristics is what is responsible for the big data: providing efficiently information to manage and process large volumes of data to extract knowledge from the same mechanism.
In the context of big data the meaning of "large volumes" is from tens of terabytes up to several petabytes; however, we noted that one year of operation at remote management, IRIS need less than 2 terabytes, occupying 80% of this space billing values and historical PLC.

 

Then came the big data

As we indicated, the big data pretend to offer a lot of tools to store huge amounts of information to be analyzed later. So, big data is not:
  • A simple box of huge amounts of data backups.
  • It is not a way of structuring information for a multi-user operating system for daily work.
In our environment Big Data is:
  • An extremely useful tool to extract value of huge amounts of information, for example:
    • Patterns of behavior on the daily closures for fraud detection.
    • Geographic areas with poor communications coverage.
    • Patterns of behavior at PLC level.
    • Demand management: Given an historical consumption we can estimate the "probable" consumption for August 15, as example.
    • Gateways whose consumption does not match with the consumption of their meters.
    • Network topologies about PLC problems.
    • Times zones with higher / lower consumption by area.
  • An homogeneous, secure and scalable mechanism with which to implement what is necessary to answer the above questions at a reasonable cost.
Therefore, we conclude that a system implementing big data should not we supposed to be the heart of a remote management system, but, with the help of this fits into the next level of information processing of remote management to extract useful value of this.

 

A real business proposition: IRIS + Hadoop (HDInsight)

IRIS includes a set of interfaces and mechanisms for the export of all information managed by third parties:
  • Xml files to ftp client.
  • Opening trouble tickets.
  • Automatic sending of closures / events to ERP ( such as SAP )
  • Etc.
Similarly, from the perspective of big data , IRIS provides naturally all the information that a big data needed to draw conclusions and make analysis of the data obtained by Telemanagement platform.
In the big data market, was being established the way of widespread use of Apache Hadoop, and supported by major software companies such as Oracle and Microsoft may be.
 
IRIS can be deployed in a cloud environment such as Microsoft Azure, as already indicated earlier in this document relating to the deployment of 1M of virtual meters. Azure provides Hadoop implementation for secure and agile big data creating environments through what Microsoft calls HDInsight.
 
For this reason, the IRIS platform Telemanagement aligns naturally with Hadoop and its implementation by Microsoft in HDInsight, offering a total package with all the technological tools necessary for a large electric utility to operate successfully the deployment of smart meters and gateways.
 
Also, we are pleased to indicate that given the distributed and scalable nature of IRIS, the Remote Management Platform can be deployed on Azure as well as in local, hybrid mode, with SQL Server / Oracle and in conjunction with any other solution big data (SAP HANA, Oracle Exadata, etc).

 

Referencias

Thursday, February 27, 2014