The way we manage energy data hasn't changed much over the past decade. New products offer more detailed visualisation and reporting of meter data, and building optimisation products are providing information on HVAC (heating, ventilation and air conditioning) and building management systems, with some real time monitoring and alerts. In the industrial world, SCADA systems still generally rely on on-site historian systems that collect historical data from site systems.
However energy management in Australia is becoming increasingly complex, as large users address substantial costs and risks. Factors driving complexity include persistent volatility in our energy markets, the penetration of renewable energy sources, emissions reduction commitments, and developments such as the electrification of transport and emerging models of decentralised energy generation, storage and distribution. As a result, the need is growing for sophisticated data management systems that offer valuable insights into energy usage and efficiency opportunities.
The limitations of energy data systems today
The trouble with almost all products on the market is that they are limited in the amount of data they can collect, analyse and report on. Most building systems collect limited data sets from sources such as the main power meters and HVAC systems. Most energy data management systems and SCADA historians are still running on traditional databases. Up to now this has not presented significant issues.
The problem these systems are now encountering is that the world of data from both buildings and industrial processes is rapidly changing. Rather than simply obtaining hourly meter data from a site, it is possible to install cheap sub-meters across all parts of the business and obtain minute by minute updates (even more frequently if desired). Most industrial equipment now has the ability to produce information for hundreds, if not thousands of metrics that would enable real time monitoring and reporting. Building managers can now install cheap wireless temperature/humidity meters into every room of a building and obtain minute by minute data on the impact of any changes to HVAC settings.
Big data solutions and where to find them
Just this month the Rocky Mountain Institute stated in an article, "Declining costs for metering systems, new automation capabilities, and the advent of cloud computing are creating enormous opportunities that we haven’t fully captured. The potential for big data is palpable."
The answer to questions about how to take full advantage of large amounts of energy and process data may lie in the world of Netflix, Salesforce and other companies who have to deal with data on very large scales. Netflix currently runs "tens of thousands of servers on the Amazon cloud” . Salesforce also has thousands of servers supporting both their online software and platforms for others to run their own applications. Both companies need to know exactly what is happening with these servers and collect many hundreds of pieces of data from each machine, every minute. For Netflix, as their business has grown, the number of data points has increased from hundreds of thousands in 2011, to billions of points today.
To process and extract full value from this data, Salesforce and Netflix have built their own systems.
Salesforce produced Argus, a system that can handle 25 million data points a minute using a standard set of 16 machines and up to 250 million points per minute with 25 machines.
Netflix produced an alternative system called Atlas. Once again it is optimised for time series data and can deal with "1.2 billion time series (corresponding to publishing billions of data points per minute)".
Both systems can deal with hundreds of millions (or billions) of data points per minute and more importantly give Salesforce and Netflix the ability to be alerted to any anomalies and easily act upon them.
The good news for the energy data world is that both Netflix and Salesforce have open sourced their systems, which means that they are free to use and build upon. You can find Saleforce's Argus here.
Are we on the verge of a transformation in the way energy is managed in Australia?
Energetics closely follows trends in big data management set by leading firms – whether local or global; in the energy sector or in any sector. Throughout 2017 we will continue to comment on the innovation we are seeing in data management and the opportunities that may arise for business to address the challenges of managing energy and carbon.
If you are interested in understanding how innovative big data solutions can inform your energy management strategy, please contact any of the experts below.
