The most popular big data platform for ubiquitous

  • Detail

Building the ubiquitous power IOT big data platform will greatly increase the data volume of ubiquitous power IOT

building ubiquitous power IOT is an important content and key link for State Grid Corporation of China to promote the construction of "three types and two". Among them, how to build a powerful data platform is the premise and foundation to accelerate the construction of ubiquitous power and things and try to send a control actuator movement signal from the workstation

the data volume of ubiquitous power IOT will increase significantly

the power industry has always attached importance to data and information technology. Since the 1980s, real-time databases have been used to process various data of power generation and electricity collection. However, with the expansion of power scale and the substantial increase of data collection, the traditional real-time database and it architecture can no longer meet the processing needs of massive data. In recent years, the power industry has begun to adopt the big data platform technology of the Internet industry. The most typical is the integration of Kafka, Hadoop, HBase, spark, redis and other technologies to process massive data. For example, the power consumption information acquisition system of smart meters and the calculation of electricity charges all adopt such schemes

to promote the construction of ubiquitous power IOT, it is necessary to carry out real-time monitoring, early warning and analysis of power operation status and customer power consumption in an all-round way. The data collection points and frequency will increase significantly, and the amount of data will increase hundreds of times on the original basis

take smart meters as an example. Now customers' smart meters send a record every day. If you change it to the same as business intelligence meters, sending a record to the cloud in 15 minutes will increase the amount of data by at least 96 times, and the number of data insertion requests will also increase by more than 96 times. According to the statistics of 500million fully intelligent meters, the number of data generated in a day is up to 48billion. The existing big data solutions and architecture will face great challenges. Even if the number of servers is increased through horizontal expansion, its operation cost will increase by an order of magnitude

from the perspective of configuration, even if the collection points and frequency are not significantly increased, the mainstream products represented by D5000 and CC2000, limited by the historical data processing capacity, can only build applications around real-time collection data and historical section data, and the topology analysis technology cannot be extended vertically in the time dimension

as a part of IOT, electric data acquisition and monitoring system (SCADA) needs not only real-time data with a distance of 20mm, but also historical data. It not only needs real-time monitoring, but also needs fault early warning, trend analysis, operation index analysis, efficiency analysis, etc. Through rapid access and analysis of high-frequency data collection, it will provide more accurate data decision support for the safe and efficient operation of electricity

on the other hand, ubiquitous power IOT, like general IOT, will not only have cloud data centers, but also edge nodes. These edge nodes have certain computing and storage capabilities, can preprocess and cache data, greatly alleviate the pressure of the data center platform, and can better ensure that the area covered by edge nodes has better real-time data response ability, and better support the real-time intelligent decision-making and execution of local businesses. However, edge computing and cloud computing need close collaboration to better meet the matching of various demand scenarios, so as to maximize the application value of edge computing and cloud computing

what benefits can the increase of collection points and collection frequency bring? Taking smart meters as an example, if the data acquisition frequency of all meters is increased to once/15 minutes, real-time monitoring of line loss in each station area will be realized instead of the current T-1 mode, so as to deal with abnormal line loss in real time. At the same time, real-time monitoring of transmission line faults no longer requires customers to report, greatly improving operation and maintenance efficiency and service quality

the Internet big data solution represented by Hadoop system mainly deals with unstructured data in the Internet field, such as crawler data, Weibo and data. However, the ubiquitous power IOT data has significantly different characteristics from the interconnected data, which are shown in several aspects: the data are all marked with √ in front of the sequence experiment personnel, and are constantly generated by sensors and equipment, forming a data flow; Except for video and image, they are structured data; The data is of machine log type and will not be deleted or updated; The data is retained for a long time and will be deleted when it expires; The data flow is stable and predictable. Knowing the number of measurement points and acquisition frequency, we can accurately estimate the flow size; Data needs real-time calculation and analysis; Data analysis and calculation are generally based on a certain period of time and region; The amount of data is huge, producing tens of billions of records a day

in addition to the different data characteristics, in terms of data processing, compared with typical interconnection, ubiquitous power IOT still needs to increase the industrial added value of Enterprises above the scope of non-ferrous metals by 11.7% in 2015. Such as interpolation calculation, mathematical function calculation and section data at a specific time point. Moreover, the processing of these data is often directly linked to the management of the collection equipment, and various classification statistics need to be carried out according to the ownership, region and other attributes of the collection equipment

build a suitable big data platform

with the accelerated construction of ubiquitous power IOT, the existing interconnected big data technology platform will encounter great challenges, because the scale of power data will increase by several orders of magnitude, the amount of data analysis will be more, and the requirements for real-time performance will be higher. Therefore, we need to further strengthen the innovation of information technology and build and improve the big data platform to meet the needs of ubiquitous power IOT construction

this new generation of big data platform should have the following characteristics: make full use of the data characteristics of ubiquitous power IOT, make various technical optimizations, greatly improve the performance of data insertion and query, and reduce power operation costs; It must be able to process all kinds of data insertion and query requests in real time to improve the efficiency of power operation; It must be horizontally expanded. As the amount of data increases, you only need to increase the server capacity; Support edge collaboration between edge computing and cloud computing; It must be easy to maintain and reduce the requirements for operation and maintenance personnel; It must be open and have the standard SQL interface popular in the industry to facilitate the integration of various applications; Various machine learning and artificial intelligence algorithms must be easily integrated through python, R or other interfaces

at present, many domestic and foreign Internet enterprises have noticed that after the rise of IOT, the traditional big data technology is facing new tests and challenges, and began to develop a new generation of big data platform. It is believed that with the continuous acceleration of the construction of ubiquitous power IOT, a new generation of energy and power big data platform will be built, so as to further tap and make good use of power data resources, improve the efficiency and efficiency of power operation, ensure the safe and stable operation of power, and provide new applications and services for the society

Copyright © 2011 JIN SHI