As the machine-to-machine (M2M) revolution gets underway, there are some interesting developments in the technology that makes it possible and cost effective.
The term normally used is big-data, and there are lots of moving parts where big-data is concerned. The main thing is the reduction in the cost of storage. There have been petabyte databases in the past couple of decades, but these were restricted to high value applications. Some of these include tracking the nuclear stockpile and data mining for very large retail organizations (e.g., Wal-Mart). The storage costs for these projects were huge, but the scale made them cost effective. Now, for much lower "value" data we are seeing an explosion of the data kept and analyzed.
As Al Presher mentioned on this site in his article The Industrial Internet of Things, companies like General Electric see a continuing growth in M2M data with a consequent increase in efficiency. It is the ability to analyze this data and apply it to both operation and design that is significant. Some of the interesting things that can be done include real-time analysis of trends for failure prediction as opposed to failure analysis. Thus, large amounts of historical data can be included in the control loop. In design, the ability to access large datasets for similar designs can help improve a new design.
The cost of storage is coming down all the time because of innovations in the storage devices and control. And that is just for the current technology of spinning disks. As solid state storage progresses, it promises to decrease energy consumption and increase speed of access. This should also lead to increases in density and ultimately storage capacity.
But, there are even more exotic technologies being developed that promise even greater efficiencies and capacity. One is the use of DNA to store information. This technology may be a few years off, but it shows promise in the lab today. Rather than storing the information on electronic devices, this technology uses actual strands of DNA. One interesting aspect of this is that the compounds are stable for very long periods of time. They are also much denser than current technologies. Add to that the fact that they do not require power when sitting idle and you have the ability to store huge amounts of data much more cheaply.
With the M2M population expected to reach 50 billion connected devices in the not too distant future, the amount of data generated is going to be tremendous. Storing all that information is truly a big-data challenge.