The definition of Big Data is simple -- it’s the collection of large amounts of information. Going deeper, we include the ability to manipulate this data through analysis. It’s not a storage issue; it’s a transaction and analytics issue. If storing massive data were the point, we wouldn’t be obsessing about big data. The point is using data from a wide range of sources -- sensor data, demographic info, physical qualities -- to detect patterns and make decisions based on the knowledge derived from those patterns.
The ability to process massive data changes our behavior. Data analysis of stress on new materials allows the automotive and aerospace industries to bring strong, lightweight, sustainable materials into production. Big-data analytics allows plants to become ultra-optimized, greatly reducing energy consumption and reducing the overall cost of manufacturing. Big data is helping sustainable energy sources compete against fossil fuels. Ultimately, big data will keep our cars from bumping into each other.
Click on the image below to start the slideshow:
In a few short years, we will all feel the effect of big data in our cars. They won’t bump into each other as often, and quality problems with deadly accidents and massive recalls will begin to taper off. Google's autonomous car is the icon for big data in automotive. But it will be more. It will include intelligent design, intelligent manufacturing, and intelligent sensors that will reduce accidents dramatically. (Source: thebigdatainsightgroup.com)
Rob, what you describe are the three V's of Big Data. These are Volume, Velocity and Variety. I put them in this order because this is how they appeared in the database market. First there were very large databases. These typically very expensive to store and process, thus they were confined to very high value applications. Next was Velocity. In the early 1990s some telcos were using a DBMS to collect data on calls in real time to manage their network. Most recently we have seen the addition of Variety. Since storage has become so cheap and dense (both are important) and processing has become very cheap, we now keep stuff we could not afford to store before. In addition, we keep information that has a low information density, such as social media data. This can still be very useful even to industrial companies. It allows them to find out things they could not in other ways.
Interesting slideshow. I always sort of wondered what "big data" meant--it's one of those technical yet abstract concepts that can baffle someone if you don't know exactly what it means. This gives me a much better idea--sometimes visualizing a concept helps. Obviously it still can mean a lot of things depending on the application, and is an interesting concept for the future.
I appreciated the reference to the movie Money Ball because "Big Data" is really a huge paradigm change and can be hard to accept - just like the resistance for the system in Money Ball:
"For forty-one million, you built a playoff team. You lost Damon, Giambi, Isringhausen, Pena and you won more games without them than you did with them. You won the exact same number of games that the Yankees won, but the Yankees spent one point four million per win and you paid two hundred and sixty thousand. I know you've taken it in the teeth out there, but the first guy through the wall. It always gets bloody, always. It's the threat of not just the way of doing business, but in their minds it's threatening the game. But really what it's threatening is their livelihoods, it's threatening their jobs, it's threatening the way that they do things. And every time that happens, whether it's the government or a way of doing business or whatever it is, the people are holding the reins, have their hands on the switch. They go crazy. I mean, anybody who's not building a team right and rebuilding it using your model, they're dinosaurs."
Thanks for the quote, Nancy. In the case of Money Ball, as in the case with much of Big Data, is to use calculations to discover true efficiencies -- and to discover those efficiencies free from prejudice.
Of course Big Data does other things as well -- physical calculations that used to take months can be done in a few hours. That's helping with composites and with creating materials that are lighter than steel, yet stronger.
Ultimately, Big Data will be used to keep cars from bumping into each other. That could save 30,000 lives each year in the US alone.
Big Data is a big deal. I loved Money Ball for its demonstration of the value of data over "gut" feelings about ball players.
I loved hearing the logic behind the kid's explanations in Money Ball...
I'm still trying to grasp the overall concept of "Big Data" Rob, but anyone in the semiconductor industry can tell you of the value of data - lots and lots of data. Pre-stress testing, post- stress testing qual data. Production test data...A product engineer is often a master at detecting trends that can vastly effect production and design decisions. Now stepping that task up to the level of "Big Data" and the analysis task required is mind boggling.
The slide that caused some concern was the one that showed how big data could bring HR policies in line with actual behavior. While I do agree that statistically there is merit to relying upon big data for desired HR results, over-reliance on big data alone will also cause some bad decisions and eliminate some potentially good candidates. A good balance between big data and common-sense must be found for optimum results in the future.
Greg, I totally agree with you that big data alone is just zero and of no use in order to make big data usefull one should use his or her brains as well. Totally or blindly relying on data will be stupidity one should analyze the data and for analysis brain is required .
Greg no doubt big data is very usefull for HR department as well because we all know these days the functioning of HR has changed completely . It is not just about setting pay roles , increments incentives and so on but the actual function of HR is now employee development , training , making the employee grow , analyzig the skills and positive points and then making strategies and identifying trainings to give. All this requires data , deep understanding of data , analysis and having the potential to work , play and taking out meaningfull results with data .
Exactly Greg these days most of the companies have piles and piles of data but the successfull oranisations are one who have the skill, brains , power and aptitude to analyze that data and dig deep down information and make the data usefull for the growth of not only organisation but also for employees working in it .
Nancy is certainly correct: It is quite an irritation to attempt to locate information on a topic and instantly have 492 sites offering it for sale, whatever it is. And a large portion of the sites don't even have that item actually for sale, they just claimed that they did. So it would be good if there were an added code for seeking to purchase, versus seeking information. And the penalty for claiming to have something for sale and not having it for sale to be a years banishment from the internet. Sometimes it is more like 492,000000 sites claiming to have a product, or at least something with some of the same letters in the name.
Thanks Rob for such an interesting post no doubt these days Big data is becomming very popular ,Now a days there is not a single multinational which is unaware of big data or which is not equipped with it because today big data is playing a very important role in the success of any organisation.
I really like the concept of big data is equal to the sum of transaction , interaction and observations . Every telecom firm is using this concept they create an activity, interacts with the customers and then observe what is the reaction of customers. They use CRM systems which comprise of detailed information of every customer and this sytem helps to create BIG DATA
One problem is Big Data getting it wrong. My son frequently jumps on his friends' computers to purposefully perform searches on subjects that give rather annoying pop-up ads and movie suggestions. Sure, he does it to tease his friends, but it's not uncommon to let someone borrow your computer to do a little work and the next thing you know you're getting ads for a Judy Garland Convention weekend complete with commemorative plates and the entire Andy Hardy, DVD box set ("Hey kids, let's put on a show!").
The first explanation that I heard of the alleged value of big data was that somehow there would be searches discovering previously unknown relationships between trends. That is an interesting concept that could certainly have potential value. But determining a relationship, or even just an aapparent correlation, between happeningsof any kind and any other kind would certainly require both huge amounys of computational power and huge amounts of relevent data. I see two possible concerns immediately, which the first is "who owns that data?", and the second concern is about privacy of that data. Who has a right to data about what I eat for breakfast on Thursdays? Or how much coffee I drink? And I am certain that a whole lot of more personal data would be somehow tossed into that blob called Big Data. And who would benefit financially from the analysis of all that informnation about me? And who might just be amused to read my statistics?
My point is that a whole lot of the more obvious uses of big data certainly are not going to improve the quality of life for most folks, except for those who need advertising to tell them what they need to purchase and where they need to purchase it.
Really, it looks a whole lot like the future "benefits" from big data will be similar to blasting the lid off of Pandora's Box. And unfortunately for us all, once that happens it will be really hard to put the demons back in the bottle.
Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.