{"id":16104,"date":"2014-07-18T20:06:00","date_gmt":"2014-07-19T03:06:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/want-to-get-started-with-big-data-stop-throwing-data-away-2\/"},"modified":"2024-04-17T14:05:23","modified_gmt":"2024-04-17T21:05:23","slug":"want-to-get-started-with-big-data-stop-throwing-data-away","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/want-to-get-started-with-big-data-stop-throwing-data-away\/","title":{"rendered":"Want to get Started with Big Data? Stop Throwing Data Away!"},"content":{"rendered":"
\n

\"DoRecently I\u2019ve been talking with a number of customers with similar problems. They have systems that are generating high volumes of transaction-level data that they are unable to collect and store in order to use it for analysis. For example, most manufacturers have sensors on their production lines. Every piece of equipment has one or more sensors that is collecting data about the operation. Information like tolerances, temperatures and cycle times all give valuable insight into how the process is going. Very often, these systems are measuring multiple times a minute or even second. This data is used to keep track of processes as they happen but since the data volume is so high, it is either archived or discarded after it is a week or even just a day old. Because the data volume is so high, it\u2019s not really feasible to keep around 1, 5 or 10 years\u2019 worth of it. But, if you had that deep history, imagine how much insight you could gain into your processes. The ability to do trend analysis to detect anomalies that might be missed in a span of days but stick out over months could indicate places where you could improve processes and quality.<\/p>\n

<\/p>\n

Manufacturing is one example but many other types of businesses have data that they regularly throw away or ignore because they don\u2019t think it\u2019s relevant to business operations or there is too much of it in a format that is hard to deal with. Web server and other software logs usually accumulate until someone decides to either delete them or archive them. How about survey results?\u00a0Companies do lots of surveys and then act on the outcome. Do you keep the data accessible so that you can combine past results with new ones and\u00a0analyze them over time? Do you cross-compare them with your sales results? How about clicks, tweets and likes? Do you have customer comments or product reviews on your web site? Can you correlate them over time and show trends in sentiment toward your products and demonstrate how they impact sales?<\/p>\n

The promise of the big data<\/a> is that it will give you access to all of your data no matter what the volume, source or format. But many companies are saying things like \u201cwe want to look into big data but we don\u2019t have time\/resources\/money right now\u201d. OK, so you\u2019re going to do big data in six months. You still don\u2019t have to throw all that valuable data away between now and then. Those log files, process streams and social feeds have value. Don\u2019t just arbitrarily say \u201cwe\u2019ll get to it when we can\u201d. Start collecting that data so that when you are ready to try some analysis on it, you\u2019ll already have built up a stockpile.<\/p>\n

\u201cBut wait!\u201d I hear you saying. \u201cIT told me it\u2019s too expensive to keep all that data around! They told me all that disk space costs too much!\u201d IT has a point. Disk space in the data center is limited and it can be costly to add more. One great way to mitigate that is to utilize cloud storage like Azure Blob Storage. If you\u2019re new to the concept of cloud<\/a> beyond the buzzword, it really just means computing services that are hosted by a service provider and made available over the Internet. With cloud-based services, you don\u2019t have to buy any infrastructure or software, you just pay a monthly fee. Microsoft Azure<\/a> is Microsoft\u2019s umbrella for their cloud-based offerings one of which includes data storage. The term \u201cBlob<\/a>\u201d (written correctly it\u2019s BLOb) is an acronym for Binary Large Object. It just means a collection of binary data that is stored together. So, put that all together and Azure Blob Storage is Microsoft\u2019s cloud offering for storing large blocks of data. Whew!<\/p>\n

Cloud storage has a lot of advantages over storage in the data center:<\/p>\n