{"id":10288,"date":"2018-03-27T00:00:00","date_gmt":"2018-03-27T05:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/do-you-need-a-relational-data-warehouse\/"},"modified":"2023-11-29T12:07:55","modified_gmt":"2023-11-29T20:07:55","slug":"do-you-need-a-relational-data-warehouse","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/do-you-need-a-relational-data-warehouse\/","title":{"rendered":"Do You Need a Relational Data Warehouse?"},"content":{"rendered":"

Are you looking to do a major update to your data warehouse or looking to modernize? Many technologies have come about that are changing the landscape of what data warehouses<\/a> are made of. In this Azure Every Day session, I\u2019d like to talk about 3 technologies in Azure and HDInsight that break the rules.<\/span><\/p>\n

1. HDInsight (with Spark in particular)<\/strong> \u2013 You don\u2019t have to use SSIS to get ETL into your big data storage. HDInsight with Spark can give us ETL, as well as bring to light a lot of machine learning and other technologies, so we no longer have a dependency on SSIS.<\/p>\n

2. Azure Data Lake<\/strong> \u2013 This is a great place to store your data, and no, it won\u2019t cost you a lot of money and it\u2019s not hard to work with. Azure Data Lake gives you the ability to store all your data, regardless of where it comes from or how it looks, in that space \u2013 including real time data.<\/p>\n

Have you ever considered putting real time data into your data warehouse? Streaming data into your data warehouse would break everything we consider conventional. So, don\u2019t put it in a data warehouse, move it into a Hadoop structure in Azure Data Lake, and avoid the issue in a structured data warehouse of: I got it in there, how do I get it out?<\/p>\n

3. Interactive Hive and Spark SQL<\/strong> (also part of HDInsight) \u2013 The everyday report writers and users are not going to learn how to do map reduce or all those other technologies that make us the cool big data\/data science people. These users just need to build a report.<\/p>\n

These technologies bring us to a place where we can write SQL against those data structures and not care where the data came from, how it got there, or the type of data format it\u2019s in. We apply the schema after the fact and it means the same data that we put in our data lake, can be used in multiple scenarios.<\/p>\n