Microsoft Hive ODBC Driver<\/a>, you can import entire Hive tables into Power BI or write Hive queries to import data directly into Power BI.<\/p>\n![\"hiveodbc.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/hiveodbc.png\")
<\/p>\n
Importing from Hive tables\/queries provides a familiar SQL-like feel and syntax, and you are able to use the whole\u00a0gamut of modeling capabilities within Power BI Desktop. \u00a0It is important to understand, however, that data refreshes with this method can often be slow as a Hive job will be executed on your cluster before transferring the data. \u00a0Being that this method imports data directly into the Power BI model, you may also be limited in the size of the data that you can work with, as the model will have to fit into the memory of your machine and\/or the file size limits of the Power BI service. \u00a0Your cluster will also need to be up and running at the time of a Power BI refresh so that Power BI can see\/access any Hive objects.<\/p>\n
Refresh from Flat Files<\/h2>\n
![\"diag-File.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/diag-File.png\")
<\/p>\n
Another way to import data from HDInsight into Power BI is by connecting to flat files in either Blob or the Data Lake Store. \u00a0In this scenario, you would use HDInsight to process your data\u00a0and write the resulting curated or aggregated data into text files (CSV, TAB, etc.)\u00a0\u00a0Unlike refreshing from Hive tables, refreshing from flat files would allow you to only run your HDInsight cluster while processing data (deleting a\u00a0cluster when it is not active can help save money in Azure consumption) as the resulting text files would still exist in Azure storage even when the cluster is deleted.<\/p>\n
![\"flatfiles-1.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/flatfiles-1.png\")
<\/p>\n
Data refreshes into Power BI are also likely to be faster when pulling directly from flat files versus using the Hive ODBC Driver referenced above. \u00a0You will still run into the same model size restrictions as when using Hive tables\/queries, as you are still importing data directly into a Power BI model.<\/p>\n
The previous\u00a0two methods for using Power BI with HDInsight required importing data into a Power BI model, thus limiting the size of your model to the constraints of your computer or the Power BI Service. The next\u00a0two methods will utilize DirectQuery in which your data will stay at the data source, thus removing the model size limitations discussed before.<\/p>\n
DirectQuery with Spark<\/h2>\n
![\"diag-Spark-1.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/diag-Spark-1.png\")
<\/p>\n
The first method is connecting directly to tables in a Spark cluster. \u00a0With this method, you will process your data\u00a0in Spark and then put the resulting data into tables on the cluster. Power BI can then use\u00a0Spark SQL to interactively query the tables in Power BI’s DirectQuery mode.<\/p>\n
![\"spark.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/spark.png\")
<\/p>\n
Again, the key advantage here is that the data stays at the source, which removes the need to schedule Power BI refreshes and worry about Power BI model size. \u00a0A downside to this approach is that in order for Power BI to connect to a table in Spark, the cluster must be running, which likely means the cluster is on all the time (depending on when your users will be using the reports\/model). \u00a0This can sometimes be more expensive than only turning your cluster on for processing data and deleting it when you are done (as\u00a0with the flat files\u00a0method).<\/p>\n
DirectQuery from Azure SQL DB<\/h2>\n
![\"diag-ASQL.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/diag-ASQL.png\")
<\/p>\n
The final method\u00a0we will look at is DirectQuery using Azure SQL Database (DB). \u00a0In this approach, similar to the flat files approach, you would process your data in your cluster, but write the resulting curated and\/or aggregated data to tables in Azure SQL DB (or Azure SQL Data Warehouse).<\/p>\n
![\"azsqldb.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/azsqldb.png\")
<\/p>\n
As with the flat files approach, this allows you to delete your cluster when you are done processing data (which helps control costs) as the processed data will still reside in the SQL DB when the cluster is gone. \u00a0However, unlike the flat files approach, Azure SQL DB can be highly\u00a0optimized and is well suited for DirectQuery in Power BI. \u00a0This means that you will not have to worry about data refreshes at the Power BI level. \u00a0When connected to Azure SQL DB with Power BI using DirectQuery, you also still have a lot of additional modeling capabilities included in Power BI, like creating new measures, calculated columns, and relationships between tables. \u00a0The downside to this approach is that you will need to set up and configure an Azure SQL DB service, and that service will likely need to run all the\u00a0time (depending on when your users will be using the reports\/model). \u00a0Also, while DirectQuery virtually removes any limitations on the Power BI model size, you still have to be concious of the size, performance, and cost of your Azure SQL DB instance.<\/p>\n
While there are likely other ways to work with HDInsight using Power BI, these are some of the more common scenarios we have run into and deployed. \u00a0Hopefully this helps in planning your architecture for visualizing your big data in the cloud.<\/p>\n","protected":false},"excerpt":{"rendered":"
Outline of design patterns for implementing a big data cloud solution with Microsoft Power BI and Azure HDInsight.<\/p>\n","protected":false},"author":21,"featured_media":14782,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[343,305,304],"class_list":["post-15988","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-hdinsight","tag-modern-bi","tag-modern-data-platform","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15988","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15988"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15988\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/14782"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15988"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15988"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"diag-Hive.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/diag-Hive.png\")
<\/p>\n