{"id":15656,"date":"2021-11-02T13:30:00","date_gmt":"2021-11-02T20:30:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/creating-efficient-power-bi-datasets-over-snowflake-3\/"},"modified":"2023-10-04T10:20:06","modified_gmt":"2023-10-04T17:20:06","slug":"creating-efficient-power-bi-datasets-over-snowflake","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/creating-efficient-power-bi-datasets-over-snowflake\/","title":{"rendered":"Creating Efficient Power BI Datasets Over Snowflake"},"content":{"rendered":"

For the past few months, we’ve been helping a global Financial Services company with their ongoing Snowflake implementation, which will ultimately sunset older, on-premises data marts in Oracle and SQL Server. Part of this engagement has included Power BI QuickStarts, where we’ve gotten to roll up our sleeves and put Power BI to work on top of Snowflake warehouses. As we partnered with the client’s various technology teams, we quickly realized that success lay less in implementing additional features into their environment, than in working to correct several modeling assumptions they were making within Power BI.<\/p>\n

Inspired by ideas presented in Microsoft and Snowflake\u2019s joint webinar<\/a>, and actual issues encountered during our client\u2019s QuickStarts, this post will aim to educate on how to create efficient Power BI solutions on top of Snowflake.<\/span><\/p>\n

\"iStock-665418054-1\"<\/span>Connecting to Snowflake<\/span><\/p>\n

The first step to building Power BI solutions on top of Snowflake is to connect to a Snowflake warehouse. This post assumes you already know how to use the native Snowflake connector in Power BI to connect to a warehouse. If this is a new concept, Microsoft\u2019s documentation has an easy-to-follow tutorial<\/a><\/span> on how to establish a connection from Power BI Desktop.<\/p>\n

Power BI also supports SSO access to Snowflake. While the topic of Snowflake SSO is out-of-scope for this post, this Snowflake documentation<\/a><\/span> has a thorough walkthrough on the process end-to-end. For Power BI Administrators, there is a tenant setting<\/span><\/a> you need to be aware of, and enable, if looking to leverage Snowflake SSO.<\/p>\n

<\/p>\n

Power BI Dataset Connectivity Modes<\/span><\/h2>\n

When working with Snowflake, or any relational source system<\/a><\/span>, it\u2019s important to understand what your options are for building semantic models on top of it in Power BI. Below, we\u2019ll introduce the three different types of dataset connectivity modes in Power BI<\/a><\/span> and provide general considerations for each.<\/p>\n

Import<\/span><\/h3>\n

Import mode is the most common mode we see when working with customers. With import mode, you ingest a copy of the source data into an in-memory dataset. Power BI\u2019s underlying storage engine (VertiPaq\/xVelocity) provides significant compression capabilities. A quick rule of thumb is that you can typically expect about 10x compression when importing data into Power BI. That said, there are several variables that can affect compression, which we\u2019ll talk more about below.<\/p>\n

\n\n\n\n\n\n\n
Import Mode<\/strong><\/span><\/td>\n<\/tr>\n
Pros<\/strong><\/span><\/td>\nCons<\/strong><\/span><\/td>\n<\/tr>\n
Typically, the fastest report and query performance due to the data being stored in-memory within Power BI.<\/td>\nData latency could be an issue since this is a cache of the source data. Use scheduled refreshes<\/a><\/span> to keep your dataset current.<\/td>\n<\/tr>\n
Full Power BI feature set is available.<\/td>\nLarge source data will need thoughtful up-front planning. Features like Power BI premium Large Dataset Storage<\/span><\/a> and Incremental Refresh<\/span><\/a> should be considered for importing large data volumes.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

<\/span><\/p>\n

Direct Query<\/span><\/h3>\n

With DirectQuery datasets, no data is imported into Power BI. Instead, your Power BI dataset is simply metadata (e.g., tables, relationships) of how your model is structured to query the data source. Data is only brought into Power BI reports and dashboards at query-time (e.g., a user runs a report, or changes filters).<\/p>\n\n\n\n\n\n\n\n
DirectQuery Mode<\/strong><\/span><\/td>\n<\/tr>\n
Pros<\/strong><\/span><\/td>\nCons<\/strong><\/span><\/td>\n<\/tr>\n
Dataset size limits do not apply as all data still resides in the underlying database.<\/td>\nTypically, query-time performance will suffer, even when querying a cloud data warehouse like Snowflake.<\/td>\n<\/tr>\n
Datasets do not require a scheduled refresh as data is only retrieved at query-time.<\/td>\nConcurrency (e.g., multiple users running reports) against DirectQuery datasets could cause performance issues against the underlying database.<\/td>\n<\/tr>\n
Near real-time reports can be developed by leveraging automatic page refresh<\/a> and Power BI Premium.<\/td>\nLimited Power Query and data modeling feature set.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Here’s a good reference for a thorough list of considerations<\/span><\/a> when using DirectQuery mode.<\/p>\n

<\/span><\/p>\n

Composite<\/p>\n

Composite models attempt to combine the best aspects from Import and DirectQuery modes into a single dataset. With Composite models, data modelers can configure the storage mode for each table in the model. A common design pattern for working with large, frequently-updated fact tables is to set the central fact table as DirectQuery mode, and set the surrounding dimension tables as Import or dual<\/span><\/a> mode. This pattern allows your contextual data points (dimension attributes), that are used for filtering and slicing-and-dicing, to be stored in memory.<\/p>\n

<\/p>\n

Common Mistakes<\/h2>\n

Direct Query all the Things!<\/h3>\n

The first common mistake we encountered in our QuickStarts was that developers only used, or wanted to use, DirectQuery mode. While DirectQuery mode isn\u2019t inherently bad, it should only be explored if you have specific requirements that make it a necessity. For example, extremely large data volumes, frequently changing data, or the need for Snowflake SSO<\/span><\/a>, are all candidates for DirectQuery.<\/p>\n

BFTs<\/span><\/h5>\n

The next most common mistake we encountered in our customers\u2019 datasets was what I call BFTs, or Big Fat\/Flat Tables. While it may be easy to get started with a BFT model (after all, it\u2019s only a single table), you will quickly realize the importance of a well-defined data model. We always recommend dimensional models<\/a>\/ Star Schemas for our customers because the benefits far outweigh the initial data shaping it will take to create one. We outline reasons you should use a star schema model in the Power BI Optimization Best Practices section below.<\/p>\n

Speaking of dimensional models, that brings us to our next common mistake\u2026<\/p>\n

Bidirectional Relationships Galore!<\/span><\/h5>\n

One technology team we partnered with had a great star schema design implemented in Snowflake. However, once that design was imported into Power BI, they broke it with bidirectional relationships. Because they were leveraging bidirectional relationships between each table, they were unable to model a constellation schema (multiple facts) because they kept running into the dreaded \u201cambiguity\u201d errors. This happens when the tabular engine has multiple paths, or directions, it can take to filter a table within the model, but no clear instruction on which one, so rather than guess, it doesn\u2019t evaluate at all. Since the relational model was already comprised of one-to-many relationships between dimensions and facts, simply changing all the relationships to a \u201csingle\u201d filter direction allowed them to consolidate several Power BI datasets into one. This dataset consolidation then allowed them to easily analyze different business processes by their conformed dimensions within the same reports.<\/p>\n

Size of Data > Number of Records<\/span><\/span><\/h5>\n

This mistake refers to a common statement we heard throughout our QuickStarts:
\n\u201cWe want to stress test Power BI with n million rows.\u201d<\/p>\n

These customer statements inevitably led to the typical consultant response:
\n\u201cWell, that depends on\u2026\u201d<\/p>\n

The number of records you can import into Power BI depends on several variables, including:<\/p>\n