{"id":15701,"date":"2021-02-01T14:40:41","date_gmt":"2021-02-01T22:40:41","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/power-bi-for-government-agencies-connect-and-store-data-part-2-3\/"},"modified":"2024-04-17T08:39:49","modified_gmt":"2024-04-17T15:39:49","slug":"power-bi-for-government-agencies-connect-and-store-data-part","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/power-bi-for-government-agencies-connect-and-store-data-part\/","title":{"rendered":"Power BI For Government Agencies &#8211; Connect and Store Data (Part 2)"},"content":{"rendered":"<p>In this series, learn how 3Cloud is helping a large local government with user and data security as they modernize operational reporting using Power BI.<\/p>\n<p><span style=\"font-size: 12.0pt;\">3Cloud recently worked with public safety agencies that are a part of a large, combined City-County government, focusing on modernizing systems, while also keeping data secure within and across these agencies. This project helped securely develop and distribute sensitive data and reports while also monitoring and auditing Power BI usage to meet internal, state, and federal data <\/span><span style=\"font-size: 12.0pt;\">protection requirements.<\/span><!--more--><\/p>\n<p><img decoding=\"async\" style=\"width: 1200px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/12\/Blog_Power-BI-For-Government-Agencies-Connect-and-Store-Data-Part-2.png\" alt=\"Blog_Power BI For Government Agencies - Connect and Store Data (Part 2)\" width=\"1200\" \/><\/p>\n<h2 aria-level=\"1\"><span style=\"color: #007cba;\"><br \/>\nChallenges\u00a0<\/span><\/h2>\n<p><span style=\"font-size: 12.0pt;\">All levels of government struggle to manage data quality, which can often make reporting challenging as end-users question the accuracy and usability of the information. Specific to departments of public safety, there are issues and questions related to security and data auditing as they consider moving to cloud computing. <\/span><\/p>\n<ul>\n<li><span style=\"font-size: 12.0pt;\">Will\u00a0the\u00a0agency\u2019s\u00a0data be stored\u00a0outside the agency databases?\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">What if the agency or department has a policy against having data in the cloud?\u00a0\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">With data\u00a0spanning dozens of departments\u00a0and tens of thousands of users,\u00a0what if the\u00a0data\u00a0is\u00a0too large to move\u00a0or\u00a0fit in a reporting tool?\u00a0\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">Will the data in the reports\u00a0reflect real-time updates\u00a0that occur in the database?\u00a0\u00a0<\/span><\/li>\n<\/ul>\n<p><span style=\"font-size: 12.0pt;\">These questions are central concerns for incorporating a reporting\u00a0solution\u00a0into an agency.\u00a0The potential for moving to a tool like <a href=\"https:\/\/www.bluegranite.com\/modern-bi\">Power BI<\/a> means giving end-users the power to create and share their own reports and easily filter and drill down into information to get a better understanding about the key pieces of information that can drive decision-making. The alternative is emailing PDFs or spreadsheets, which are almost immediately out of date and difficult to track. <\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">The good news is that Power BI is able to comply with all of the questions asked above. What follows is a technical walk-through of the steps 3Cloud took in this project. If you are interested in more information about how Power BI can work for your organization, <\/span><a href=\"https:\/\/www.bluegranite.com\/resource-center#modern-bi\"><span style=\"font-size: 12.0pt;\">check out some of our resources<\/span><\/a><span style=\"font-size: 12.0pt;\">.<\/span><\/p>\n<p>&nbsp;<\/p>\n<h2 aria-level=\"1\"><span style=\"color: #007cba;\" data-contrast=\"none\">Solutions<\/span><span data-ccp-props=\"{\">\u00a0<\/span><\/h2>\n<p><span style=\"font-size: 12.0pt;\">Concerns about Power BI data storage and the cloud are largely answered by one question: How does Power BI connect to the data? Power BI has two primary options for connecting to data: DirectQuery and Import (and a third hybrid option called \u201ccomposite model\u201d, but we will set that aside for now). Of course, there are many nuances to how those connection methods work, but we\u2019re going to just focus on the ones relevant to the data storage concerns of government agencies. In brief, here\u2019s how they both work:\u00a0<\/span><\/p>\n<p><strong><span style=\"font-size: 12.0pt;\">Import: <\/span><\/strong><span style=\"font-size: 12.0pt;\">When the data source is set to \u201cImport,\u201d the data will be imported and stored in the .pbix file. Because the data is stored in the file, when that file is published to the Power BI Service (i.e., the cloud), the data follows.\u00a0<\/span><\/p>\n<p><strong><span style=\"font-size: 12.0pt;\">DirectQuery<\/span><\/strong><span style=\"font-size: 12.0pt;\">: Setting\u00a0the\u00a0data source to\u00a0\u201cDirectQuery\u201d\u00a0only creates the connection between the Power BI data model and the data in\u00a0the\u00a0source. It does\u00a0<strong>not<\/strong>\u00a0store the data in the file.\u00a0Based on those simple definitions, it looks like\u00a0DirectQuery\u00a0is the way to go for mitigating data in the cloud.\u00a0But there are a few questions that need answering before switching all connections to\u00a0DirectQuery.\u00a0<\/span><\/p>\n<ul>\n<li><span style=\"font-size: 12.0pt;\">What are the pros and cons of\u00a0DirectQuery?\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">How does the connection method influence\u00a0report\u00a0development process? <\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">What about report performance?\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">Can\u00a0setting the connections to\u00a0DirectQuery\u00a0really\u00a0avoid sending\u00a0all\u00a0data to the cloud?\u00a0<\/span><\/li>\n<\/ul>\n<p><span style=\"font-size: 12.0pt;\">Let\u2019s start with the last one:\u00a0Does\u00a0DirectQuery\u00a0prevent\u00a0data\u00a0from being stored\u00a0in the cloud?\u00a0For the most part,\u00a0yes.\u00a0When\u00a0DirectQuery\u00a0is used to connect to the data source,\u00a0only\u00a0the connection information is stored in the file, so when published\u00a0to\u00a0the\u00a0cloud the\u00a0data\u00a0is not contained in the file.\u00a0While the data itself\u00a0is not\u00a0published to the cloud, the\u00a0<em>meta<\/em>data is included in the file. Things like table names, schemas, and measure names are stored in the file. The metadata is a necessary part of the connection details, and even DirectQuery cannot minimize that part of the data being sent to the cloud.<br \/>\n<\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">Now that we have uncovered that the Power BI Service can accommodate an on-premises data requirement, there are a couple of other key considerations. Unlike an Imported data set, which issues queries against a cached dataset in-memory, DirectQuery sends a query back to the data source each time a visual is refreshed, filtered, etc. As one might expect, an imported dataset can perform much faster than DirectQuery. However, DirectQuery performance is highly dependent on network speed and data source configurations, so there are plenty of options for tweaking to optimize performance.\u00a0\u00a0<\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">By\u00a0transitioning\u00a0to\u00a0DirectQuery\u00a0in order to manage data storage requirements,\u00a0it\u2019s important to know that\u00a0certain features of the\u00a0data modeling capabilities in Power Query\u00a0are limited.\u00a0Report development teams\u00a0will\u00a0need to be aware of how their development practice\u00a0&#8211; specifically around data modeling &#8211;\u00a0will change with\u00a0DirectQuery.\u00a0<\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">Now let\u2019s dig into the details of how to implement\u00a0DirectQuery.<\/p>\n<p><\/span><\/p>\n<h2 aria-level=\"1\"><span style=\"color: #007cba;\">Implementation\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">F<\/span><span style=\"font-size: 12.0pt;\">rom our experience working with government agencies, we know\u00a0that\u00a0oftentimes\u00a0they\u00a0have policies or\u00a0preferences around storing data in the cloud.\u00a0Using the cloud-based\u00a0Power BI Service doesn\u2019t have to conflict with\u00a0an\u00a0on-premises\u00a0requirement if you utilize\u00a0DirectQuery\u00a0for your data sources. Let\u2019s look at how\u00a0to\u00a0go about implementing that\u00a0DirectQuery\u00a0for government agency data reporting needs.\u00a0<\/span><\/p>\n<p aria-level=\"3\"><strong>Configure your data source<\/strong><span data-ccp-props=\"{\"><strong>\u00a0<\/strong><br \/>\n<\/span><span style=\"font-size: 12pt; font-weight: normal;\">Setting up\u00a0DirectQuery\u00a0starts\u00a0during the initial connection set up in Power BI Desktop.\u00a0As mentioned above,\u00a0when selecting \u201cGet Data\u201d and choosing a database option,\u00a0there are\u00a0two different Data Connectivity modes.\u00a0\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{\"> <img decoding=\"async\" class=\"WACImage SCXW256944148 BCX7\" style=\"margin: 1px auto; padding: 2px; user-select: text; white-space: pre !important; vertical-align: baseline; color: #000000; font-family: 'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; width: 464px; height: 222px; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/blob-310.png\" \/><\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">After the data is connected via DirectQuery, there will be the opportunity to model and transform the data in the Power Query editor. Features in the Power Query editor are limited with DirectQuery and care is required or there is the potential to unintentionally undo the DirectQuery setup. When executing many of the transformations a warning will display: \u201cThis step results in a query that is not supported in DirectQuery mode\u201d with the option to switch the tables to import mode.\u00a0<strong>Stop right there!\u00a0<\/strong><\/span><\/p>\n<p><span data-ccp-props=\"{\"> <img decoding=\"async\" class=\"WACImage SCXW46358344 BCX7\" style=\"margin: 1px auto; padding: 2px; user-select: text; white-space: pre !important; vertical-align: baseline; color: #000000; font-family: 'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; width: 662px; height: 122px; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/blob-311.png\" width=\"662\" \/><\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">Switching to import mode will negate the efforts of not storing data in the cloud. Instead, it\u2019s best to limit which transformations are performed in Power Query and make the transformations back into the source system.\u00a0<\/span><\/p>\n<p aria-level=\"3\"><strong>Checking\u00a0file size<\/strong><span data-ccp-props=\"{\"><strong>\u00a0<\/strong><br \/>\n<\/span><span style=\"font-size: 12.0pt;\">Once\u00a0the\u00a0data is modeled with\u00a0DirectQuery\u00a0(and the rest of\u00a0the\u00a0report is built),\u00a0the\u00a0file size will be\u00a0substantially smaller when only storing\u00a0the metadata instead of importing the data.\u00a0A\u00a0small file size is a good indicator\u00a0that\u00a0DirectQuery\u00a0is set up\u00a0successfully.\u00a0\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{\"> <img decoding=\"async\" class=\"WACImage SCXW155206210 BCX7\" style=\"margin: 1px auto; padding: 2px; user-select: text; white-space: pre !important; vertical-align: baseline; color: #000000; font-family: 'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; width: 670px; height: 88px; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/blob-312.png\" width=\"670\" \/><\/span><\/p>\n<p aria-level=\"3\"><strong>Publishing to\u00a0the\u00a0Power BI Service<\/strong><span data-ccp-props=\"{\"><br \/>\n<\/span><span style=\"font-size: 12.0pt;\">Now comes the time to publish the report \u2013 but not the data \u2013 to the cloud service. From Power BI Desktop,\u00a0developers can\u00a0publish\u00a0the\u00a0reports\u00a0to a Workspace just like\u00a0they\u00a0would for any other report. However, when\u00a0viewing\u00a0the report\u00a0in the Power BI\u00a0Service, a notice will display across the top.\u00a0\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{\"> <img decoding=\"async\" class=\"WACImage SCXW75912987 BCX7\" style=\"margin: 1px auto; padding: 2px; user-select: text; white-space: pre !important; vertical-align: baseline; color: #000000; font-family: 'Segoe UI', 'Segoe UI Web', Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; width: 634px; height: 93px; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/blob-313.png\" width=\"634\" \/><\/span><\/p>\n<p><span style=\"font-size: 12.0pt;\">For\u00a0the\u00a0data to live on-premises and\u00a0be available to display through\u00a0DirectQuery\u00a0in the Power BI Service,\u00a0the data needs to go through a <a href=\"https:\/\/www.bluegranite.com\/blog\/monitoring-power-bi-on-premises-data-gateway-performance\">Gateway<\/a>.\u00a0A Gateway is a piece of software\u00a0installed on a server\u00a0that\u00a0connects\u00a0the\u00a0on-premises data source with the cloud service.\u00a0After the gateway\u00a0is configured with\u00a0credentials\u00a0for\u00a0the\u00a0data source and Power BI account,\u00a0published reports\u00a0can\u00a0directly query on-premises data from the cloud.\u00a0<\/span><\/p>\n<p aria-level=\"3\"><strong>Implementation Results<\/strong><span data-ccp-props=\"{\"><strong>\u00a0<\/strong><br \/>\n<\/span><span style=\"font-size: 12.0pt;\">Once\u00a0reports with\u00a0DirectQuery\u00a0configured are published to the Power BI Service, a\u00a0few\u00a0key results are available.\u00a0<\/span><\/p>\n<ul>\n<li><span style=\"font-size: 12.0pt;\">Near Real-Time Data\u00a0is available.\u00a0One advantage of the\u00a0DirectQuery\u00a0data source\u00a0is that the report data is\u00a0kept in sync with your data source.\u00a0DirectQuery\u00a0issues a query against the data source\u00a0and returns the most recent version of your data source.\u00a0Imported data sets require configured\u00a0scheduled\u00a0refreshes\u00a0for\u00a0the\u00a0report\u00a0data\u00a0to\u00a0reflect\u00a0the data source. With the Gateway up and running, a\u00a0DirectQuery\u00a0report always has the most up to date data.\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">Data size has (nearly) no limits.\u00a0Rather than facing the size limits imposed for data imported into a\u00a0Power BI model,\u00a0DirectQuery\u00a0provides access to nearly unlimited data volumes.\u00a0The space required to store the dataset metadata is\u00a0minimal and\u00a0removes the storage space ceiling in the Power BI Service.\u00a0\u00a0<\/span><\/li>\n<li><span style=\"font-size: 12.0pt;\">Data isn\u2019t stored in the cloud.\u00a0This, of course, is an important requirement for government agencies. The data remains in the\u00a0on-premises or existing secure data infrastructure.\u00a0\u00a0<\/span><\/li>\n<\/ul>\n<p style=\"line-height: 1;\"><span data-ccp-props=\"{\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><span style=\"color: #007cba;\" data-contrast=\"none\">Summary<\/span><span data-ccp-props=\"{\">\u00a0<\/span><\/h2>\n<p><span style=\"font-size: 12.0pt;\">A government agency that is moving to Power BI from a legacy system, or trying to increase adoption of the platform, must face the handle the questions of how to connect and store data. Power BI offers great flexibility for agencies that are cloud adverse, cloud-friendly, or somewhere in between. It can scale to accommodate massive datasets but is also nimble enough for departmental data marts.\u00a0<\/span><\/p>\n<p style=\"line-height: 1;\"><span data-ccp-props=\"{\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><span style=\"color: #007cba;\" data-contrast=\"none\">Learn More<\/span><span data-ccp-props=\"{\">\u00a0<\/span><\/h2>\n<p>Follow these links to learn more about how 3Cloud has helped other municipalities \u2013 including evolving education strategies, fostering traffic safety, and ensuring water quality. Or take a moment to check out our Modern Data Platform and Modern BI pages to discover how the right advanced technology can transform outcomes.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this series, learn how 3Cloud is helping a large local government with user and data security as they modernize operational reporting using Power BI.<\/p>\n","protected":false},"author":21,"featured_media":12728,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[305,316],"class_list":["post-15701","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-modern-bi","tag-public-sector","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15701","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15701"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15701\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/12728"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15701"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15701"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}