{"id":15636,"date":"2022-08-04T12:00:00","date_gmt":"2022-08-04T17:00:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/automating-your-power-bi-refreshes-synapse-edition-3\/"},"modified":"2023-10-04T15:51:15","modified_gmt":"2023-10-04T22:51:15","slug":"automating-your-power-bi-refreshes-synapse-edition","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/automating-your-power-bi-refreshes-synapse-edition\/","title":{"rendered":"Automating Your Power BI Refreshes: Synapse Edition"},"content":{"rendered":"<p>In <a href=\"\/blog\/automating-your-power-bi-refresh-history\" rel=\"noopener\">our previous Power BI automation blog<\/a>, we utilized the Power Platform in combination with Azure SQL DB to store refresh history of our datasets leveraging the Power BI REST API when on a Power BI shared capacity. A solution to manage Power BI dataset refreshes is particularly useful when data source refresh times are variable and end users need to know when dataset refreshes are successful. However, Power Automate and Azure SQL create two failure points. This solution also requires additional licensing cost in Power Automate to leverage premium data connectors to Azure SQL. So, how can we consolidate the number of tools we use and at the same time leverage the security features and access controls that Azure provides?<\/p>\n<p><span data-ccp-props=\"{\"><!--more--><\/span><span data-contrast=\"auto\"><img decoding=\"async\" style=\"width: 848px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/12\/Untitled-design65-1.png\" alt=\"Untitled design(65)\" width=\"848\" \/><br \/>\n<\/span><\/p>\n<h2><span style=\"color: #007cba;\"><strong>Azure Synapse Analytics Has Entered the Chat<\/strong><\/span><\/h2>\n<p><a href=\"\/blog\/azure-synapse-is-redrawing-the-modern-data-platform-landscape\" rel=\"noopener\">Azure Synapse Analytics<\/a> provides a unified framework for analytics, data integration, storage, and security all in one user interface. In this section of the blog, we will document how we can leverage <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/get-started-pipelines\" rel=\"noopener\">pipelines<\/a> in Synapse, <a href=\"\/blog\/azure-logic-apps-an-overview\" rel=\"noopener\">Logic Apps<\/a> for emailing our user base that refreshes have started and completed, and <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/key-vault\/\" rel=\"noopener\">Azure Key Vault<\/a> to securely store all our Power BI App Client ID and secrets.<\/p>\n<p><span style=\"color: #000000;\"><strong>Key Tools<\/strong><\/span><\/p>\n<ul>\n<li><strong>Azure Active Directory<\/strong>\n<ul>\n<li>Power BI App Registration with a Client ID\/Secret<\/li>\n<li>Security Group to Add App Registration Created<\/li>\n<\/ul>\n<\/li>\n<li><strong>Azure Synapse Workspace <\/strong>\n<ul>\n<li>To build our pipelines and create integration datasets to populate results to our data lake storage account<\/li>\n<\/ul>\n<\/li>\n<li><strong>Azure Logic Apps<\/strong>\n<ul>\n<li>To send emails to our business users that refreshes have started and completed<\/li>\n<\/ul>\n<\/li>\n<li><strong>Azure Key Vault<\/strong>\n<ul>\n<li>Securely store client ID and secrets<\/li>\n<li>Ensure your Synapse Workspace has a linked service and connection to Key Vault<\/li>\n<\/ul>\n<\/li>\n<li><strong>O365 Service Account<\/strong>\n<ul>\n<li>Dedicated Service Account to perform automation of emails in Logic Apps<\/li>\n<\/ul>\n<\/li>\n<li><strong>Power BI Tenant Settings<\/strong>\n<ul>\n<li>Enable Service Principals to use Power BI REST API<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"text-align: center;\"><img decoding=\"async\" style=\"margin-left: auto; margin-right: auto; display: block; width: 650px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-27-2022-03-45-49-33-PM.png\" width=\"650\" \/><span style=\"font-size: 12px;\"><em>Figure 1: Tenant Settings + Security Group Option<\/em><\/span><\/p>\n<p>&nbsp;<\/p>\n<p style=\"font-weight: normal; text-align: left;\"><span style=\"font-size: 12px;\"><span style=\"font-size: 17px;\"><span style=\"font-weight: bold;\">Note:<\/span> App Registration must be an Admin to all the Power BI Workspaces that we will be getting refresh data from.<\/span><br \/>\n<em><br \/>\n<\/em><\/span><\/p>\n<ul>\n<li><strong>Other<\/strong>\n<ul>\n<li>Knowledge of API testing and sending GET\/POST requests<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><span style=\"color: #000000;\"><strong>Logic Apps Workflow UI<\/strong><\/span><\/p>\n<p>We will create a base template for Logic Apps. This template will be called by Synapse via the HTTP Web request and parameters will be sent via Synapse to fill the user emails, subject line, and message.<\/p>\n<p><img decoding=\"async\" style=\"width: 650px; margin-left: auto; margin-right: auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/Picture1_censored1.jpg\" alt=\"Picture1_censored(1)\" width=\"650\" \/><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em><br \/>\nFigure 2: Logic Apps Template<\/em><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #000000;\"><strong>Azure Synapse: Integration Datasets + Data Lake Architecture<\/strong><\/span><\/p>\n<p>Before we review the pipelines set up, we need to configure the following integration datasets and linked services.<\/p>\n<ul>\n<li><strong>Linked Services Needed<\/strong>\n<ul>\n<li>Key Vault<\/li>\n<li>REST API<\/li>\n<\/ul>\n<\/li>\n<li><strong>Integration Datasets<\/strong>\n<ul>\n<li><strong>ds_PowerBIResponse<\/strong>: To leverage the Power BI REST API<\/li>\n<li><strong>ds_Parameters<\/strong>: To access our csv file of our parameters that we will be looping over to refresh<\/li>\n<\/ul>\n<ul>\n<li><strong>ds_RefreshHistory<\/strong>: To provide a sink point to take our results and store in our data lake<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 450px; margin-left: auto; margin-right: auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-27-2022-03-56-52-08-PM.png\" width=\"450\" \/><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 3: Integration Datasets<\/em><\/p>\n<p>Our Data Lake folder set up will have the following items.<\/p>\n<ul>\n<li>Our Parameter Table with our Power BI Datasets and Workspace Information (GUIDs)<\/li>\n<li>A Refresh History Table where we will drop our storage information\n<ul>\n<li>Will have a nested folder structure based on time of data refresh<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 750px; margin-left: auto; margin-right: auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-27-2022-04-02-41-36-PM.png\" width=\"750\" \/><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 4: Synapse Storage Setup<\/em><\/p>\n<p style=\"text-align: justify;\"><span style=\"color: #000000;\"><strong>Azure Synapse: Master Pipeline Overview<\/strong><\/span><\/p>\n<p style=\"text-align: justify;\">This pipeline below is the master pipeline that will kick start our Power BI Refreshes and checking refresh status. In order, it does the following:<\/p>\n<ol>\n<li style=\"text-align: justify;\">Grabs our secret information from Azure Key Vault<\/li>\n<li style=\"text-align: justify;\">Gets an AAD Token by calling the Power BI REST API<\/li>\n<li style=\"text-align: justify;\">Gets our dataset GUID parameters from our flat file that houses the datasets we will need to refresh\n<ol style=\"list-style-type: lower-alpha;\">\n<li style=\"text-align: justify;\">From here, we will start the refresh process for all the datasets and trigger the Logic Apps email to let us know the refreshes have started<\/li>\n<li style=\"text-align: justify;\">Datasets will be refreshed in parallel via the for loop. The check status pipeline will be called within this loop to ensure proper timings before sending refresh completion<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<ol>\n<li style=\"text-align: justify;\"><\/li>\n<\/ol>\n<p><img decoding=\"async\" style=\"width: 750px; margin-left: auto; margin-right: auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-26-33-82-PM.png\" width=\"750\" \/><\/p>\n<p style=\"text-align: center; font-size: 14px;\"><em><span style=\"font-size: 12px;\">Figure 5: Main Master Pipeline<\/span><\/em><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center; font-size: 14px;\"><em><img decoding=\"async\" style=\"width: 750px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-27-14-11-PM.png\" width=\"750\" \/><\/em><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 6: Loop Step to Call Power BI Refresh and Call Pipeline to Check Status<\/em><\/p>\n<p style=\"text-align: center; font-size: 14px;\"><em><img decoding=\"async\" style=\"margin-left: auto; margin-right: auto; display: block; width: 750px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/undefined-Jul-29-2022-02-28-22-73-PM.png\" width=\"750\" \/><\/em><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 7: Email Trigger that Refreshes Have Started<\/em><\/p>\n<p style=\"text-align: justify;\"><span style=\"color: #000000;\"><strong>Azure Synapse: Check Status Pipeline Overview<\/strong><\/span><\/p>\n<p>Below is the check status pipeline. At a high level, this pipeline is called multiple times per dataset that is needed to be refresh and will run until all datasets either complete or fail. Some things to note are the following:<\/p>\n<ul>\n<li>The Reset Token step provides a failsafe when a refresh takes more than a certain amount of time to complete. The Power BI REST API token by default, is only available for 1 hour and this step allows us to get a new token once we are close to that threshold<\/li>\n<li>The Until Loop will stop only when the dataset refresh status something that is not in progress. Once it is not in progress, we will log the results as a parquet file in our data lake storage account<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" style=\"margin-left: auto; margin-right: auto; display: block; width: 500px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-31-26-05-PM.png\" width=\"500\" \/><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 8: Check Status Pipeline Main Level<\/em><\/p>\n<p><span style=\"color: #007cba;\"><img decoding=\"async\" style=\"width: 750px; margin-left: auto; margin-right: auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-34-24-54-PM.png\" width=\"750\" \/><\/span><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 9: Check Status Pipeline Until Loop<\/em><\/p>\n<p>&nbsp;<\/p>\n<h2 style=\"font-size: 14px; text-align: center;\"><span style=\"color: #007cba;\"><em>\u00a0<\/em><\/span><\/h2>\n<h2><span style=\"color: #007cba;\"><img decoding=\"async\" style=\"margin-left: auto; margin-right: auto; display: block; width: 750px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-36-03-97-PM.png\" width=\"750\" \/><\/span><\/h2>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 10: Refresh History Logging in ALDS Gen 2<\/em><\/p>\n<p>Once all the dataset refreshes are done, we will get an email that tells us the refresh is completed from our master pipeline.<\/p>\n<p style=\"text-align: center;\"><img decoding=\"async\" style=\"margin-left: auto; margin-right: auto; display: block; width: 750px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/image-png-Jul-29-2022-02-37-17-41-PM.png\" width=\"750\" \/><\/p>\n<p style=\"text-align: center; font-size: 12px;\"><em>Figure 11: Complete Email from Logic Apps<\/em><\/p>\n<h2><span style=\"color: #007cba;\">Conclusion and Extensions<\/span><\/h2>\n<p>Using Azure Synapse Analytics and Logic Apps, we were able to consolidate and reduce the number of services required with Azure Synapse\u2019s unified framework. Data storage, and pipelining was done entirely in the Synapse environment, and we leveraged Logic Apps to orchestrate sending emails to our end users. Azure storage provides an inexpensive alternative to leveraging a standalone database by storing all our refresh history in our default Azure Data Lake Gen 2 Storage provided by Synapse.<\/p>\n<p>Some extensions are also possible and are outlined below<\/p>\n<ul>\n<li>Pulling history information from other artifacts like dataflows<\/li>\n<li>Pulling and logging other information from the PowerBI REST API<\/li>\n<li>Adding this pipeline to a broader ETL pipeline in Synapse. Once ETL is done this pipeline can be used to refresh datasets downstream<\/li>\n<li>Creating auditing reports to monitor PowerBI refresh performance<\/li>\n<\/ul>\n<h2><span style=\"color: #007cba;\">Connect with us!<strong><br \/>\n<\/strong><\/span><\/h2>\n<p><span data-contrast=\"auto\">Please reach out to us today to find out more about how BlueGranite\u2019s talented team can help you bring clarity and efficiency to your Modern Analytics endeavors today!<\/span><span data-ccp-props=\"{\">\u00a0<\/span><\/p>\n<p>BlueGranite offers a variety of <a href=\"https:\/\/www.bluegranite.com\/resources\" target=\"_blank\" rel=\"noopener\">resources<\/a> and <a href=\"https:\/\/www.bluegranite.com\/events\" target=\"_blank\" rel=\"noopener\">free training events<\/a> to help you learn how you can leverage Modern Data Analytics.<\/p>\n<p>Please visit our <a href=\"https:\/\/www.bluegranite.com\/events\" target=\"_blank\" rel=\"noopener\">website<\/a> and <a href=\"https:\/\/www.youtube.com\/c\/BlueGraniteInc\" rel=\"noopener\">YouTube Channel<\/a> to learn more, or <a href=\"https:\/\/www.bluegranite.com\/contact-us\" target=\"_blank\" rel=\"noopener\">contact us<\/a> directly to see how we can help you explore your about modern data analytics options and accelerate your business value.<\/p>\n<h2><span style=\"color: #007cba;\">About the Authors<\/span><\/h2>\n<p><span data-contrast=\"auto\"><span style=\"font-weight: bold;\"><img decoding=\"async\" style=\"width: 200px; float: left; margin: 0px 20px 20px 0px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/NithinSankar_Circle.png\" alt=\"NithinSankar_Circle\" width=\"200\" \/><\/span><\/span><span data-contrast=\"auto\"><span style=\"font-weight: bold;\">Nithin Sankar<\/span><br \/>\nNithin is a Solution Consultant at 3Cloud with experience working with stakeholders in the healthcare, hospitality, and tourism industries to design and develop robust data-driven solutions across different service lines and business units. Nithin holds a B.S. in Industrial &amp; Systems Engineering from the University of Florida and is currently pursuing a M.S. in Analytics from the Georgia Institute of Technology. <a href=\"https:\/\/www.linkedin.com\/in\/nithin-s\/\" target=\"_blank\" rel=\"noopener\">Check him out on LinkedIn<\/a>!<br \/>\n<\/span><\/p>\n<p><span data-contrast=\"auto\"><span style=\"font-weight: bold;\"><img decoding=\"async\" style=\"width: 200px; float: left; margin: 21px 20px 15px 0px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/10\/CallenSmerker.png\" alt=\"CallenSmerker\" width=\"200\" \/>Callen Smerker<\/span><br \/>\nCallen is a Solution Architect at 3Cloud who, since 2014, has had a variety of experience in database and business intelligence solutions. This includes leading large, complex projects as both Data Architect and Project Management Professional. He enjoys designing solutions within the Microsoft BI stack (SQL Server, SSIS, SSAS, and PowerBI) in combination with Azure DevOps CI\/CD pipelines to deliver enterprise grade analytics solutions. Callen has experience working in a variety of industries including healthcare, life science, pharmaceutical, and logistics. <a href=\"https:\/\/www.linkedin.com\/in\/callen-smerker-a2527090\/\" target=\"_blank\" rel=\"noopener\">Check him out on LinkedIn<\/a>!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In our previous Power BI automation blog, we utilized the Power Platform in combination with Azure SQL to store refresh history of our datasets leveraging the Power BI REST API when on a Power BI shared capacity. A solution to manage Power BI dataset refreshes is particularly useful when data source refresh times are variable and end users need to know when dataset refreshes are successful. However, Power Automate and Azure SQL create two failure points. This solution also requires additional licensing cost in Power Automate to leverage premium data connectors to Azure SQL. So, how can we consolidate the number of tools we use and at the same time leverage the security features and access controls that Azure provides?<\/p>\n","protected":false},"author":21,"featured_media":12116,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[394,260],"tags":[303,304,273],"class_list":["post-15636","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business-intelligence","category-data-ai","tag-modern-analytics","tag-modern-data-platform","tag-power-bi","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15636"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15636\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/12116"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}