{"id":10335,"date":"2020-09-04T00:00:00","date_gmt":"2020-09-04T05:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/how-to-run-a-databricks-notebook-using-azure-data-factory-2\/"},"modified":"2022-11-30T09:24:37","modified_gmt":"2022-11-30T15:24:37","slug":"how-to-run-a-databricks-notebook-using-azure-data-factory","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/how-to-run-a-databricks-notebook-using-azure-data-factory\/","title":{"rendered":"How to Run a Databricks Notebook Using Azure Data Factory"},"content":{"rendered":"<p style=\"font-size: 18px;\">In today\u2019s installment in our Azure Databricks mini-series, I\u2019ll cover <strong>running a Databricks notebook using Azure Data Factory (ADF)<\/strong>. With Databricks, you can run notebooks using different contexts; in my example, I\u2019ll be <strong>using Python<\/strong>.<\/p>\n<p style=\"font-size: 18px;\">To show how this works, I\u2019ll do a simple Databricks notebook run: I have a file on Azure Storage, and I\u2019ll read it into Databricks using Spark and then transform the data. <strong>Let\u2019s say you have something you\u2019re running, and you want to run it in an automated fashion, Databricks is a great way to incorporate that. If the end result is something you want to start productionizing, you can pull that into your ETL pipelines and run that notebook as part of your ETL solution.<\/strong><\/p>\n<p style=\"font-size: 18px;\">Some set up for my demo:<\/p>\n<ul style=\"font-size: 18px;\">\n<li>We need to have an Azure Storage Account and to connect Databricks to this, we\u2019ll need an Access Key which we\u2019ll store in Azure Key Vault.<\/li>\n<li>For our Databricks workspace, we\u2019re going to connect a Secret Scope to the Key Vault (a Preview feature) and mount that to an Azure Blob Storage container in Databricks using the Databricks file system.<\/li>\n<li>We will have an Azure Data Factory resource set up with the linked service to the Databricks workspace.<\/li>\n<\/ul>\n<p style=\"font-size: 18px;\">Once that is set up, <strong>my demo will show you how to create and run an ADF pipeline with a Databricks Notebook task<\/strong>. To see how easy this is, I\u2019ll walk you through it in my brief demo below.<\/p>\n<div class=\"hs-embed-wrapper\" style=\"position: relative; overflow: hidden; width: 100%; height: auto; padding: 0; max-width: 560px; max-height: 315px; min-width: 256px; display: block; margin: auto;\" data-service=\"youtube\" data-responsive=\"true\">\n<div class=\"hs-embed-content-wrapper\">\n<div style=\"position: relative; overflow: hidden; max-width: 100%; padding-bottom: 56.25%; margin: 0px;\"><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/X61myqUkb9o\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/div>\n<\/div>\n<\/div>\n<p style=\"font-size: 18px;\"><strong>If you want to discuss how to leverage Azure Databricks in your organization or have questions about any Azure product or service or Azure in general, reach out to us.<\/strong><\/p>\n<p><strong>Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or \u00a0<a href=\"mailto:sales@3cloudsolutions.com\">sales@3cloudsolutions.com<\/a>.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In today\u2019s installment in our Azure Databricks mini-series, I\u2019ll cover running a Databricks notebook using&mldr;<\/p>\n","protected":false},"author":37,"featured_media":10774,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10335","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/37"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10335"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10335\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/10774"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}