{"id":10206,"date":"2021-01-22T16:33:12","date_gmt":"2021-01-22T22:33:12","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/azure-data-factory-101\/"},"modified":"2022-11-30T09:23:54","modified_gmt":"2022-11-30T15:23:54","slug":"azure-data-factory-101","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/azure-data-factory-101\/","title":{"rendered":"Azure Data Factory 101"},"content":{"rendered":"<p>Are you just starting out with Azure Data Factory? In this post, I\u2019ll give you an introduction to Azure Data Factory, covering what it is, how it works and how to set it up. Within the video included this post is a short demo of how to create and access the platform.<\/p>\n<p>What is Data Factory?<\/p>\n<ul>\n<li>Here is a clear definition that I found from the Cloud Academy. Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation.<\/li>\n<\/ul>\n<p>How Data Factory works?<\/p>\n<ul>\n<li>The key components of Data Factory are pipelines, activities, and datasets.<\/li>\n<li><strong>Pipelines<\/strong> are made up of activities. <strong>There are 3 types of activities:<\/strong>\n<ul>\n<li>Movement \u2013 the copy activity<\/li>\n<li>Transformation \u2013 including Azure Functions, HD Insight, Stored Procedures, Spark and Databricks<\/li>\n<li>Control \u2013 ForEachLoops, If Condition Branching, Wait Times, and Validations<\/li>\n<\/ul>\n<\/li>\n<li><strong>Datasets<\/strong> represent the inputs and the outputs of the activities.<\/li>\n<li><strong>Linked Services<\/strong> \u2013 these are the connection strings and authentication for all types of sources for the data sets.<\/li>\n<li><strong>Data Flows<\/strong> \u2013 are the results of the datasets where you can apply logic and transform the data.\n<ul>\n<li>Mapping Data Flows are graphical with drag and drop functionality.<\/li>\n<li>Wrangling Data Flows are more like using Power Query or M.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Integration Runtime<\/strong> \u2013 allows you to do data integration across different network environments. There are three types of runtimes: Azure, Self-hosted, and Azure SSIS. Depending on where the data is that you need to copy will determine which of these is appropriate for the use case.<\/li>\n<\/ul>\n<p>In the video below, I provide a brief walk through of how to access and create in Azure Data Factory. Please check it out, as I think it is a good resource for those just starting out.<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/WS1Pg2vOBRQ\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>Our Azure Every Day series is another great resource. 3Cloud consultants posts weekly blogs and videos around all things Azure. You\u2019ll find all current and past blogs on our website <a href=\"https:\/\/3cloudsolutions.com\/resources\/blog\/\">or by clicking here<\/a>.<\/p>\n<p>Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or\u00a0 <a href=\"mailto:sales@3cloudsolutions.com\">sales@3cloudsolutions.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Are you just starting out with Azure Data Factory? In this post, I\u2019ll give you&mldr;<\/p>\n","protected":false},"author":28,"featured_media":10737,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10206","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10206"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10206\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/10737"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}