{"id":10210,"date":"2019-02-19T00:00:00","date_gmt":"2019-02-19T06:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/azure-data-factory-pipeline-templates\/"},"modified":"2022-11-30T09:11:58","modified_gmt":"2022-11-30T15:11:58","slug":"azure-data-factory-pipeline-templates","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/azure-data-factory-pipeline-templates\/","title":{"rendered":"Azure Data Factory Pipeline Templates"},"content":{"rendered":"<p><strong>If you\u2019re new to Azure Data Factory and unsure what you can do with it, I want to tell you about a new option within Data Factory called Pipeline Templates.<\/strong><\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/WCVzuBTRBGc\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>When you go to the Azure website, open the portal and go into the Data Factory Designer, there\u2019s a new option on the \u2018Let\u2019s Get Started\u2019 page for create a pipeline from a template.<\/p>\n<p><img decoding=\"async\" style=\"width: 600px; margin: 0px auto; display: block;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADF_Templates_01.jpg\" alt=\"ADF_Templates_01\" width=\"600\" \/><\/p>\n<p>When you hit that button, it will take you to the template gallery, which has several tiles for different pipeline templates provided by Microsoft.<\/p>\n<p><img decoding=\"async\" style=\"width: 1011px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADF_Templates_02.jpg\" alt=\"ADF_Templates_02\" width=\"1011\" \/><\/p>\n<p>Again, <strong>if you\u2019re new to Data Factory, this is a great way to get some ideas about what it can do and some of the services it can connect to. Also, if you\u2019re using Data Factory for a certain technology, this is a way to learn about how to configure other technologies that you may not be using.<\/strong><\/p>\n<p>I think it\u2019s always interesting to learn about the way other people are doing things. If you\u2019ve been doing things a certain way, this may give you some other patterns or even technology to use in place of what you\u2019re currently using. I also found some references to code, in particular some of the variable expressions and such that I found in the templates to use as a resource.<\/p>\n<p>So, after I select one of the templates from the gallery, it will take me to configuration page. Look at my example below:<\/p>\n<p><img decoding=\"async\" style=\"width: 875px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADF_Templates_03.jpg\" alt=\"ADF_Templates_03\" width=\"875\" \/><\/p>\n<p>I&#8217;ve selected the copying data from Oracle to Azure SQL Data Warehouse. When I select this template, it gives me a preview of the activities that are included. In this case it\u2019s a simple one where I just have a single activity in the template.<\/p>\n<p>Then it\u2019s asking me for a couple of inputs. Here it\u2019s just looking at the two linked services that are required by the copy activity. I can select those from my drop down or I can create them like I would do with creating a link service outside of here.<\/p>\n<p>So, I make a reference to my two connections, hit the button and it\u2019s going to create a pipeline and it will be configured with those selections that I\u2019ve made. It\u2019s that simple.<\/p>\n<p>Once you\u2019ve created that pipeline from the template you could certainly modify the pipeline, but these templates are a good way to get started.<\/p>\n<p>I also want to point out I can also get to the template gallery from the Designer. I have a new option under \u2018Adding a Pipeline\u2019 where I can select that I want to create a pipeline from a template. See below:<\/p>\n<p><img decoding=\"async\" style=\"width: 235px; display: block; margin: 0px auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADF_Templates_04.jpg\" alt=\"ADF_Templates_04\" width=\"235\" \/><\/p>\n<p>Another nice thing is we can create our own templates. So, if you have a pipeline and you\u2019ve been copying it to maybe replicate the logic, etc., you can create your own template and make it available in the gallery. To do that you do need to be working within a Git repository, so you must have version control configured here.<\/p>\n<p>If you\u2019re working within your version control branch, you can create a pipeline and you\u2019ll have an option (see screenshot below) that says Save as Template. This way once I get my pipeline working in the condition I want, I can save it and this template will now appear in the template gallery.<\/p>\n<p>If I filter it down to show just my templates, you can see in my example here that I have one template created.<br \/>\n<img decoding=\"async\" style=\"width: 536px; display: block; margin: 0px auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADF_Templates_06-1.jpg\" alt=\"ADF_Templates_06-1\" width=\"536\" \/><\/p>\n<p>A nice feature here whether you\u2019re new to Data Factory or some new options if you\u2019re already a user. I hope this was helpful to you.<strong>\u00a0<\/strong><\/p>\n<p>Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or\u00a0 <a href=\"mailto:sales@3cloudsolutions.com\">sales@3cloudsolutions.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you\u2019re new to Azure Data Factory and unsure what you can do with it,&mldr;<\/p>\n","protected":false},"author":32,"featured_media":9374,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10210","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10210","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10210"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10210\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/9374"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10210"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}