{"id":10412,"date":"2018-09-05T00:00:00","date_gmt":"2018-09-05T05:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/reusable-data-factory-copy-activity\/"},"modified":"2022-11-30T09:12:05","modified_gmt":"2022-11-30T15:12:05","slug":"reusable-data-factory-copy-activity","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/reusable-data-factory-copy-activity\/","title":{"rendered":"Reusable Data Factory Copy Activity"},"content":{"rendered":"<p>We are all looking for ways to save time, right? Today, I want to tell you about a time saving feature of the Azure Data Factory Copy Activity called Implicit Column Mapping.<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/eOemp5iaAu0\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>What this does is if you were pulling data from a source data set and loading it to a sync or target data set and those two data sets have the same column names, you do not need to define the schema or map the columns from your source to your destination &#8211; with the Azure Data Factory Copy Activity, it will handle all that for you.<\/p>\n<p>This allows you to use a single copy activity and re-use it simply by changing the connections properties or locations of your source and your destination. A couple of examples:<\/p>\n<ul>\n<li>If you were extracting data from a file in Data Lake and loading it to a table in SQL Server and the columns of the file in Data Lake and the table you\u2019re loading to are the same, all you have to do is define the file you want to extract from and the table you\u2019re loading to and the Azure Data Factory Copy Activity takes care of the rest.<\/li>\n<li>If you have a number of files in Azure Data Lake, possibly in different folders but all have different structure, and you\u2019re mapping them to a corresponding set of tables in a database that also have varying structure but there\u2019s a one-to-one between the file and the table, you can use that single task repeatedly by changing the names of the connections and it will map this automatically.<\/li>\n<\/ul>\n<p>If you\u2019re developing in Azure Data Factory and you\u2019ve got a repetitive pattern this is a powerful and time saving tool.<\/p>\n<p>In contrast, if you\u2019ve ever used integration services, you know that with a data flow task, you\u2019d have to map columns from your source to your destination. If the columns matched, the designer would help you do that automatically, but you\u2019re left with explicit column mappings and you couldn\u2019t really use the same data flow task to load data sources of varying structure.<\/p>\n<p>But you can use the same task to load data files with varying structure in the Data Factory Copy Task.<\/p>\n<p>Part of the trick is when you\u2019re defining your source and sync data sets, you do not need to define the schema (see graphic below). You can define the schema if you want but if you know the source and target column names are the same, you do not need to do that.<\/p>\n<p><img decoding=\"async\" style=\"width: 524px; display: block; margin: 0px auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ImplicitColumnMappings_1-002-1.jpg\" alt=\"ImplicitColumnMappings_1 (002)\" width=\"524\" \/><\/p>\n<p>In the screenshot below, you\u2019ll see this is the same with mapping \u2013 you do not need to define the mapping if your target and source columns are matching.<\/p>\n<p><img decoding=\"async\" style=\"width: 422px; display: block; margin: 0px auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ImplicitColumnMappings_2-002-1.jpg\" alt=\"ImplicitColumnMappings_2 (002)\" width=\"422\" \/><\/p>\n<p>So, with the Implicit Column Mapping feature in Azure Data Factory, it\u2019s that simple.<\/p>\n<p>Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or\u00a0 <a href=\"mailto:sales@3cloudsolutions.com\">sales@3cloudsolutions.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We are all looking for ways to save time, right? Today, I want to tell&mldr;<\/p>\n","protected":false},"author":32,"featured_media":9476,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10412","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10412","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10412"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10412\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/9476"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10412"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10412"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10412"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}