{"id":10448,"date":"2019-03-08T00:00:00","date_gmt":"2019-03-08T06:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/using-stored-procedure-in-azure-data-factory\/"},"modified":"2022-11-30T09:11:56","modified_gmt":"2022-11-30T15:11:56","slug":"using-stored-procedure-in-azure-data-factory","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/using-stored-procedure-in-azure-data-factory\/","title":{"rendered":"Using Stored Procedure in Azure Data Factory"},"content":{"rendered":"<p>In recent posts I\u2019ve been focusing on Azure Data Factory. <strong>Today I\u2019d like to talk about using a Stored Procedure as a sink or target within Azure Data Factory\u2019s (ADF) copy activity.<\/strong><\/p>\n<p><strong>Most times when I use copy activity, I\u2019m taking data from a source and doing a straight copy, normally into a table in SQL Server for example. But with using a Stored Procedure I can apply some additional processing or logic or add columns to my dataset.<\/strong><\/p>\n<p>It\u2019s a bit tricky to do, so let me give you a run through:<\/p>\n<ul>\n<li>We need to set up a few things ahead of time to use the Stored Procedure sink. I\u2019ve listed that on the screenshot below.<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 763px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/StoredProcedureInADFCopy_01.jpg\" alt=\"StoredProcedureInADFCopy_01\" width=\"763\" \/><\/p>\n<ul>\n<li>Part of the trick to using Store Procedure is we need to define in our target database a user defined table type. This is a user defined data type that has a structure of a table.<\/li>\n<li>We want this user defined table type to have similar structure to our incoming source data. So, ultimately in a few more steps we\u2019re going to map our source data to this table type.<\/li>\n<li>Then we need to create our Stored Procedure and we\u2019ll create a parameter in that Stored Procedure using this data type.<\/li>\n<li>The last piece of the trick here is setting up your target dataset within ADF to use this Stored Procedure.<\/li>\n<li>The screenshot below is an illustration of SQL Server Management Studio. In this case, I\u2019ve created a user defined table data type that I called &#8220;stage.PassingType.&#8221; Stage is my schema name and Passing Type is the name of the data type.<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 426px; display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/StoredProcedureInADFCopy_02.jpg\" alt=\"StoredProcedureInADFCopy_02\" width=\"426\" \/><\/p>\n<ul>\n<li>You\u2019ll see the columns that I defined here, and in this example, these align very closely to the file data that I\u2019m importing into my database.<\/li>\n<li>Once I\u2019ve got that table data type defined, I want to move on and create my Stored Procedure (see below). Inside this Stored Procedure I might have some logic. At a minimum maybe I want to add some metadata from the ADF process to my target table.<\/li>\n<li><\/li>\n<li><img decoding=\"async\" style=\"width: 558px; display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/StoredProcedureInADFCopy_03.jpg\" alt=\"StoredProcedureInADFCopy_03\" width=\"558\" \/><\/li>\n<li>You can see in this Stored Procedure I\u2019m doing a simple insert, but I can call out to some functions or maybe do some data cleansing or get rid of some values that I don\u2019t want. Anything I can do in a Stored Procedure, I can do here.<\/li>\n<li>The key here is at the top you\u2019ll see I created a parameter using that data type that I created earlier. The parameter name I chose here was Passing. Take note of that parameter, we\u2019ll need that in a second.<\/li>\n<li>The next thing I need to do before I get into the copy activity (see below) is configure my target data set in Data Factory.<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 625px; display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/StoredProcedureInADFCopy_04.jpg\" alt=\"StoredProcedureInADFCopy_04\" width=\"625\" \/><\/p>\n<ul>\n<li>What I want to do, and this is a little tricky, is where it says to choose a table, I want to provide the name of the parameter that I used in my Stored Procedure. So, the Stored Procedure has a parameter of a user defined table type. It\u2019s the name of that parameter here that I put into that table name.<\/li>\n<li>If I hit that Preview Data button, it actually won\u2019t work because I don&#8217;t have a table in my database\u00a0named &#8220;Passing&#8221;.<\/li>\n<li>The other key here is when I define the schema, I\u2019m really mapping or including the schema of the user defined table type and not the target table.<\/li>\n<li>Once I have that all set up, I\u2019m ready to move on to the copy activity (next screenshot). For the copy activity, I\u2019ll go ahead a choose my Stored Procedure that I defined from my list of Stored Procedures.<\/li>\n<\/ul>\n<p><img decoding=\"async\" style=\"width: 675px; display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/StoredProcedureInADFCopy_05.jpg\" alt=\"StoredProcedureInADFCopy_05\" width=\"675\" \/><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>I like to use the Import Parameter button. And if this is done right, it should recognize that I have a parameter of the table type.<\/li>\n<li>In my case I had to add the schema name to my data type there. So again, this is the name of the data type of that parameter, not the name of the parameter. In my case, I had a schema name in front of my data type name, which is what I\u2019ve shown here.<\/li>\n<li>The final thing is the mapping and here you\u2019re mapping the source (in my case from a file) columns to columns in that user defined table data type.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/rSN-AGkPrFY\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>A little bit tricky but I hope this overview of how to use a Stored Procedure as a sink in Azure Data Factory was helpful. Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or sales@3cloudsolutions.com.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In recent posts I\u2019ve been focusing on Azure Data Factory. Today I\u2019d like to talk&mldr;<\/p>\n","protected":false},"author":32,"featured_media":9359,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10448","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10448","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10448"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10448\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/9359"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10448"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10448"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10448"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}