{"id":10207,"date":"2019-02-20T00:00:00","date_gmt":"2019-02-20T06:00:00","guid":{"rendered":"https:\/\/threecloud.wpengine.com\/post\/azure-data-factory-event-triggers\/"},"modified":"2022-11-30T09:11:57","modified_gmt":"2022-11-30T15:11:57","slug":"azure-data-factory-event-triggers","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/azure-data-factory-event-triggers\/","title":{"rendered":"Azure Data Factory Event Triggers"},"content":{"rendered":"<p><strong>If you\u2019re still triggering your ETL processes on a schedule (maybe once an hour or at a set time each night), I want to tell you about a better way.<\/strong> <strong>In this post I\u2019ll tell you about Event Triggers within Azure Data Factory.<\/strong><\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/SDbDVP9Tsxk\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><strong>In today\u2019s world we prefer an approach where processes are initiated by the occurrence of an event. Azure Data Factory Event Triggers do this for us<\/strong>. Event Triggers work when a blob or file is placed into blob storage or when it\u2019s deleted from a certain container. When you place a file in a container, that will kick off an Azure Data Factory pipeline.<\/p>\n<p><strong>These triggers use the Microsoft Event Grid technology.<\/strong> The Event Grid can be used for a variety of event driven processing in Azure; Azure Data Factory is using Event Grid under the covers.<\/p>\n<p>It\u2019s important to know that as of today, the Event Triggers are only triggered by the creation\/deletion of a blob in a blob container but as we are using Event Grid, I\u2019d like to think we may have the option in the future to enable them through other events.<\/p>\n<p>Also, good to note is that <strong>we don\u2019t have to use the functionality within Azure Data Factory to kick off a pipeline; we can use our own triggers.<\/strong> For example, we can have a Logic App that uses an Azure function and that Azure function might kick off a pipeline based on some event that happens inside our app.<\/p>\n<p><strong>If we do use our own triggers, we are outside of the framework of Azure Data Factory. The nice thing about Event Triggers is they are all managed inside the framework of Data Factory.<\/strong><\/p>\n<p><strong>Before we can use Event Triggers, we must register the Microsoft Event Grid resource<\/strong> (which I show in the screenshot below):<\/p>\n<p><img decoding=\"async\" style=\"width: 849px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADFEventTriggers_02.jpg\" alt=\"ADFEventTriggers_02\" width=\"849\" \/><\/p>\n<p>This is a property of your subscription, so you\u2019ll have to register that resource before you can use Event Triggers in Azure Data Factory.<\/p>\n<p><strong>In my next screenshot shows how to create a new trigger:\u00a0<\/strong><\/p>\n<p><img decoding=\"async\" style=\"width: 630px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADFEventTriggers_03.jpg\" alt=\"ADFEventTriggers_03\" width=\"630\" \/><\/p>\n<p>We do this from the Azure Data Factory designer. When creating a trigger, we have the choice of doing a schedule, a tumbling window, but in my case, I want to choose event. Once I choose event, I need to select the storage account that we\u2019re going to keying on and the file path or pattern that we\u2019re going to be looking for.<\/p>\n<p>So, I can set this up to say, when a file is placed in a certain container, then trigger the pipeline, for instance. Or if there\u2019s a file that\u2019s created that has a certain naming pattern then I can trigger the pipeline; this is all created in the designer.<\/p>\n<p>In my example, you\u2019ll see I\u2019ve made the selection to fire a trigger when a blob is created. You can also choose to do it when a blob is deleted.<\/p>\n<p><strong>Now, that I\u2019ve created that trigger, next what I need to do is associate my pipeline to that trigger. See below:<\/strong><\/p>\n<p><img decoding=\"async\" style=\"width: 621px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2021\/09\/ADFEventTriggers_04.jpg\" alt=\"ADFEventTriggers_04\" width=\"621\" \/><\/p>\n<p>This works the same as we\u2019ve done with other triggers. So, from my pipeline, I say that I want to create a trigger and then I would choose the event trigger that I created in the previous step. What\u2019s interesting here is if I have pipeline parameters configured on my pipeline, I can take the values of the file path in the file name from the blob that was created in my example and pass those as inputs or context into my execution which is very handy and powerful.<\/p>\n<p><strong>Event Triggers are incredibly useful and give us a much better option for triggering our ETL processes.\u00a0<\/strong><\/p>\n<p>Need further help? Our expert team and solution offerings can help your business with any Azure product or service, including Managed Services offerings. Contact us at 888-8AZURE or\u00a0 <a href=\"mailto:sales@3cloudsolutions.com\">sales@3cloudsolutions.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you\u2019re still triggering your ETL processes on a schedule (maybe once an hour or&mldr;<\/p>\n","protected":false},"author":32,"featured_media":9372,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[],"class_list":["post-10207","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10207","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=10207"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/10207\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/9372"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=10207"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=10207"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=10207"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}