{"id":15828,"date":"2019-01-08T14:46:00","date_gmt":"2019-01-08T22:46:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/azure-data-factory-adds-ssis-inspired-visual-data-transformation-2\/"},"modified":"2023-08-07T16:49:04","modified_gmt":"2023-08-07T23:49:04","slug":"azure-data-factory-adds-ssis-inspired-visual-data-transformation","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/azure-data-factory-adds-ssis-inspired-visual-data-transformation\/","title":{"rendered":"Azure Data Factory adds SSIS-Inspired Visual Data Transformation"},"content":{"rendered":"

You may have heard about upcoming enhancements to Data Factory. Back in September we talked here<\/a> about some of the architectural differences between SQL Server Integration Services (SSIS) and Azure Data Factory Version 2 (ADF V2), and the questions to pose as you try to select one of these products for data processing work. Well, the latest enhancements are out and they are exciting.<\/p>\n

\"SSIS-Inspired<\/p>\n

Data Flow<\/h2>\n

SSIS has always delivered two key features to ETL development: a graphical user interface for building data transformations, and the metaphors of a control flow<\/em> and a data flow<\/em> in creating packages, which act like small programs. A control flow is a series of separate tasks that you can chain together into a visually authored program, and a data flow is a similar, visually authored pipeline that processes rows of tabular data through transformations, filters, lookups, joins, and so on. Data flows are always called from control flows, generally in the context of some larger process. This metaphor is so commonplace to users of SSIS that it\u2019s sort of like the water we swim in \u2013 and when initial versions of ADF arrived without this, it was puzzling for some.<\/p>\n

Never fear, though \u2013 because the team at Microsoft knows how successful the control flow and data flow concepts are, they have been actively phasing it into ADF. The first version of Data Factory was based on moving data using a tumbling time window approach, but V2 brought the introduction of a Control Flow (June 2018). Soon, the Data Flow* feature will also become generally available. Since September we\u2019ve been able to trial the data flow features in a preview<\/a>. It\u2019s worth kicking the tires, and it looks really promising.<\/p>\n

\"Data<\/p>\n

How is this Different? (Enter Databricks)<\/h2>\n

While the visual metaphors and the ease of use look similar to SSIS, the technology under the hood here is radically different.<\/p>\n

SSIS works, at a high level, by the visual editor building your packages as XML files. There\u2019s an executable dtexec.exe<\/em> that can read, parse, and execute those package files. Over the years different flavors of this have come along, such as Project Deployment in the SSIS catalog in SQL Server, but the fundamental architecture is the same. It works well, but it\u2019s not necessarily cloud-optimized, and it\u2019s a bit limited in terms of taking advantage of new innovations that have been created in the big data world \u2013 especially with respect to parallelism\/massively parallel processing (MPP)\/multi-machine scale-out.<\/p>\n

The approach in ADF V2 is sort of 180 degrees from SSIS \u2013 it starts with the premise that data is going to be processed on a Spark cluster (specifically an Azure Databricks cluster<\/a>, which is a specific flavor of Spark) in the cloud, and the engineering of the data flow ADF components is all about making it easier and more intuitive to harness that massive processing horsepower.<\/p>\n

We\u2019ll return to how it does that in a second, but first, why is it important? Well, it means that we get two wonderful capabilities right in the product:<\/p>\n

    \n
  1. Automatic scaling to fit workload requirements. We don\u2019t have to buy machinery big enough to handle the peak, maybe infrequent workload, and the infrastructure can be tailored for best fit, all the time.<\/li>\n
  2. On top of auto scaling, there\u2019s the opportunity for this system to do execution optimization<\/em> on your code. Where SSIS will reliably run the packages you feed it, it does little in the way of tuning or optimizing those, and they just run in the sequence that you\u2019ve written. Think about SQL Server\u2019s Query Optimizer \u2013 it\u2019s not an exact match, but Spark has some similar features that enable it to do optimization of code that you provide. Because ADF is deliberately built over the top of these cluster technologies, we get them with no additional or special effort. That\u2019s huge.<\/li>\n<\/ol>\n

    ADF Data Flow Design<\/h2>\n

    There are several excellent, existing walkthroughs out there showing how to build your first data flow, so I won\u2019t take you through that, but I do want to talk about how ADF harnesses the different technology in this new world.<\/p>\n

    If you try ADF Data Flows, you\u2019ll find the visual editor looks cosmetically different, but conceptually is much the same as SSIS, so the learning curve should be short. Many SSIS transformations map directly to equivalents in ADF, such as joins, filters, branching, derived column expressions, and so on. What\u2019s happening behind the scenes is quite different, though.<\/p>\n

    First, you\u2019ll be working in a browser instead of a desktop tool, but it\u2019s a fully featured authoring experience. Second, the Data Factory setup will compose and store your Data Flow as a JSON object (think: a modern version of the SSIS XML file).<\/p>\n

    Third, and this is the new bit: Data Factory will automatically compile your work into ready-to-run code for Apache Spark, on a Databricks<\/a> cluster \u2013 with no additional effort from developers. This is really the vital difference: ADF V2 Data Flow is, in some sense, a visual editor to enable you to \u201cwrite\u201d code for a Databricks\/Spark cluster without code-writing<\/em>. This means it can handle huge sets of data with a lot less time and energy invested in infrastructure concerns. Sure, there is some important knowledge a team will need to gain to configure the cluster correctly, but it\u2019s dramatically faster and easier than the old days of building out physical Hadoop clusters with their administrative workload, and it\u2019s much easier for an ETL developer to take advantage of the power of that underlying technology.<\/p>\n

    New Decisions<\/h2>\n

    So how does this change the math for SSIS vs. ADF from our last installment? Until now, the fact that ADF didn\u2019t have the equivalent of the SSIS data flow was a barrier for some teams and some types of workloads. Once this feature becomes generally available, that barrier will disappear and the ADF service will become viable for teams who:<\/p>\n