{"id":15790,"date":"2019-08-01T15:50:12","date_gmt":"2019-08-01T22:50:12","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/retail-analytics-product-dimension-load-pattern-using-azure-databricks-3\/"},"modified":"2024-02-21T13:27:10","modified_gmt":"2024-02-21T21:27:10","slug":"retail-analytics-product-dimension-load-pattern-using-azure-databricks","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/retail-analytics-product-dimension-load-pattern-using-azure-databricks\/","title":{"rendered":"Retail Analytics: Product Dimension Load Pattern using Azure Databricks"},"content":{"rendered":"

The heart of any retail analytics solution is product data.\u00a0 Retail and CPG organizations<\/a> need the ability to model and load complex product granularities, attributes, and relationships in order to fulfill valuable analytics, visualizations, and Artificial Intelligence (AI) capabilities.\u00a0 These requirements often materialize into the Product dimension in the organizational Data Warehouse.<\/p>\n

<\/p>\n

The Azure Data Services<\/a> provide a rich and robust set of tools for modeling, loading, and maintaining product data from various data sources.\u00a0 With such a wide variety of options and a lack of official documentation on best practices, it can be confusing to know which tools and patterns to use to load your Data Warehouse in Azure.\u00a0 The purpose of this blog post, and the subsequent ones to follow, is to help provide clarity on this topic.\u00a0 In each post, we\u2019ll review the tools and patterns to load your product dimension.<\/p>\n

3 Simplified Patterns<\/h2>\n

Heavily simplified, there are three Data Warehouse load patterns and tool combinations in Azure to perform ETL\/ELT.\u00a0 Each option has its own pros\/cons that should be evaluated by organizations to understand the best fit for their use cases, requirements, cost, and staff skillsets.<\/p>\n

<\/p>\n

In today\u2019s blog post, we\u2019ll review the pattern and tools associated with what I am calling In-Memory Transformations with Spark<\/strong>.\u00a0 The focus on this pattern is to use Azure Databricks to transform the data in the Data Lake into meaningful data and land it into the Data Warehouse and our Product dimension.<\/p>\n

The architecture diagram below outlines the tools used in this pattern and the order\/flow of the data.<\/p>\n

    \n
  1. Ingest Data using the Azure Data Factory Copy Activity (ADF)<\/strong> \u2013 ADF <\/a>Copy Activities are used to copy data from source business applications and other sources of data into the RAW area of the Data Lake.<\/li>\n
  2. Store Data in Azure Data Lake Store Gen2 (ADLS) <\/strong>\u2013 ADLS Gen2 <\/a>is used to store the data for our Data Lake. It combines the best of Azure Storage and ADLS Gen1 to enable the Hadoop Distribute File System (HDFS) as a service.\u00a0 It is a cheap and robust storage system built for analytics.<\/li>\n
  3. Transform Data using Azure Databricks <\/strong>\u2013 Databricks <\/a>is used to source data from the Data Lake and enhance\/transform the data in-memory before landing it into the Data Warehouse star schema.<\/li>\n
  4. Model Data in Azure SQL Database (DB) or Azure SQL Data Warehouse (DW)<\/strong> \u2013 Azure SQL DB<\/a> or Azure SQL DW<\/a> are used to store the data for the Data Warehouse star schema. Both solutions are SQL Server based and provide an easy consumption layer for business\/data analysts and dashboard\/report writers.\u00a0 Azure SQL DB is a robust SMP (symmetric multiprocessing<\/a>) based relational database solution enabling scalability up to single-digit terabyte (TB) sized Data Warehouses.\u00a0 For larger Data Warehouse solutions (single-digit TBs to triple-digit TBs) Azure SQL DW provides an MPP (massively parallel processing<\/a>) capability based upon SQL Server.<\/li>\n
  5. Serve Data using Azure Analysis Services or Power BI Premium<\/strong> \u2013 Azure Analysis Service<\/a> or Power BI Premium<\/a> enable a rich semantic layer capability in the architecture. Tables are pre-joined, columns use business vernacular and are organized for ease of consumption, security is embedded into the model, and business calculations are added for analytics.<\/li>\n
  6. Consumption of Data using Power BI<\/strong> \u2013 Power BI<\/a> provides a rich experience to Azure Analysis Services or Power BI Premium via Live Connection. Users can create ad-hoc reports, consume reports and dashboards, and even create paginated reports.\u00a0 Sophisticated authors can even create feature rich reporting solutions that look and feel like a reporting application<\/a>.<\/li>\n<\/ol>\n

    <\/p>\n

    Ingesting Data to the Data Lake<\/h2>\n

    Ingesting data into the Data Lake occurs in steps 1 and 2 in our architecture.\u00a0 Azure Data Factory (ADF) provides an excellent mechanism for loading data from source applications into a Data Lake stored in Azure Data Lake Store Gen2.\u00a0 In fact, Microsoft offers a template in the ADF Template gallery which provides a metadata driven approach for doing so.\u00a0 The template comes with a control table example in a SQL Server Database, a data source dataset and a data destination dataset.\u00a0 More on this template can be found here<\/a> in the official documentation.<\/p>\n

    <\/p>\n

    The ADF pipeline starts with a Lookup<\/strong> activity which retrieves the record set from the control table and makes it available to the pipeline.\u00a0 The control table holds records which have the table name, the query to use for filtering the data in the source system, and the destination folder path for the data in the Data Lake.\u00a0 The ForEach<\/strong> activity iterates over the GetPartitionList<\/strong> data set.<\/p>\n

    <\/p>\n

    <\/p>\n

    Within the ForEachPartition activity, two Copy Activities execute for each record of the control table.\u00a0 The CopySourceToStage<\/strong> Copy Activity runs the SQL query in the control table against the source system.\u00a0 The results are then loaded to the Staging destination folder location in ADLS which is also provided by the control table.\u00a0 The CopyStageToRaw<\/strong> Copy Activity copies the data from the Staging folder location and moves the data copy to a RAW location in ADLS that is partitioned by a timestamp.<\/p>\n

    An example of an output directory structure of the Data Lake is below.\u00a0 We can see that after our ADF pipeline completes, our Data Lake has a directory structure that mimics the following topology: storage container -> subject area -> data lake zone -> source system -> table -> table data.<\/p>\n

    <\/p>\n

    In this example, the data for our tables are stored in Apache Parquet<\/a> format (Product.Parquet).\u00a0 Parquet file format is useful because it provides columnar compression by default and it stores the metadata of the file in the file itself, which can be used by downstream systems.<\/p>\n

    <\/p>\n

    In the second Copy Activity, we can see that the data is copied from the Staging zone into a Raw zone of the Data Lake, which includes a timestamp of when the ETL was executed.\u00a0 Subsequent runs will create new folders which will allow the Data Lake to show history in the data.<\/p>\n

    <\/p>\n

    Loading the Production Dimension with Azure Databricks<\/h2>\n

    Pulling data from the Data Lake, transforming that data, and loading it into the Data Warehouse occurs in steps 3 and 4 in our architecture.\u00a0 Azure Databricks does the bulk of the work for these steps.\u00a0 Before we dive into our ETL code, it\u2019s important to know why we\u2019re using Databricks in this pattern. \u00a0The following list outlines why organizations should use Databricks for their ETL\/ELT workloads:<\/p>\n