{"id":15650,"date":"2022-02-03T13:30:00","date_gmt":"2022-02-03T21:30:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/generate-a-calendar-dimension-in-spark-3\/"},"modified":"2024-01-04T09:14:36","modified_gmt":"2024-01-04T17:14:36","slug":"generate-a-calendar-dimension-in-spark","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/generate-a-calendar-dimension-in-spark\/","title":{"rendered":"Generate a Calendar Dimension in Spark"},"content":{"rendered":"
\n

With the Data Lakehouse<\/a> architecture shifting data warehouse workloads to the data lake, the ability to generate a calendar dimension (AKA date dimension) in Spark has become increasingly important. Thankfully, this task is made easy with PySpark and Spark SQL. Let’s dive right into the code!<\/p>\n<\/div>\n

\n

\"Calendar<\/h2>\n

How to Begin<\/span><\/h3>\n

The process starts by generating an array of dates, then exploding this array into a data frame, and creating a temporary view called dates<\/span>.<\/p>\n<\/div>\n

from pyspark.sql.functions import explode, sequence, to_date\r\n\r\nbeginDate = '2000-01-01'\r\nendDate = '2050-12-31'\r\n\r\n(\r\n  spark.sql(f\"select explode(sequence(to_date('{beginDate}'), to_date('{endDate}'), interval 1 day)) as calendarDate\")\r\n    .createOrReplaceTempView('dates')\r\n)<\/code><\/pre>\n
The dates<\/span> temporary view has a single column, with a row for every date in the range specified above. It looks like this:<\/p>\n
\"generate.calendar.dimension.in.spark.select.dates\"<\/p>\n
Now that we have a temporary view containing dates, we can use Spark SQL to select the desired columns for the calendar dimension.<\/p>\n
select\r\n  year(calendarDate) * 10000 + month(calendarDate) * 100 + day(calendarDate) as dateInt,\r\n  CalendarDate,\r\n  year(calendarDate) AS CalendarYear,\r\n  date_format(calendarDate, 'MMMM') as CalendarMonth,\r\n  month(calendarDate) as MonthOfYear,\r\n  date_format(calendarDate, 'EEEE') as CalendarDay,\r\n  dayofweek(calendarDate) AS DayOfWeek,\r\n  weekday(calendarDate) + 1 as DayOfWeekStartMonday,\r\n  case\r\n    when weekday(calendarDate) < 5 then 'Y' else 'N' end as IsWeekDay, dayofmonth(calendarDate) as DayOfMonth, case when calendarDate = last_day(calendarDate) then 'Y' else 'N' end as IsLastDayOfMonth, dayofyear(calendarDate) as DayOfYear, weekofyear(calendarDate) as WeekOfYearIso, quarter(calendarDate) as QuarterOfYear, \/* Use fiscal periods needed by organization fiscal calendar *\/ case when month(calendarDate) >= 10 then year(calendarDate) + 1\r\n    else year(calendarDate)\r\n  end as FiscalYearOctToSep,\r\n  (month(calendarDate) + 2) % 12 + 1 AS FiscalMonthOctToSep,\r\n  case\r\n    when month(calendarDate) >= 7 then year(calendarDate) + 1\r\n    else year(calendarDate)\r\n  end as FiscalYearJulToJun,\r\n  (month(calendarDate) + 5) % 12 + 1 AS FiscalMonthJulToJun\r\nfrom\r\n  dates\r\norder by\r\n  calendarDate<\/code><\/pre>\n
When you’re satisfied with the results, the same query can be used to load the calendar dimension into a Delta Lake<\/a> table and register it in the Hive Metastore.<\/p>\n
create or replace table dim_calendar\r\nusing delta\r\nlocation '\/mnt\/datalake\/dim_calendar'\r\nas select\r\n  year(calendarDate) * 10000 + month(calendarDate) * 100 + day(calendarDate) as DateInt,\r\n  CalendarDate,\r\n  year(calendarDate) AS CalendarYear,\r\n  date_format(calendarDate, 'MMMM') as CalendarMonth,\r\n  month(calendarDate) as MonthOfYear,\r\n  date_format(calendarDate, 'EEEE') as CalendarDay,\r\n  dayofweek(calendarDate) as DayOfWeek,\r\n  weekday(calendarDate) + 1 as DayOfWeekStartMonday,\r\n  case\r\n    when weekday(calendarDate) < 5 then 'Y' else 'N' end as IsWeekDay, dayofmonth(calendarDate) as DayOfMonth, case when calendarDate = last_day(calendarDate) then 'Y' else 'N' end as IsLastDayOfMonth, dayofyear(calendarDate) as DayOfYear, weekofyear(calendarDate) as WeekOfYearIso, quarter(calendarDate) as QuarterOfYear, \/* Use fiscal periods needed by organization fiscal calendar *\/ case when month(calendarDate) >= 10 then year(calendarDate) + 1\r\n    else year(calendarDate)\r\n  end as FiscalYearOctToSep,\r\n  (month(calendarDate) + 2) % 12 + 1 as FiscalMonthOctToSep,\r\n  case\r\n    when month(calendarDate) >= 7 then year(calendarDate) + 1\r\n    else year(calendarDate)\r\n  end as FiscalYearJulToJun,\r\n  (month(calendarDate) + 5) % 12 + 1 as FiscalMonthJulToJun\r\nfrom\r\n  dates<\/code><\/pre>\n
Examine the Calendar Dimension<\/span><\/h3>\n
Let’s examine the calendar dimension with simple query. The first few columns should look like this:<\/p>\n
\"generate.calendar.dimension.in.spark.select.calendar.dimension\"<\/p>\n
Now we have a calendar dimension in the data lake<\/a><\/span> that can be used to query fact tables or used as a source for semantic models. As a test, let’s perform a Spark SQL query that aggregates sales data by month. I like using robust datasets for tests, so I’m going to query a 2.75 billion row version of store_sales from the TPC-DS benchmark<\/a>.<\/p>\n
\"generate.calendar.dimension.in.spark.select.aggregation\"<\/p>\n
Notice how we’re using a calendar dimension in the data lake just like we’ve done for ages in the data warehouse<\/a><\/span>. If you’d like to generate a calendar dimension in your own environment, you can find the code in the calendar-dimension-spark repository<\/a>.<\/p>\n
The ability to create a calendar dimension in Spark allows for easy navigation of fact tables in the data lake. The benefits of this dimension will be obvious to data warehouse users and analysts – it can be reused across multiple analysis, it is scalable, and it is extremely user friendly.<\/span><\/span><\/p>\n
3Cloud has strong experience in generating calendar dimensions in Spark. If you’re wondering how these tools could benefit your business outcomes, contact us<\/a> today!<\/p>\n","protected":false},"excerpt":{"rendered":"
With the Data Lakehouse architecture shifting data warehouse workloads to the data lake, the ability to generate a calendar dimension in Spark has become increasingly important.<\/p>\n","protected":false},"author":21,"featured_media":12294,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260,297],"tags":[303,304],"class_list":["post-15650","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","category-data-platform","tag-modern-analytics","tag-modern-data-platform","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15650","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15650"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15650\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/12294"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15650"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15650"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15650"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}