{"id":15744,"date":"2020-05-13T12:45:00","date_gmt":"2020-05-13T19:45:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/generate-data-with-databricks-using-tpc-ds-3\/"},"modified":"2024-01-04T09:16:16","modified_gmt":"2024-01-04T17:16:16","slug":"generate-data-with-databricks-using-tpc-ds","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/generate-data-with-databricks-using-tpc-ds\/","title":{"rendered":"Generate Data with Databricks USING TPC-DS"},"content":{"rendered":"

Sample data is critical for learning data systems and techniques, developing proofs of concept, and performance testing. And while sample datasets are easy to find, a limitation shared by most of them is that they are small. This is fine in many cases, but if you really want to evaluate a big data system, you\u2019re going to need big data. Unfortunately, big datasets are difficult to find. One solution to this problem is the TPC-DS benchmark dataset<\/a>. Instead of having to download and move this data to the desired location, it is generated using software provided by the TPC (Transaction Processing Performance Council).<\/p>\n

<\/p>\n

The TPC-DS dataset has some important advantages; the first being that it is variable in size \u2013 supporting datasets up to 100 terabytes (TB)!<\/p>\n

\"TPC-DS<\/p>\n

Another big advantage is that the data is modeled as multiple snowflake schemas, with fact and dimension tables having realistic proportions. This makes the dataset representative of typical data warehouse workloads. A nice summary of the TPC-DS benchmark can be found here<\/a>.<\/p>\n

Databricks spark-sql-perf Library<\/h2>\n

You can run the data generator as is from the TPC (dsdgen) on your personal computer or other machines, but the features are limited and it\u2019s difficult to impossible to generate data at the larger scales on modest hardware. This is where the spark-sql-perf<\/a> library from Databricks comes in handy. The spark-sql-perf library allows you to generate TPC-DS data on a Databricks cluster size of your choosing, and provides some important added features, such as:<\/p>\n