{"id":16136,"date":"1970-01-01T00:00:00","date_gmt":"1970-01-01T08:00:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/query-millions-of-genomic-variants-at-scale-using-azure-synapse-genomics-ebook-2\/"},"modified":"2024-01-04T10:45:27","modified_gmt":"2024-01-04T18:45:27","slug":"query-millions-of-genomic-variants-at-scale-using-azure-synapse-genomics-ebook","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/query-millions-of-genomic-variants-at-scale-using-azure-synapse-genomics-ebook\/","title":{"rendered":"Query Millions of Genomic Variants At-Scale using Azure Synapse (Genomics eBook)"},"content":{"rendered":"

One struggle for genomics research is the ability to analyze the vast amounts of data in an efficient way. Previously, this would have been performed using large, on-premise high performance computing (HPC) cluster jobs. Today, the cloud offers us opportunities to perform bioinformatics analyses interactively and at-scale.<\/p>\n

In this demo (see the video at the end of this post), I’ll be showing how Azure Synapse can be used to analyze millions of variants quickly using basic structured query language (SQL) commands. I retrieved >80 million variant records from over 2,500 individuals from the 1000 Genomes Project<\/a> (Phase 3 release<\/a>). In its variant call format (VCF) form, this data is about 168GB in size, which encompasses all 22 autosomes, X and Y sex chromosomes, and mitochondrial variants.<\/p>\n

\"actg_synapse\"<\/p>\n

<\/p>\n


\nWhat’s Azure Synapse?<\/span><\/h2>\n

Azure Synapse Analytics is a recent amalgamation\/rebranding of some services in Azure, including Azure SQL Data Warehouse with connectors to Power BI and other services. Synapse includes serverless pool, dedicated SQL pool, and Apache Spark pool options for flexible and scalable data workloads in Azure.<\/p>\n

Learn more about Azure Synapse Analytics here<\/a><\/span>.<\/p>\n

This service isn’t marketed for any specific industry, but its immense scalability makes it a perfect tool for browsing tons of genomics data. All we need to do is get our data in a format that’s usable in Synapse.<\/p>\n

<\/p>\n

Converting from VCF to Parquet<\/span><\/h3>\n

VCF files, though popular in bioinformatics, are a mixed file type that include a metadata header and a more structured table-like body. Using the Glow<\/span><\/a> package in Apache Spark, we can convert VCF files into the Parquet format, which works excellently in distributed contexts like a Data Lake or in Azure Synapse.<\/p>\n

We can perform this conversion at scale in Spark (either in Azure Databricks or Azure Synapse) using only four lines of code (plus two more lines for calculating optional summary statistics). The following code reads a VCF file from a mounted genomics data lake location.<\/p>\n

\n
\n

<\/p>\n

\n
input_vcf_path = \"\/mnt\/1000genomes\/phase3_vcfs\/chr1.vcf.gz\"<\/span>\r\noutput_parquet_path = \"\/mnt\/1000genomes\/phase3_parquets\/chr1.parquet\"<\/span>\r\n\r\nvcf_df = spark.read.format(\"vcf\"<\/span>).load(input_vcf_path) \r\n\t       .withColumn(\"hardyweinberg\"<\/span>, expr(\"hardy_weinberg(genotypes)\"<\/span>)) \r\n\t       .withColumn(\"stats\"<\/span>, expr(\"call_summary_stats(genotypes)\"<\/span>))\r\n\r\nvcf_df.write.format(\"parquet\"<\/span>).save(output_parquet_path)\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
When this VCF file is read into Spark, it converted into a Spark DataFrame. Then, after calculating some optional summary statistics, the cluster will write out multiple partitioned Parquet files back to the data lake. For the 1000 Genomes data I’m using here, the Parquet format reduces the ~168GB of VCF data down to ~74GB.<\/p>\n
External Tables<\/span><\/h3>\n
In Azure Synapse, external tables are a really awesome capability to connect to data that lives in your data lake. Once you create an external table, you can query the data just as if it were a real table in your database. This works on delimited text files (like CSV or TSV), Hive Orc files, or Parquet.<\/p>\n
A snippet of the CREATE EXTERNAL TABLE script for VCF data looks like this:<\/p>\n
\n\n\n\n
\n
\n
\n
<\/p>\n
\n
CREATE<\/span> EXTERNAL<\/span> TABLE<\/span> phase3_variants (\r\n\t[contigName] varchar(50<\/span>),\r\n\t[start<\/span>] bigint,\r\n\t[end<\/span>] bigint,\r\n\t[names<\/span>] varchar(1000<\/span>),\r\n\t[referenceAllele] varchar(1000<\/span>),\r\n\t[alternateAlleles] varchar(1000<\/span>),\r\n\t[qual] float,\r\n    ...\r\n\t[hardyweinberg] varchar(8000<\/span>),\r\n\t[stats] varchar(8000<\/span>)\r\n\t)\r\n\tWITH<\/span> (\r\n\tLOCATION<\/span> = 'phase3_parquets\/ALL.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.parquet'<\/span>,\r\n\tDATA_SOURCE = [1000<\/span>genomes_genomicsdls_dfs_core_windows_net],\r\n\tFILE_FORMAT = [SynapseParquetFormat]\r\n\t)\r\nGO<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
While you could write this yourself, Synapse will help you generate scripts automatically. Find the Parquet file(s) that you want to load, right-click it, click New SQL Script<\/span>, and then click Create external table<\/span>. This will generate a basic script, which you should edit to fit your specific VCF file needs. (Specifically, make sure the data types are correct.)<\/p>\n
<\/p>\n
Speedy Queries<\/span><\/h3>\n
Azure Synapse can perform queries on this data very quickly. In fact, for some of my example queries, it only took a couple minutes to return the results from >80 million variant records. Here are some example queries:<\/p>\n
\n\n\n\n\n\n\n
Sample Task<\/strong><\/span><\/td>\nQuery Time
\n(in minutes)<\/strong><\/span><\/td>\n<\/tr>\n
\n
\nFind Minor Structural Variants<\/h3>\n
\n
\n
<\/p>\n
\n
SELECT<\/span>  *\r\nFROM<\/span>    phase3_variants\r\n\r\nWHERE<\/span>   JSON_VALUE(INFO_VT, '$[0]'<\/span>) = 'SV'<\/span>                      --Structural variants<\/span>\r\nAND<\/span>     qual > 95<\/span>                                               --High quality<\/span>\r\nAND<\/span>     alternateAlleles LIKE<\/span> '%ALU%'<\/span>                           --Transposable elements<\/span>\r\nAND<\/span>     JSON_VALUE(hardyWeinberg, '$.hetFreqHwe'<\/span>) >= 0<\/span>.05<\/span>       --Higher heterozygous frequency<\/span>\r\nAND<\/span>     JSON_VALUE(stats, '$.alleleFrequencies[1]'<\/span>) <= 0<\/span>.05<\/span>     --Rare minor alleles (variants)<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n
1:11<\/td>\n<\/tr>\n
\n
\nCalculate Indel Size Distribution<\/h3>\n
\n
\n
<\/p>\n
\n
SELECT<\/span>  LEN(JSON_VALUE(alternateAlleles, '$[0]'<\/span>)) - LEN(referenceAllele) AS<\/span> InsertionLength\r\n        ,COUNT<\/span>(DISTINCT<\/span>(names<\/span>)) AS<\/span> VariantCount\r\nFROM<\/span>    phase3_variants\r\n\r\nWHERE<\/span>   JSON_VALUE(INFO_VT, '$[0]'<\/span>) = 'INDEL'<\/span>                   --Indels<\/span>\r\nAND<\/span>     INFO_MULTI_ALLELIC = 'False'<\/span>                            --Biallelics Only<\/span>\r\nGROUP<\/span> BY<\/span> LEN(JSON_VALUE(alternateAlleles, '$[0]'<\/span>)) - LEN(referenceAllele)\r\nORDER<\/span> BY<\/span> LEN(JSON_VALUE(alternateAlleles, '$[0]'<\/span>)) - LEN(referenceAllele) DESC<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n
1:34<\/td>\n<\/tr>\n
\n
Find Motif Matches<\/h3>\n
\n
\n
<\/p>\n
\n
DECLARE<\/span> @motif varchar(1000<\/span>)\r\nSet<\/span>     @motif = 'TA'<\/span>\r\n\r\nSELECT<\/span>  contigName\r\n        ,[start<\/span>]\r\n        ,[end<\/span>]\r\n        ,names<\/span>\r\n        ,referenceAllele\r\n        ,JSON_VALUE(alternateAlleles, '$[0]'<\/span>) AS<\/span> alternateAllele\r\n        ,(LEN(JSON_VALUE(alternateAlleles, '$[0]'<\/span>)) - LEN(REPLACE<\/span>(JSON_VALUE(alternateAlleles, '$[0]'<\/span>), @motif, ''<\/span>))) \/ LEN(@motif) AS<\/span> MotifMatches\r\nFROM<\/span>    phase3_variants\r\n\r\nWHERE<\/span>   JSON_VALUE(INFO_VT, '$[0]'<\/span>) = 'INDEL'<\/span>                   --Indels<\/span>\r\nAND<\/span>     INFO_MULTI_ALLELIC = 'False'<\/span>                            --Biallelics Only<\/span>\r\nAND<\/span>     JSON_VALUE(alternateAlleles, '$[0]'<\/span>) LIKE<\/span> '%'<\/span> + @motif + '%'<\/span>      --TA Matches<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n
\u00a01:49<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
Note: Processing times may vary. These are based on the serverless built-in pool that comes standard with Azure Synapse. If you create a dedicated SQL pool, you may reduce query times even further.<\/p>\n
To learn more about best practices around making dedicated SQL pools, click here<\/span><\/a>.<\/p>\n
Resources
\n<\/span><\/h3>\n
To view the full demonstration, check out the video below.<\/p>\n
\n
\n