{"id":15714,"date":"2020-11-16T15:45:00","date_gmt":"2020-11-16T23:45:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/reading-bioinformatics-and-genomics-files-in-power-bi-3\/"},"modified":"2024-04-17T09:02:30","modified_gmt":"2024-04-17T16:02:30","slug":"reading-bioinformatics-and-genomics-files-in-power-bi","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/reading-bioinformatics-and-genomics-files-in-power-bi\/","title":{"rendered":"Reading Bioinformatics and Genomics Files in Power BI"},"content":{"rendered":"

For many users in the finance, insurance, retail, manufacturing, and even healthcare industries, Power BI<\/a> has become a staple in their business intelligence plan. From interactive visualizations to advanced data wrangling, Power BI offers a one-stop shop for gaining insights from your data.<\/p>\n

However, for any of us that are researchers in the bioinformatics and genomics space, we know that our files can be a bit difficult to work with. From FASTA to BAM, working with files in bioinformatics add a layer of uniqueness that requires some special care.<\/p>\n

<\/p>\n

\"Blog_Using<\/p>\n

Today, if you take a look at Power BI Desktop’s options for getting data, you’ll see a ton of sources to which you can easily connect. One problem: none of these uniquely help us bioinformaticians.<\/p>\n

\"get_data_window\"<\/p>\n

<\/h2>\n

Putting the “Power” in Power Query<\/h2>\n

In bioinformatics, there are a plethora of file types for every occasion. Among these are very popular ones such as FASTA (or FASTQ) and BAM and, more recently, GFF3 and BGEN. We can break these data sources down into three main types:<\/p>\n\n\n\n\n
\"text-based\"<\/td>\n\"binary\"<\/td>\n\"online_sources\"<\/td>\n<\/tr>\n
Text-Based Files<\/span><\/strong><\/p>\n

Files that are human readable and can be open using virtually any text editor.<\/span><\/td>\n

Binary Files<\/strong><\/span><\/p>\n

Files that are serialized and must be read by machines.<\/span><\/td>\n

Online Sources<\/strong><\/span><\/p>\n

Databases, webpages, or FTP sites on the internet.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In Power BI, we can take advantage of Power Query to read in data and parse it appropriately. You’ll notice that, while Power BI has tons of connects to everything from CSV files to Spark clusters, there are no built-in connectors for our beloved genomics file types (yet?). So, we’ll have to use the Blank Query editor.<\/p>\n

\"get_data_window_blank_query\"<\/p>\n

<\/h3>\n

Text Files<\/h3>\n

In the query editor, we can write a custom M script to parse our files. For example, to parse a SAM file:<\/p>\n

\n\n\n\n
\n
\n
\n
\n

<\/p>\n

\n\n\n\n
\n
 1\r\n 2\r\n 3\r\n 4\r\n 5\r\n 6\r\n 7\r\n 8\r\n 9\r\n10\r\n11\r\n12\r\n13\r\n14\r\n15<\/pre>\n<\/td>\n
\n
\/\/ Read SAM Files<\/span>\r\n\r\nlet\r\n    \/\/ Read in file<\/span>\r\n    Source = Table.FromColumns({Lines.FromBinary(File.Contents(\"C:UsersColbyDocumentsGitHubbioPowerBIbam_and_samsample.sam\"<\/span>), null, null, 65001<\/span>)}),\r\n    \/\/ Skip @ lines<\/span>\r\n    #<\/span>\"Filtered Rows\"<\/span> = Table.SelectRows(Source, each not Text.StartsWith([Column1], \"@\"<\/span>)),\r\n    \/\/ Split into columns by t character and assign names<\/span>\r\n    \/\/ Note: This removes and values past the 11 standard columns<\/span>\r\n    #<\/span>\"Split Column by Delimiter\"<\/span> = Table.SplitColumn(#<\/span>\"Filtered Rows\"<\/span>, \"Column1\"<\/span>, Splitter.SplitTextByDelimiter(\"#(tab)\"<\/span>, QuoteStyle.Csv), {\"QNAME\"<\/span>,\"FLAG\"<\/span>,\"RNAME\"<\/span>,\"POS\"<\/span>,\"MAPQ\"<\/span>,\"CIGAR\"<\/span>,\"RNEXT\"<\/span>,\"PNEXT\"<\/span>,\"TLEN\"<\/span>,\"SEQ\"<\/span>,\"QUAL\"<\/span>}),\r\n    \/\/ Change data types<\/span>\r\n    #<\/span>\"Changed Type\"<\/span> = Table.TransformColumnTypes(#<\/span>\"Split Column by Delimiter\"<\/span>,{{\"QNAME\"<\/span>, type text}, {\"FLAG\"<\/span>, Int64.Type}, {\"RNAME\"<\/span>, type text}, {\"POS\"<\/span>, Int64.Type}, {\"MAPQ\"<\/span>, Int64.Type}, {\"CIGAR\"<\/span>, type text}, {\"RNEXT\"<\/span>, type text}, {\"PNEXT\"<\/span>, Int64.Type}, {\"TLEN\"<\/span>, Int64.Type}, {\"SEQ\"<\/span>, type text}, {\"QUAL\"<\/span>, type text}})\r\n\r\nin<\/span>\r\n    #<\/span>\"Changed Type\"<\/span>\r\n<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
Binary Files<\/h3>\n
We can use R or Python<\/a> to read in those pesky binary files as well. (This sometimes make even parsing text files simpler, too.) For example, to parse a BGEN file:<\/p>\n
\n\n\n\n
\n
\n
\n
<\/p>\n
\n\n\n\n
\n
1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n8<\/pre>\n<\/td>\n
\n
\/\/ Read BGEN Files<\/span>\r\n\r\nlet\r\n    \/\/ Use rbgen in an R Script to get the data from the .bgen file<\/span>\r\n    Source = R.Execute(\"library(rbgen)#(lf)file <- \"\"C:\\Users\\Colby Ford\\Desktop\\bioPowerBI\\bgen\\sample.bgen\"\"#(lf)dataset <- as.data.frame(bgen.load(file))\"<\/span>),\r\n    sample = Source{[Name=\"dataset\"<\/span>]}[Value]\r\nin<\/span>\r\n    sample\r\n<\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
…which easily makes the file available as a table.<\/p>\n
\"bgen_example\"<\/p>\n
<\/h3>\n
Online Sources<\/h3>\n
Lastly, Power BI makes it easy to grab data from the web. For example, let’s say that I wanted to get a list of all annotated genes of SARS-CoV-2<\/em><\/a> from NCBI<\/a>.<\/p>\n
\"from_web_ncbi_sc2\"I can quickly grab the URL from my browser and paste it into Power BI, which will then search the page for any tables of information.<\/p>\n
\"from_web_ncbi_sc2_tbl\"<\/p>\n
This enables users to take advantage of data from virtually any site. Try it out on the Protein Data Bank<\/a>, NCBI<\/a>, PlasmoDB<\/a>, and more!<\/p>\n
Takeaway Messages<\/h2>\n
    \n
  • Be mindful of memory. Bioinformatics files can be large and, if you\u2019re running on a machine with limited resources, you might bog it down.<\/li>\n
  • Check the defined specifications of any file format you\u2019re looking to parse.<\/li>\n
  • R or Python can be your BFF, especially for binary or really complex file types.<\/li>\n<\/ul>\n
    Demo Video<\/h2>\n
    \n
    \n