![\"ZeppelinChart.png\"](\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/ZeppelinChart.png\")
<\/p>\nSome of the key tasks in data science involve basic exploration of new or existing data.\u00a0 Raw data is given structure, data can be joined to other datasets, features are selected for later analysis, and much more.\u00a0 Depending on the questions to which you seek answers as well as other requirements, the process repeats until you have data that is ideal for further, more advanced, analytics.<\/p>\n
<\/p>\n
With Apache Spark on Azure HDInsight<\/a>, these core tasks are made simpler with the inclusion of both the Apache Zeppelin and Jupyter notebooks.\u00a0 In this Demo Day video, I walk through basic exploration of a city’s traffic crash history using Zeppelin with both Spark DataFrames and Spark SQL.\u00a0 I discuss some of the advantages of using Zeppelin and Spark for data of any volume.\u00a0 Working with a new text file, I obtain an initial look at what features are available, see what cleansing may need to take place, and obtain a basic feel for the dataset through querying and visualization.\u00a0 At this stage, I compute summary statistics as well as develop a repeatable process that can be used later.\u00a0 While this is descriptive analysis, how can the data be prepared for other applications such as predictive analytics<\/a>?<\/p>\n Overall, I can use the data to help bring me closer to answering my initial questions as well as prompt new questions.\u00a0 For example:<\/p>\n View the video below to see how the Zeppelin notebook on a Spark on Azure HDInsight cluster can help me get answers.<\/p>\n\n