{"id":15686,"date":"2021-05-17T13:15:00","date_gmt":"2021-05-17T20:15:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/better-together-blog-series-emphasizing-data-engineering-3\/"},"modified":"2023-12-22T14:00:53","modified_gmt":"2023-12-22T22:00:53","slug":"better-together-blog-series-emphasizing-data-engineering","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/better-together-blog-series-emphasizing-data-engineering\/","title":{"rendered":"Better Together Blog Series – Emphasizing Data Engineering"},"content":{"rendered":"

In part 3 of our “Better Together” blog series, I will recap my presentation from a series of BlueGranite Tech and Career Talks conducted in partnership with BDPA<\/a> (formerly known as ‘Black Data Processing Associates’\u200b), in which I, along with my fellow Data Scientist Dr. Tom Weinandy<\/a>, talked about \u201cA Day in the Life of a Data Engineer and Data Scientist<\/a>\u201d. Since both of our backgrounds are as data scientists, I focused my section on how data engineering is not only intertwined with data science<\/a>, but is in fact an essential prerequisite.<\/p>\n

<\/span><\/h2>\n

\"iStock-1271619512-1\"<\/p>\n

Data Engineering Overview
\n<\/span><\/h3>\n

The key roles of data engineers are to build and maintain an organization\u2019s data pipeline as well as clean and wrangle the data into usable sets for analytical purposes. Real world data is normally messy, unstructured, and derived from multiple sources without a clearing understanding of associations. A data engineers\u2019 job is to take this web of data and weave it into a sustainable process<\/span> that produces a data set that data scientists can not only work with, but that the company can understand. As the saying goes, garbage in equals garbage out; a statistical model can only perform as well as the input data it receives.<\/p>\n

While some companies rely on data scientists to perform these essential steps, as the data and technology expands, the person building the algorithm should not be the same<\/a><\/span> as the person building the data pipeline. Both pursuits require in-depth knowledge of specific technologies and domains that, while overlapping, require differing degrees of comprehension. Furthermore, a data scientist would design and clean data with the bias of data for data modeling purposes, even if the pipeline needs to be designed with multiple use cases as the goal.<\/p>\n

<\/p>\n

Types of Data
\n<\/span><\/h3>\n

Adding further to the challenge of data engineering, real world data comes in a variety of formats. These formats can be classified into two main categories: Structured and Unstructured. Structured data refers to data saved in a fixed field or record. It is generally easier to work with this type of data. Unstructured data is essentially every other piece of data; it tends to be more complex and lacking a data model. The chart below highlights some of the key differences between structured and unstructured data.<\/p>\n

\n\n\n\n\n\n\n\n
Structured<\/strong><\/span><\/td>\nUnstructured<\/strong><\/span><\/td>\n<\/tr>\n
Clearly defined data types<\/td>\nNo predefined data model<\/td>\n<\/tr>\n
Stored in rows\/columns<\/td>\nStored in native format (i.e., audio file)<\/td>\n<\/tr>\n
Usually quantitative<\/td>\nQualitative<\/td>\n<\/tr>\n
Ease of analysis<\/td>\nRequires preprocessing (i.e., data mining blogs\/social media)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

A data engineer must be able to process both types of data, as well as discover meaningful relationships between the data.<\/p>\n<\/div>\n

Tools of the Trade
\n<\/span><\/h3>\n

To accomplish these feats of data magic, data engineers rely on several tools<\/span><\/a> on both the transformation and storage side. The most basic data engineering pipeline involves accessing data in its raw format, performing data wrangling\/cleaning tasks, and then saving the cleaned data to a centralized location. Python, SQL, and Java are coding languages commonly used to perform all steps of this pipeline. Enabling Apache Spark with these coding languages allows for high performance with large-scale data. For a less code intensive solution, Azure Data Factory<\/span><\/a> allows data engineers to construct pipelines with built in features or through making calls to code-based notebooks.<\/p>\n

<\/p>\n

On the storage side, many options are also at a data engineer\u2019s disposal. The challenge is selecting the correct format based not only on the type and amount of data, but on the stakeholder\u2019s intended usage and budget. Common storage platforms include Azure Data Lake<\/span><\/a> and SQL Database. Data can also be stored On-Prem or in a Cloud-Based<\/span><\/a> solution.<\/p>\n

In Summary<\/span><\/h2>\n

Overall, a data engineer\u2019s role is complex and involves multiple factors. However, without data engineers, data scientists would struggle to optimally perform their role. To get a more in depth understanding of data engineering, as well as see real case study examples, watch the BDPA video recording below.<\/p>\n

\n
\n