{"id":15927,"date":"2017-07-18T13:05:00","date_gmt":"2017-07-18T20:05:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/now-presenting-fun-with-functions-string_split-2\/"},"modified":"2023-09-20T09:36:02","modified_gmt":"2023-09-20T16:36:02","slug":"now-presenting-fun-with-functions-string-split","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/now-presenting-fun-with-functions-string-split\/","title":{"rendered":"Now Presenting: Fun with Functions (String_Split)"},"content":{"rendered":"

In my 17 years of database development, a few projects stand out from requirements so elaborate that I really had to stretch my imagination. One such assignment required the importation of comma separated value (CSV) files where the fields may not be in the same place and new fields may appear without warning. As you can imagine, this wouldn\u2019t\u00a0have been\u00a0difficult if the source was a semi-structured format like JSON, but in this case, it was\u00a0good old CSV.<\/p>\n

\"String_Split<\/p>\n

Thanks to the new functions within SQL Server 2016, this once-daunting task becomes easily doable\u00a0when you combine dynamic SQL statements and String_Split. String_Split is a unique function where you can take a list of strings and parse them into rows. See the example from MSDN<\/a> below:<\/p>\n

\n
\n
\n
\n

SELECT value
\n<\/code>FROM STRING_SPLIT(‘Lorem ipsum dolor sit amet.’, ‘ ‘);<\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\n\n\n
value<\/strong><\/td>\n<\/tr>\n
Lorem<\/td>\n<\/tr>\n
ipsum<\/td>\n<\/tr>\n
dolor<\/td>\n<\/tr>\n
sit<\/td>\n<\/tr>\n
amet.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

If you are already thinking that this will provide you with a separate row for each field name of the CSV, you are headed in the right direction. String_Split can provide a glimpse into the format of any CSV file, and with the help of the Pivot operator and dynamic SQL statements, you can build a process to handle changing file structures or the addition of new fields without having to rewrite your code.<\/span><\/p>\n

Technical Requirements<\/h2>\n

In this instance, sensor data is coming from a 3rd<\/sup> party vendor, and after it is processed by a proprietary tool, a CSV file is cut for the client to consume. Per the specifications, only the 1st<\/sup> field is consistent and will contain a date, but all other fields may be in a different order or include never-before-seen fields. Another important piece of information to make note of is that all fields, aside from the date, will be floats (large decimals).<\/p>\n\n\n\n\n
date<\/strong><\/td>\nsensor1<\/strong><\/td>\nsensor2<\/strong><\/td>\nsensor3<\/strong><\/td>\nsensor4<\/strong><\/td>\n<\/tr>\n
19-06-2017\u00a012:00:00.500<\/td>\n0.1<\/td>\n5.0<\/td>\n600.0<\/td>\n1700.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

File Example 1<\/em><\/strong><\/p>\n\n\n\n\n
date<\/strong><\/td>\nsensor5<\/strong><\/td>\nsensor1<\/strong><\/td>\nsensor3<\/strong><\/td>\nsensor17<\/strong><\/td>\n<\/tr>\n
20-06-2017\u00a012:00:00.500<\/td>\n6.5<\/td>\n21.0<\/td>\n168.5<\/td>\n2.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

File Example 2<\/em><\/strong><\/p>\n

High-Level Approach<\/h2>\n

Let\u2019s walk through each step of the process, using the file examples above as a guide. Below is a summary of the steps we will take with more details to follow further down.<\/p>\n

Step 1:<\/strong> Import CSV into a single-column staging table
\nStep 2:<\/strong> Parse 1st<\/sup> row into a staging metadata table (field\u00a0names)
\nStep 3:<\/strong> Compare staging metadata to historic metadata and add new fields if necessary
\nStep 4:<\/strong> Assemble dynamic pivot query to parse and insert values into staging table
\nStep 5:<\/strong> Assemble dynamic query to insert into production table\u00a0<\/span><\/p>\n

\"String_Split.png\"<\/span><\/p>\n

Step 1:<\/h3>\n

Import the CSV into a staging table so that all the data is in 1 column. This can be done by using an obscure delimiter that doesn\u2019t appear in the file like \u2018|\u2019 or \u2018~\u2019. The point is to keep all the data together so that String_Split can be used on the entire row of data.<\/p>\n

Step 2:<\/h3>\n

Capture the first row of data into a variable and use the String_Split function to capture the Fields into a staging metadata table (example code below). The staging metadata table is very important when creating the dynamic Insert<\/span><\/code>\u00a0\/\u00a0Select<\/span><\/code>\u00a0statements to persist the correct order of the fields since they can vary.<\/p>\n

\n
\n
\n
\n

DECLARE<\/span> @Header VARCHAR<\/span>(<\/span>5000)<\/span>, @FieldName VARCHAR<\/span>(<\/span>5000<\/span>);<\/span><\/span>
\n\u00a0 \u00a0 \u00a0SELECT<\/span> @Header = <\/span>DataTXT
\n<\/span>\u00a0 \u00a0 \u00a0FROM<\/span> stage.<\/span>sensor
\n<\/span>\u00a0 \u00a0 \u00a0WHERE<\/span> RowID =<\/span> 1
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0AND <\/span>FileID =<\/span> @FileID;<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0WHILE<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0(<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0SELECT<\/span> CHARINDEX<\/span>(<\/span>‘,’<\/span>,<\/span> @Header,<\/span> 1)<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0) ><\/span> 1
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0BEGIN<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0SELECT<\/span> @FieldName = LEFT(<\/span>@Header,<\/span> CHARINDEX<\/span>(<\/span>‘,’<\/span>,<\/span> @Header,<\/span> 1)-<\/span>1);<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0INSERT INTO<\/span> stage.<\/span>metadata
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0(<\/span>FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 FieldName
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0)<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0VALUES<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0(<\/span>@FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 @FieldName
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0);<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0SET<\/span> @Header =<\/span> SUBSTRING<\/span>(<\/span>@Header,<\/span> CHARINDEX(<\/span>‘,’<\/span>,<\/span> @Header,<\/span> 1)+<\/span>1,<\/span> 5000);<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0END<\/span>;<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0INSERT INTO<\/span> stage.<\/span>metadata
\n<\/span>\u00a0 \u00a0 \u00a0(<\/span>FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 FieldName
\n<\/span>\u00a0 \u00a0 \u00a0)<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0VALUES<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0(<\/span>@FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 @Header
\n<\/span>\u00a0 \u00a0 \u00a0);<\/span><\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Step 3:<\/h3>\n

After capturing the fields from the incoming file, you should compare them to the existing metadata fields expected in the resulting table that the data will be inserted into. The following code creates dynamic SQL statements that can be executed against the production table to add additional fields. Since it\u2019s a given that the fields are all floats except \u2018date\u2019, then you don\u2019t have to worry about conversion issues. However, we worked out a solution that added an additional field to the metadata table indicating the data type for future flexibility.<\/p>\n

\n
\n
\n
\n

DECLARE cur_fields CURSOR FORWARD_ONLY
\nFOR SELECT
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0s.<\/span>FieldName,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0‘Alter Table dbo.sensor ADD [‘<\/span>+<\/span>s.<\/span>FieldName+<\/span>‘] FLOAT’<\/span> AS Remediation
\n<\/span>\u00a0 \u00a0 FROM stage.<\/span>metadata AS s
\n<\/span>\u00a0 \u00a0 WHERE NOT EXISTS<\/span>
\n<\/span>\u00a0 \u00a0 (<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 SELECT
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0*<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 FROM dbo.<\/span>metadata AS m
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 WHERE s.<\/span>FieldName =<\/span> m.<\/span>FieldName
\n<\/span>\u00a0 \u00a0 )<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 AND<\/span> s.<\/span>FileID =<\/span> @FileID
\n<\/span>
\nDECLARE @FieldName VARCHAR(<\/span>500),<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 @sqlAlter\u00a0 NVARCHAR(<\/span>1000);<\/span>
\n<\/span>
\n<\/span>OPEN cur_fields;<\/span>
\n<\/span>
\n<\/span>FETCH NEXT FROM cur_fields INTO @FieldName,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 @sqlAlter;<\/span>
\n<\/span>
\n<\/span>WHILE @@FETCH_STATUS<\/span> =<\/span> 0
\n<\/span>\u00a0 \u00a0 BEGIN
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 INSERT INTO dbo.<\/span>metadata_history
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 (<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0FieldName
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 )<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 VALUES
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 (<\/span>@FileID,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0@FieldName
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 );<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 INSERT INTO dbo.<\/span>metadata
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 (<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0FieldName
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 )<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 VALUES(<\/span>@FieldName);<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 EXEC sp_executesql<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0@sqlAlter;<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 FETCH NEXT FROM cur_fields INTO @FieldName,<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 @sqlAlter;<\/span>
\n<\/span>\u00a0 \u00a0 END;<\/span><\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Steps 4 & 5:<\/h3>\n

The last 2 steps are accomplished using 1 procedure as shown in the code below. The first part will use the metadata tables to pull out the fields necessary for the Select<\/code><\/span>\u00a0and the Insert<\/code><\/span>. A dynamic SQL statement is then created to parse the data within the CSV, skipping the header row. The final dynamic SQL statement creates a Merge based on some other client requirements, but a simple insert would also work depending on your situation.<\/p>\n

\n
\n
\n
\n

DECLARE<\/span> @sqlCMD\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 NVARCHAR<\/span>(<\/span>MAX<\/span>),<\/span>
\n\u00a0 \u00a0 \u00a0 \u00a0 @sqlHeader\u00a0\u00a0\u00a0 NVARCHAR<\/span>(<\/span>MAX<\/span>),<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 @sqlStage\u00a0\u00a0\u00a0\u00a0 NVARCHAR<\/span>(<\/span>MAX<\/span>),<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 @Fields\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 NVARCHAR<\/span>(<\/span>MAX<\/span>),<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 @SourceFields NVARCHAR<\/span>(<\/span>MAX<\/span>),<\/span>
\n<\/span>\u00a0 \u00a0 \u00a0 \u00a0 @SetFields\u00a0\u00a0\u00a0 NVARCHAR<\/span>(<\/span>MAX<\/span>);<\/span>
\n<\/span>
\n<\/span>SET<\/span> @Fields =<\/span> ”<\/span>;<\/span>
\n<\/span>SET<\/span> @SourceFields =<\/span> ”<\/span>;<\/span>
\n<\/span>SET<\/span> @SetFields =<\/span> ”<\/span>;<\/span>
\n<\/span>
\n<\/span>SELECT<\/span>
\n<\/span>\u00a0 \u00a0@Fields =<\/span> @Fields+<\/span>‘[‘<\/span>+<\/span>FieldName+<\/span>‘], ‘<\/span>,<\/span>
\n<\/span>\u00a0 \u00a0@SourceFields =<\/span> @SourceFields+<\/span>‘source.[‘<\/span>+<\/span>FieldName+<\/span>‘], ‘<\/span>,<\/span>
\n<\/span>\u00a0 \u00a0@SetFields =<\/span> @SetFields+<\/span>‘target.[‘<\/span>+<\/span>FieldName+<\/span>‘] = source.[‘<\/span>+<\/span>FieldName+<\/span>‘], ‘<\/span>
\n<\/span>FROM<\/span> stage.<\/span>metadata
\n<\/span>WHERE<\/span> FileID =<\/span> @FileID
\n<\/span>\u00a0 \u00a0 \u00a0 AND<\/span> FieldID ><\/span> 0;<\/span>
\n<\/span>
\n<\/span>SET<\/span> @Fields = LEFT(<\/span>@Fields,<\/span> LEN<\/span>(<\/span>@Fields) –<\/span> 1);<\/span><\/p>\n

<\/span>SET<\/span> @SourceFields = LEFT(<\/span>@SourceFields,<\/span> LEN<\/span>(<\/span>@SourceFields) –<\/span> 1);<\/span><\/p>\n

<\/span>SET<\/span> @SetFields = LEFT(<\/span>@SetFields, LEN<\/span>(<\/span>@SetFields) –<\/span> 1);<\/span><\/p>\n

<\/span>TRUNCATE TABLE<\/span> stage.<\/span>sensor_parsed;<\/span><\/p>\n

<\/span>SET<\/span> @sqlHeader =<\/span> ‘WITH RawData(RowID, FieldID, Date, SensorValue) AS (SELECT RowID, ROW_NUMBER() OVER(PARTITION BY RowID ORDER BY RowID) AS FieldID, LEFT(DataTXT, CHARINDEX(‘<\/span>+<\/span>””<\/span>+<\/span>‘,’<\/span>+<\/span>””<\/span>+<\/span>‘, DataTXT, 1)-1) AS Date, value FROM stage.sensor s CROSS APPLY string_split (SUBSTRING(DataTXT, CHARINDEX(‘<\/span>+<\/span>””<\/span>+<\/span>‘,’<\/span>+<\/span>””<\/span>+<\/span>‘, DataTXT, 1)+1, 5000), ‘<\/span>+<\/span>””<\/span>+<\/span>‘,’<\/span>+<\/span>””<\/span>+<\/span>‘) WHERE FileID = ‘<\/span>+<\/span>CONVERT<\/span>(<\/span>VARCHAR<\/span>(<\/span>10),<\/span> @FileID)+<\/span>‘ AND RowID >= 1 and isnumeric(value) = 1), Result(RowID, Date, FieldID, FieldName, SensorValue) AS (SELECT r.RowID, Date, m.FieldID, m.FieldName, r.SensorValue FROM stage.metadata AS m JOIN RawData AS r ON m.FieldID = r.FieldID where m.FileID = ‘<\/span>+<\/span>CONVERT<\/span>(<\/span>VARCHAR<\/span>(<\/span>10),<\/span> @FileID)+<\/span>‘) ‘<\/span>;<\/span><\/p>\n

<\/span>SET<\/span> @sqlStage =<\/span> ‘insert into stage.sensor_parsed (RowID, Date, EquipmentID, FieldID, FieldName, SensorValue) select RowID, Date, FieldID, FieldName, SensorValue from Result where SensorValue is not NULL’<\/span>;<\/span><\/p>\n

<\/span>SET<\/span> @sqlCMD =<\/span> @sqlHeader +<\/span> @sqlStage;<\/span><\/p>\n

<\/span>EXEC<\/span> sp_executesql<\/span>
\n<\/span>\u00a0 \u00a0@sqlCMD;<\/span><\/p>\n

<\/span>SET<\/span> @sqlCMD =<\/span> ”<\/span>;<\/span><\/p>\n

<\/span>SELECT<\/span>
\n<\/span>\u00a0 \u00a0@sqlCMD =<\/span> ‘merge dbo.sensor as target using (SELECT ‘<\/span>+<\/span>CONVERT<\/span>(<\/span> VARCHAR<\/span>(<\/span>10),<\/span> @FileID)+<\/span>‘, [Date], ‘<\/span>+<\/span>@Fields+<\/span>‘,\u00a0 getdate() as [CreateDT] from (select [RowID], [Date], [FieldName], [SensorValue] from stage.sensor_parsed ) as s Pivot(Max([SensorValue]) for [FieldName] in (‘<\/span>+<\/span>@Fields+<\/span>‘)) as p) as source ([FileID], [Date], ‘<\/span>+<\/span>@Fields+<\/span>‘, [CreateDT]) on (target.[Date] = source.[Date] and target.[FileID] = source.[FileID]) when matched then update set ‘<\/span>+<\/span>@SetFields+<\/span>‘ when not matched then insert([FileID], [Date], ‘<\/span>+<\/span>@Fields+<\/span>‘, [CreateDT]) values (source.[FileID], source.[Date], ‘<\/span>+<\/span>@SourceFields+<\/span>‘, source.[CreateDT]);’<\/span>;<\/span><\/p>\n

<\/span>EXEC<\/span> sp_executesql<\/span>
\n<\/span>\u00a0 \u00a0@sqlCMD;<\/span><\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Summary<\/h4>\n

These project requirements really drove the direction of this process, resulting in a SQL table that can expand horizontally as new fields come in through the CSV. However, there are endless scenarios making for an even simpler process, assuming requirements differ from the above-mentioned project, and you have the freedom to consider other storage methods, such as JSON.<\/p>\n

String_Split is a fantastic new tool for dealing with edge cases and in this engagement, it provided a fairly elegant solution which would have been next to impossible in prior versions of SQL Server.<\/p>\n

Have questions about SQL? Our team of experts would love to chat with you<\/a>!<\/p>\n","protected":false},"excerpt":{"rendered":"

New functions within SQL Server 2016 allow you to combine dynamic SQL statements & String_Split, allowing you to take a list of strings & parse them into rows.<\/p>\n","protected":false},"author":21,"featured_media":14631,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[395,260],"tags":[319],"class_list":["post-15927","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science-ai","category-data-ai","tag-machine-learning-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15927","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15927"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15927\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/14631"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15927"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15927"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15927"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}