{"id":15978,"date":"2016-11-01T19:28:36","date_gmt":"2016-11-02T02:28:36","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/how-to-migrate-data-to-microsoft-azure-sql-db-or-sql-dw-2\/"},"modified":"2024-01-08T10:35:44","modified_gmt":"2024-01-08T18:35:44","slug":"how-to-migrate-data-to-microsoft-azure-sql-db-or-sql-dw","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/how-to-migrate-data-to-microsoft-azure-sql-db-or-sql-dw\/","title":{"rendered":"How to Migrate Data to Microsoft Azure SQL DB or SQL DW"},"content":{"rendered":"<p>Microsoft\u2019s <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/sql-database\/\">Azure SQL Database<\/a> and <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/sql-data-warehouse\/\">Azure SQL Data Warehouse<\/a>\u00a0promises reliability, scalability and ease of management. \u00a0With a few clicks and a few dollars, the lowly workgroup-level server can grow to consume nationwide traffic. \u00a0Such a feat is a miracle of cloud computing, but to take advantage of it, the data needs to get there. \u00a0All the computing capacity in the world is useless if it doesn\u2019t have access to the data. \u00a0This post is going to outline just a few ways you can get that data up in the cloud.<\/p>\n<p><!--more--><\/p>\n<p>Migrating a solution to Azure is far more than the simple transfer of data to the cloud. \u00a0It requires selection of the target platform, a migration strategy, security planning, database\u00a0compatibility testing, application testing and more. \u00a0In short, it&#8217;s potentially a huge topic. Here are a few notes on what will and will not be covered.<\/p>\n<ul>\n<li>This article\u00a0focuses on migrating to Azure SQL Database (v12) and Azure SQL Data Warehouse.<\/li>\n<li>It\u00a0assumes that you can transfer your data in bulk, preferably in a downtime window. Some of these methods work in incremental transfers. However, live migration is not feasible with these methods.<\/li>\n<li>Any compatibility or security issues need to be ironed out before production transfer.<\/li>\n<li>Plus, you might want to investigate ways to optimize your data structures for the new environment, particularly if you\u2019re migrating to Data Warehouse.<\/li>\n<li>There are four primary approaches to transferring the data, and each approach has multiple implementations.<\/li>\n<\/ul>\n<h3>\u00a0<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/migrate-data-azure-db.jpg\" alt=\"migrate-data-azure-db.jpg\" width=\"798\" height=\"505\" \/><\/h3>\n<h2>Approach 1 &#8211; Data Tier Export\/Import Approach<\/h2>\n<p>The data tier export\/import approach uses BACPAC files, which are essentially database backups that are compatible with Azure.\u00a0 Both methods are of limited use, but their one-shot nature might be appealing for small datasets.<\/p>\n<h3>Using a Migration Wizard<\/h3>\n<p><strong>PROS:<\/strong> Transfers database as a single unit.\u00a0 If you can use it, it\u2019s a real \u201ceasy button.\u201d<\/p>\n<p><strong>CONS:<\/strong> Requires compatible database, interactive only, no error handling<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Testing<\/p>\n<p>With the Azure SQL Database Migration Wizard or the in-preview Data Warehouse Migration Utility, you can easily migrate a database to Azure.\u00a0 It will do all the work, including creating the database and database objects as well as\u00a0transferring data.\u00a0\u00a0 It\u2019s much like an automated backup and restore.<\/p>\n<p>However, this method is of limited use.\u00a0 If your database does not pass the compatibility check, you will be forced to use another method.\u00a0 Additionally, it cannot schedule a migration for later.\u00a0 Instead, it transfers the backup while you wait.\u00a0 Finally, it cannot be used incrementally.<\/p>\n<p>Because of its limitations, it should only be used for testing purposes.<\/p>\n<h3>Export\/Import BACPAC files<\/h3>\n<p><strong>PROS:<\/strong> Transfers database as a single unit. Can be integrated into batch scripts.<\/p>\n<p><strong>CONS:<\/strong> Requires compatible database, limited error handling<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Simple and smaller deployments<\/p>\n<p>The Migration Wizard uses these BACPAC files in the background, so many of the same limitations and recommendations apply here.\u00a0 BACPAC files are essentially database backups that are compatible with Azure SQL Database.<\/p>\n<p>When divorced from the Migration Wizard, they can be more useful for deployment.\u00a0\u00a0PowerShell scripts can back up a SQL database to a BACPAC file, transfer it to Azure and deploy it to an Azure SQL database. Those scripts can be scheduled like any other PowerShell script. \u00a0Thus, it\u2019s mostly useful for small databases whose data can easily migrate within the transfer window.<\/p>\n<h2>Approach 2 \u2013 Direct Transfer<\/h2>\n<p>The direct transfer approach copies data directly from the source SQL database to its analogue in Azure.\u00a0 There are two different options for this approach, but neither significantly outperforms the other.\u00a0 The limiting factor is ultimately the network connection, so the choice comes down to other factors.\u00a0 When loading the data, be sure to scale the target\u00a0database to maximize throughput.<\/p>\n<h3>SSIS Direct Transfer<\/h3>\n<p><strong>PROS:<\/strong> Flexibility, error handling, scheduled, incremental transfers, BIML support<\/p>\n<p><strong>CONS:<\/strong> SSIS is not part of all SQL Server installations, cannot stage to Blob Storage without Azure Feature Pack<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Most deployments (up to Gigabytes)<\/p>\n<p>SSIS is a tool that many DBAs and ETL experts know and love.\u00a0 Because Azure SQL Database and Azure SQL Data Warehouse use native SQL Server connections, the existing SSIS\u00a0tool set can be used to migrate the data to Azure as if it were migrating to any new SQL Server installation.\u00a0 <a href=\"\/blog\/good-data-integration-design-patterns-are-good-for-business\">BIML<\/a> can simplify development of these loading packages immensely.<\/p>\n<p>There are no options, however, for bulk compression of the data.\u00a0 That caps the size of any single data transfer.\u00a0 However, it can incrementally transfer the data over days or weeks, if necessary.<\/p>\n<p>It\u2019s a good general-purpose solution for all but the largest of deployments.<\/p>\n<h3>Azure Data Factory (Direct Transfer)<\/h3>\n<p><strong>PROS:<\/strong> Scheduled, incremental transfers, <a href=\"\/blog\/a-patterns-based-approach-for-solving-your-data-integration-challenges\">BIML<\/a> support<\/p>\n<p><strong>CONS:<\/strong> Error handling is limited, requires Microsoft Data Management Gateway<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Small to medium deployments<\/p>\n<p>Azure Data Factory is the closest analogue to SSIS in Azure\u2019s platform.\u00a0 It is used to coordinate data transfers to or from an Azure service.\u00a0 ADF pipeline definitions can also be built with <a href=\"https:\/\/varigence.com\/biml\">BIML<\/a>.<\/p>\n<p>Azure Data Factory can easily handle large volumes.\u00a0 However, because it can so easily add a staging step as an intermediary, it\u2019s only recommended for small to medium deployments when using a direct transfer.<\/p>\n<h2>Approach 3 \u2013 Staging in Azure Blob Storage<\/h2>\n<p>This approach breaks the deployment into\u00a0two steps: staging and load.\u00a0 The staging step migrates your data from your server to Blob Storage, and the load step gets it from staging in Blob Storage to its destination.\u00a0 It\u2019s useful for larger deployments.<\/p>\n<h3>Staging<\/h3>\n<h3>SSIS with Azure Feature Pack<\/h3>\n<p><strong>PROS:<\/strong> Flexibility, error handling, scheduled, incremental transfers, <a href=\"\/webinar\/improve-sql-server-data-warehouse-delivery-with-a-patterns-based-approach\">BIML<\/a> support<\/p>\n<p><strong>CONS:<\/strong> SSIS is not part of all SQL Server installations, requires additional Azure Feature Pack, not appropriate for landing in Azure SQL DB<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Large deployments to Azure SQL Data Warehouse<\/p>\n<p>With SSIS in a staged approach, it is only appropriate to use with PolyBase to land in Azure SQL Data Warehouse.\u00a0 But it does make an effective tool for the use case.\u00a0 It can use data flows to stage the data in Blob Storage and issue queries against Azure DW to use PolyBase external tables.\u00a0 Again, BIML makes automating package development much easier.<\/p>\n<h3>Azure Data Factory<\/h3>\n<p><strong>PROS:<\/strong> Scheduled, incremental transfers, BIML support<\/p>\n<p><strong>CONS:<\/strong> Error handling is limited, requires Microsoft Data Management Gateway<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Large deployments<\/p>\n<p>All of the previous comments about ADF apply here.\u00a0 Adding a staging step in Blob Storage to an ADF pipeline is easy.\u00a0 Its biggest draw here is that it\u2019s the only tool here that natively stages in Blob Storage and loads Azure SQL DB or DW in a single pipeline.<\/p>\n<p>If you\u2019re staging to Blob Storage over a network, this is the easiest one to set up and use.<\/p>\n<h3>Azure Import\/Export Service<\/h3>\n<p><strong>PROS:<\/strong> Data throughput, does not use network resources<\/p>\n<p><strong>CONS:<\/strong> Only useful for large batches of data, lag, complexity of export\/import<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Huge deployments<\/p>\n<p>The Azure Import\/Export Service allows a customer to ship hard drives to an Azure datacenter, where the contents are loaded to Blob Storage within the datacenter itself.\u00a0\u00a0 This is potentially faster than most any network connection can handle, so long as the volume of data is extremely high.<\/p>\n<p>Here, a user would export the database contents to flat files using bcp or SSIS packages. They\u2019d transfer the files to hard drives and ship them to Microsoft.<\/p>\n<p>For less than the recommended 20 TB data transfers, it\u2019s less useful.\u00a0 There\u2019s shipping and loading lag, fragmentation of the process, and less control.\u00a0 However, the decreased load on networks means that Microsoft recommends this method with volumes greater than 20 TB.<\/p>\n<h2>Load<\/h2>\n<h3>PolyBase<\/h3>\n<p><strong>PROS:<\/strong>\u00a0SQL queries, transfer speed<\/p>\n<p><strong>CONS:<\/strong> Does not work with Azure SQL DB<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Azure DW<\/p>\n<p>When using Azure Data Warehouse, PolyBase is the fastest way to import data from Blob Storage. PolyBase can read the flat files as tables, making import just 2-3 SQL queries per table.\u00a0 It\u2019s best paired with scripting to automate the process.<\/p>\n<h2>Approach 4 \u2013 Staging in a SQL Server Azure Virtual Machine<\/h2>\n<p><strong>PROS:<\/strong> Transfer efficiency<\/p>\n<p><strong>CONS:<\/strong> Multi-step process<\/p>\n<p><strong>APPROPRIATE FOR:<\/strong> Large to huge deployments<\/p>\n<p>Network bandwidth within data centers is much greater than bandwidth to the Internet at large.\u00a0 For huge SQL databases, it may be desirable to transfer a compressed backup of the data to a temporary SQL Server VM in Azure.\u00a0 There, you can transfer the data to an Azure Database using internal bandwidth, resulting in a net speedup.<\/p>\n<p>It\u2019s useful for larger deployments only.<\/p>\n<p>You can transfer the database backup files using any file transfer mechanism, as the endpoint is another Windows (virtual) machine.\u00a0 You can use ftp, sftp, Windows file sharing, or even the Azure Import\/Export Service.\u00a0 \u00a0Once the backups are restored in the VM, any of the direct transfer methods are applicable.<\/p>\n<h4>Wrapping It Up<\/h4>\n<p>We\u2019ve gone through\u00a0four approaches to transfer your data into an Azure SQL Database or SQL Data Warehouse.\u00a0 This outlines appropriate solutions for all sizes of transfers, from the smallest to the largest.\u00a0 When in doubt, use the tools you\u2019re most comfortable using.\u00a0 What you might potentially lose in transfer efficiency is often gained back in speed of development and QA resources.<\/p>\n<p>Please <a href=\"\/get-started\/\">contact us<\/a> for more details or if you&#8217;re looking for design or delivery support for your Azure DB or Azure DW project.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Microsoft\u2019s Azure databases promise users reliability, scalability and ease. This post outlines just a few ways you can get that data up in the cloud.<\/p>\n","protected":false},"author":21,"featured_media":14769,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[304],"class_list":["post-15978","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-modern-data-platform","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15978","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15978"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15978\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/14769"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15978"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15978"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15978"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}