{"id":15903,"date":"2017-11-16T13:33:00","date_gmt":"2017-11-16T21:33:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/microsoft-azure-databricks-cloud-scale-spark-power-2\/"},"modified":"2024-01-08T10:43:53","modified_gmt":"2024-01-08T18:43:53","slug":"microsoft-azure-databricks-cloud-scale-spark-power","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/microsoft-azure-databricks-cloud-scale-spark-power\/","title":{"rendered":"Microsoft Azure &#038; Databricks\u00a0= Cloud-Scale Spark Power"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" style=\"display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/azure-databricks-1.png\" alt=\"azure databricks.png\" width=\"805\" height=\"509\" \/><\/p>\n<p>Recently, Microsoft and\u00a0<a href=\"https:\/\/databricks.com\/\" rel=\" noopener\">Databricks<\/a>\u00a0made an exciting announcement around their partnership that will soon result in a cloud-based, managed Spark service on\u00a0<a href=\"https:\/\/azure.microsoft.com\/en-us\/\" rel=\" noopener\">Azure<\/a>. Currently, some select customers are allowed into a &#8220;private preview&#8221; mode of the service, and over the next few weeks, a &#8220;gated public preview&#8221; will ensue for around 150 clients. In January 2018, the service will be available for everyone to try. While the full details are not known about the partnership or full features of the platform, here is how Azure Databricks will likely enhance your Big Data capabilities in the cloud.<\/p>\n<p><!--more--><\/p>\n<table style=\"margin-left: auto; margin-right: auto;\">\n<tbody>\n<tr>\n<td><a style=\"width: 319px;\" href=\"https:\/\/databricks.com\/\" target=\"_blank\" rel=\"noopener\" data-mce-target=\"_blank\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/databricks-1.png\" alt=\"databricks-1.png\" width=\"319\" height=\"69\" \/><\/a><\/td>\n<td>\n<h2><span style=\"color: #808080;\"><strong>+<\/strong><\/span><\/h2>\n<\/td>\n<td><a href=\"https:\/\/azure.microsoft.com\/en-us\/\" target=\"_blank\" rel=\"noopener\" data-mce-target=\"_blank\"><img decoding=\"async\" style=\"width: 400px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/Azurelgoo.png\" alt=\"Azure\" width=\"400\" \/><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>What is Databricks?<\/h2>\n<p>Databricks is a company that was started by the team that originally created Spark at UC Berkeley. They have created a <a href=\"https:\/\/databricks.com\/product\/unified-analytics-platform\" target=\"_blank\" rel=\"noopener\">Unified Analytics Platform<\/a> that aims to be the single system for everything from analytics workflows to Spark integration to security.<\/p>\n<p>Databricks boasts various benefits of their Unified Analytics Platform such as:<\/p>\n<ul>\n<li><span style=\"background-color: transparent;\"><strong>UNIFY ANALYTICS WITH APACHE SPARK<\/strong> &#8211; Eliminate the need for disparate tools.<\/span><\/li>\n<li><span style=\"background-color: transparent;\"><strong>STREAMLINE ANALYTIC WORKFLOWS<\/strong> &#8211;\u00a0<\/span><span style=\"background-color: transparent;\">Reduce deployment time to minutes.<\/span><\/li>\n<li><span style=\"background-color: transparent;\"><strong>INCREASE PRODUCTIVITY OF DATA SCIENCE TEAMS<\/strong> &#8211;\u00a0<\/span><span style=\"background-color: transparent;\">With Databricks, they\u2019ll be 5x more productive.<\/span><\/li>\n<li><span style=\"background-color: transparent;\"><strong>REDUCE RISK<\/strong> &#8211;\u00a0<\/span><span style=\"background-color: transparent;\">Enable innovation with out-of-the-box enterprise security and compliance.<\/span><\/li>\n<\/ul>\n<p style=\"text-align: center;\"><img decoding=\"async\" style=\"display: block; margin-left: auto; margin-right: auto;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/marchitecture.png\" alt=\"databricks workspace\" \/><em style=\"background-color: transparent; font-size: 10px;\">[<a href=\"https:\/\/databricks.com\/product\/unified-analytics-platform\" target=\"_blank\" rel=\"noopener\">Source<\/a>]<\/em><\/p>\n<blockquote>\n<h5 style=\"text-align: center;\"><span style=\"font-size: 24px;\">Apache\u00ae Spark\u2122 on Databricks is said to have a 5x performance gain over that of the open-source version.<\/span><\/h5>\n<\/blockquote>\n<p style=\"text-align: left;\">Looking at the\u00a0<a href=\"https:\/\/databricks.com\/feature-comparison\" target=\"_blank\" rel=\"noopener\">Databricks&#8217; Feature Comparison<\/a> page, there are quite a few features that could likely make it into the Azure version in the near future.<\/p>\n<table>\n<tbody>\n<tr>\n<td valign=\"top\">\n<h3>CLOUD OPTIMIZATION:<\/h3>\n<ul>\n<li>Tuned Apache\u00ae Spark\u2122 clusters<\/li>\n<li>High availability for Spark Streaming<\/li>\n<li>Built-in file system<\/li>\n<\/ul>\n<h3>COST MANAGEMENT:<\/h3>\n<ul>\n<li>Autoscaling Apache Spark clusters<\/li>\n<li>Multi-user cluster sharing<\/li>\n<\/ul>\n<h3>BUILT-IN EXPLORATION TOOLS:<\/h3>\n<ul>\n<li>Notebooks with real-time collaboration + revision history<\/li>\n<li>Publish notebooks as production dashboards<\/li>\n<\/ul>\n<\/td>\n<td valign=\"top\">\n<h3>BUILT-IN PRODUCTION TOOLS:<\/h3>\n<ul>\n<li>Spark job monitoring alerts<\/li>\n<li>One-click deployment from notebooks to Spark Jobs<\/li>\n<li>APIs to build workflows in notebooks<\/li>\n<\/ul>\n<h3>SECURITY:<\/h3>\n<ul>\n<li>Access control for clusters and notebooks<\/li>\n<li>Permission-based job and workflow execution<\/li>\n<li>Authenticated SQL server<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Expect a Familiar Azure Experience<\/h2>\n<p style=\"text-align: left;\">Azure already has a managed Hadoop\u2122 offering known as HDInsight. You can spin up a custom HDInsight cluster with your specifications from the Portal. Then, you pay for the time that you have your cluster running. Support for HDInsight is provided by the Microsoft Azure support team.<\/p>\n<p style=\"text-align: left;\">As for <a href=\"\/azure-databricks\" target=\"_blank\" rel=\"noopener\">Azure Databricks<\/a>, the experience will be very similar.\u00a0Simply spin up an Azure Databricks cluster directly from the Portal and Azure will do the setup work for you.\u00a0No licensing is required other than your Azure subscription.<\/p>\n<p style=\"text-align: left;\">For support, Microsoft and Databricks will have a seamless system for users to get help with their individual needs. Since the service is within Azure, you will go through Microsoft for support, which will now be fully integrated with the Databricks expert support team.<\/p>\n<p style=\"text-align: left;\">Want to learn more about how you can take advantage of this exciting announcement at your organization?<a href=\"\/get-started\/\"> Contact 3Cloud<\/a> today!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Microsoft &amp; Databricks announced their new partnership which adds an optimized, managed Spark service to the Azure cloud &#8211; preview available in January 2018.<\/p>\n","protected":false},"author":21,"featured_media":14519,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[329],"class_list":["post-15903","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-azure-databricks","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15903"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15903\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/14519"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}