{"id":16096,"date":"2015-02-26T14:56:00","date_gmt":"2015-02-26T22:56:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/how-to-process-google-data-in-real-time-with-azure-stream-analytics-2\/"},"modified":"2024-01-04T16:28:37","modified_gmt":"2024-01-05T00:28:37","slug":"how-to-process-google-data-in-real-time-with-azure-stream-analytics","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/how-to-process-google-data-in-real-time-with-azure-stream-analytics\/","title":{"rendered":"How to Process Google Data in Real Time with Azure Stream Analytics"},"content":{"rendered":"<div class=\"hs-migrated-cms-post\">\n<p><span style=\"font-size: x-small;\"><span style=\"font-size: x-small;\"><img decoding=\"async\" id=\"img-1425392052005\" class=\"alignRight\" style=\"float: right;\" src=\"\/\/cdn2.hubspot.net\/hub\/257922\/file-2770908533-png\/images\/windows_azure_logo.png\" alt=\"Windows Azure\" border=\"0\" \/><\/span><\/span><\/p>\n<p>Whether you call it streaming analytics, complex event processing or the Internet of Things, capturing and analyzing large volumes of data in real-time has always been a challenge. Streaming analysis solutions are required to \u201cdrink from a fire hose\u201d of data without losing a drop. They must capture every message, ensure that no message is duplicated and ensure that they are processed in the right order. This type of solution can be extremely complex and can significantly tax computer and network resources. Specialized infrastructure may be required that is expensive and managing scale and fault tolerance is difficult.<\/p>\n<p><!--more--><\/p>\n<p>Microsoft\u2019s Azure cloud platform provides many features and services that make building complex applications easy, affordable and scalable. Azure is like a collection of building blocks that allow architects to assemble complex solutions quickly, experiment with them and tear them down and rearrange them in new ways.<\/p>\n<p><img decoding=\"async\" style=\"width: 5700px;\" src=\"https:\/\/3cloudsolutions.com\/wp-content\/uploads\/2022\/11\/iStock-504455506.jpg\" alt=\"iStock-504455506\" width=\"5700\" \/><\/p>\n<p>Many organizations use Google Analytics to understand how their web properties are being used. Google Analytics offers reports and API\u2019s to allow subscribers to perform a wide variety of analysis of their web traffic. I built a solution that demonstrates how to use the components of Azure to take data from the Google Analytics API and analyze what pages are being viewed on BlueGranite\u2019s video blog site in real-time.<\/p>\n<p>The specific components of Azure used in this solution include:<\/p>\n<ul>\n<li>Azure Worker Role \u2013 a worker role is a piece of published code that runs in an Azure Virtual Machine. Worker Roles are specifically designed to run background processing tasks and don\u2019t have a web interface. The virtual machine instances for a worker role are managed automatically by Azure and worker roles can be scaled out to meet any processing needs. In this example, the worker role reads data from the Google Analytics Real Time Reporting API every few seconds and sends it to an Azure Service Bus Event Hub.<\/li>\n<li>Azure Service Bus Event Hub \u2013 Service Bus is a generic, cloud-based messaging system for connecting applications, services and devices. Event hub is a managed service within Service Bus that provides a foundation for large-scale ingestion across a broad variety of scenarios. In this solution, Google Analytics data is serialized to JSON in the worker role and sent to Event Hub as individual messages.<\/li>\n<li>Azure Stream Analytics \u2013 Stream Analytics is an event processing engine that provides a SQL interface that allows real-time queries over millions of streaming events per second. I used Stream Analytics to analyze and aggregate the data coming from Google Analytics and send the result to an Azure SQL Database.<\/li>\n<li>Azure SQL Database \u2013 Azure offers SQL Server database capabilities in a Platform as a Service (PaaS) model. In this solution, the output data from Stream Analytics is stored in an Azure SQL Database to be consumed later by a user in Excel or other business intelligence tool.<\/li>\n<\/ul>\n<p>In this new video, I show you how this solution was built from beginning to end and how the data can be used for analysis in Microsoft Excel 2013.<\/p>\n<p>&nbsp;<\/p>\n<p><iframe loading=\"lazy\" id=\"img-1424960446042\" src=\"\/\/player.vimeo.com\/video\/120317291\" width=\"600\" height=\"337\" frameborder=\"0\"><\/iframe><\/p>\n<p style=\"text-align: left;\">Need help developing and executing a strategy for real-time analytics in your organization? <a title=\"Let us\u00a0know\" href=\"\/contact\/\" target=\"_self\" rel=\"noopener\">Let us\u00a0know<\/a>\u00a0and we&#8217;ll setup a call with you and one of our specialists.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Example solution that collects data from the Google Analytics real-time reporting API, aggregates it and provides analysis in Microsoft Excel.<\/p>\n","protected":false},"author":21,"featured_media":15010,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[297],"tags":[304,321],"class_list":["post-16096","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-platform","tag-modern-data-platform","tag-retail-consumer-goods","topics-blog","industries-retail"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/16096","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=16096"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/16096\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/15010"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=16096"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=16096"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=16096"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}