{"id":15801,"date":"2019-05-29T18:56:03","date_gmt":"2019-05-30T01:56:03","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/knowledge-mining-showcase-azure-search-2\/"},"modified":"2024-04-08T13:51:08","modified_gmt":"2024-04-08T20:51:08","slug":"knowledge-mining-showcase-azure-search","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/knowledge-mining-showcase-azure-search\/","title":{"rendered":"Knowledge Mining Showcase: Azure Search"},"content":{"rendered":"

Welcome to the first installment of an in-depth look at Knowledge Mining \u2013\u00a0 <\/span>the ability to use Microsoft Azure\u2019s<\/span><\/a> advanced AI Search<\/span><\/a> capabilities to comb through all of your data (PDFs, emails, scanned documents, images, etc.) to glean insight. In this series, I\u2019m going to take you through Microsoft\u2019s Azure Cognitive Search (Azure Search with human-like reasoning capabilities) and show you the ins and outs of effectively using this awesome tool to uncover insight from enterprise data, whether structured or raw.<\/p>\n

<\/p>\n

Let\u2019s begin by digging in to Azure Search.<\/p>\n

\"knowledge-mining\"<\/i><\/u><\/p>\n

Azure Search<\/h2>\n

Managed by Microsoft, the intelligent Azure Search cloud solution-as-a-service has built-in cognitive abilities. Recognizing and extracting text and identity from images; highlighting key talking points from text; and the power to recognize and classify people, places, and things from text and<\/i> images, are among its innovative capabilities.<\/p>\n

This expansive offering can help dig deep into your organization\u2019s data, often uncovering rich insight. 3Cloud is currently working on one such project for a private global energy exploration and engineering company. Leaders here needed a way look back at decades of data, recorded in paper files, without a team of archivists. We\u2019re working with the worldwide energy giant, using cloud-scale technology, to digitize, store, and offer deep search capabilities on more than six million documents. By combining Azure Search with Knowledge Mining, we will ultimately provide nearly instantaneous access to information that once would have taken a team of people potentially months (if ever) to uncover.<\/p>\n

So, what is Knowledge Mining?<\/p>\n

Knowledge Mining<\/h2>\n

Knowledge Mining is a cognitive search-based technique of extracting facts from unstructured data. It\u2019s like having a crew of experts comb through your most important documents to discover and leverage data to drive your enterprise. This content comprehension capability can be used to create in-depth search resources that inform an organization\u2019s employees and enrich its clients and customers.<\/p>\n

\"Knowledge_Mining_Solution\"<\/p>\n

In today\u2019s tutorial, I\u2019ll offer step-by-step instructions on creating a search service and an index.<\/p>\n

Creating an Azure Search Service and Index<\/h2>\n

The first step in creating an effective search is providing the data that we are going to search. Azure Search can be used against several data sources, both structured (Azure SQL Database and Cosmos DB) and unstructured, in the form of Azure Blob Storage. Blob indexers can extract text from major file formats such as Microsoft Office, PDF, and HTML documents.<\/p>\n

In this how-to series on Knowledge Mining, we\u2019ll be focusing on unstructured data, so let\u2019s create some blob storage.<\/p>\n

First, we need to create a storage account to hold our blobs. From the Azure Portal, select the Create a resource <\/strong>option and then type storage account<\/strong> into the search box. Select Storage account<\/strong> from the drop-down under the search box.<\/p>\n

\"Azure_Marketplace\"<\/p>\n

Click on the Create<\/strong> button.<\/p>\n

\u00a0\"Storage_Account\"<\/p>\n

And now add the details:<\/p>\n

    \n
  1. Select your\u00a0Subscription<\/strong> and Resource group <\/strong>that you are using for your search.<\/li>\n
  2. Enter a Storage account name<\/strong>. Note that this name must be unique across all storage account names.<\/li>\n
  3. Select a Location<\/strong> that you will use for creating all of your resources throughout this exercise.<\/li>\n
  4. Make the following selections:\n
      \n
    1. Performance<\/strong>: Standard<\/li>\n
    2. Account kind<\/strong>: Storage (general purpose v1)<\/li>\n
    3. Replication<\/strong>: Locally-redundant storage (LRS)<\/li>\n<\/ol>\n<\/li>\n
    4. Click on the Next: Advanced><\/strong> button.<\/li>\n<\/ol>\n

      \"Create_Storage_Account_Basics\"<\/p>\n

      On the next page, make the selections shown below and then click on the Review + create<\/strong> button.<\/p>\n

      \u00a0\"Create_Storage_Account_Advanced\"<\/p>\n

      Now press the Create<\/strong> button and wait for the deployment of your new storage account to complete.<\/p>\n

      \u00a0\"Create_Storage_Account_Review\"<\/p>\n

      When your deployment is complete, you will see a link to your new storage account at the bottom of the page. Click on it to move to the next step.<\/p>\n

      \u00a0\"Deployment_Complete\"<\/p>\n

      You\u2019ll now see the Storage Account Overview<\/strong> page. We will create our searchable blob storage in this storage account by clicking on the Blobs <\/strong>link in the middle of the page.<\/p>\n

      \"Storage_Account_Overview\"<\/p>\n

      Click on the big plus sign + Container<\/strong> in the upper left of your screen and enter a name for your container. Think of a container as a folder that will hold all of our documents.<\/p>\n

      Select Container (anonymous read access for containers and blobs)<\/strong> for your Public access level<\/strong> and click on the OK<\/strong> button.<\/p>\n

      \u00a0\"New_Container\"<\/p>\n

      Click on the name of your new container.<\/p>\n

      \"New_Container_Name\"<\/p>\n

      You should now see the Overview<\/strong> of your blob container. Now we need to add the files that we are going to be searching. We do this by clicking on the Upload <\/strong>option in the upper left of the screen.<\/p>\n

      \u00a0\"Blob_Overview\"<\/p>\n

      You\u2019ll see this on the right side of your screen. For this series, we\u2019ll be working with a common data set used in Microsoft\u2019s Knowledge Mining Bootcamp, which is an excellent introduction to Azure Search as well. You can go to the Bootcamp. To get the dataset, go to the GitHub repository at https:\/\/github.com\/Azure\/LearnAI-KnowledgeMiningBootcamp.git<\/span> and clone the repository. You\u2019ll find the sample data in the resources\/dataset\/<\/strong> folder.<\/p>\n

      \u00a0\"Select_Dataset_Files\"<\/p>\n

      From the Azure Portal, click on the selection button and select all of the files in the dataset folder.<\/p>\n

      \u00a0\"Upload_Blob\"<\/p>\n

      Now we are ready to create a search service and index all of the documents we just uploaded.<\/p>\n

      Create a new Azure Search Service by clicking on the Create a resource<\/strong> menu item in Azure Portal and then type Azure Search<\/strong> into the search box. Select Azure Search<\/strong> from the drop-down under the search box.<\/p>\n

      \"New_Azure_Search_Service\"<\/p>\n

      Click on the Create <\/strong>button.<\/p>\n

      \"Azure_Search_Create\"<\/p>\n

      Enter a name for your search service, the subscription and resource group you want your searching to be done in, and the location. You will want these to all match the setting on the storage account that you created earlier.\u00a0 <\/span>You can use the Free<\/strong> pricing tier for these exercises but understand that it is very limited and should only be used for dev\/test.<\/p>\n

      Click the Create<\/strong> button and wait for your new Search Service to be created.<\/p>\n

      \"New_Search_Service\"<\/p>\n

      Now that the search service is created, you should see something similar to this:<\/p>\n

      \"Dashboard_Overview\"<\/p>\n

      Next, we\u2019ll create the index that will hold all of the information for our search. Click on the Import data<\/strong> link on the top of the page. On the next screen, select Azure Blob Storage<\/strong> from the Data Source drop-down list.<\/p>\n

      \"Import_Data\"<\/p>\n

      Name your data source and then click on Choose an existing connection<\/strong> and select the storage account that you created earlier. You will then select the container that you created and press the Select<\/strong> button. Leave all other fields at their default values and press the Next: Add cognitive search (Optional)<\/strong> button on the bottom of the screen. Azure Search will try to infer index fields from the files in your storage account. Since we are working with unstructured data, it will only come back with standard search fields.<\/p>\n

      \"Connect_Data\"<\/p>\n

      We\u2019re going to skip the cognitive search settings for now (much more on this in future posts). Press the Skip to: Customize target index<\/strong> button.<\/p>\n

      \"Add_Cognitive_Search\"<\/p>\n

      We\u2019ll leave all of the settings at their default value for now. Press on the Next: Create an indexer<\/strong> button.<\/p>\n

      \"Customize_Target_Index\"<\/p>\n

      Change the Schedule<\/strong> value to Once<\/strong>. You can leave all of the other fields at their default value and press on the Submit<\/strong> button at the bottom of the screen.<\/p>\n

      \"Create_Indexer\"<\/p>\n

      When the index creation is complete, it will take you back to the Search Service Overview page. Notice that the Index, Indexers, and Data sources menu items in the middle of the page all have a (1) next to them showing the number of elements in each area.<\/p>\n

      Congratulations! You\u2019ve just built your first searchable index.<\/p>\n

      Click on the Indexers(1)<\/strong> link in the middle of the page. You\u2019ll see that the status is set to Warning. Let\u2019s look into that by clicking on the line for your indexer.<\/p>\n

      \"Indexers\"<\/p>\n

      Click on the line in the Execution details<\/strong> section for the indexer we just ran. In the new blade, you\u2019ll see several entries in the Warnings<\/strong> section. There are two kinds of warnings: 1) Document has unsupported content type, and 2) Truncated extracted text to 32768 characters. The first warning is because several of our files contain images. The standard Indexer does index images. We have to add a cognitive service for image cracking. We\u2019ll do this in our next session. The second warning is because we selected the Free<\/strong> pricing tier when we created our search service. The Free pricing tier only allows a maximum of 32,768 characters to be extracted out of a document.<\/p>\n

      \"Indexer_Summary\"<\/p>\n

      Go back to the Search Service Overview page and click on the Indexes(1)<\/strong> link and then again on the line with the index that you just created. From the Index Screen we can query a selected index and test it out. Try it out. Type \u201cMicrosoft\u201d in the Query String<\/strong> field and click Search<\/strong>. You\u2019ll see the results returned in JSON. Using the dataset from the Knowledge Mining Bootcamp, you\u2019ll see 10 documents returned.<\/p>\n

      \"Search_Explorer\"<\/p>\n

      More to Come<\/h2>\n

      In the coming weeks I\u2019ll be exploring the many ways to use Azure Cognitive Search to more easily unearth knowledge from once-difficult-to-mine data sources. Be sure to\u00a0subscribe to our blog<\/span>\u00a0so that you don’t miss a tutorial, or contact us<\/span> today<\/a> to discover the many ways 3Cloud can make the most of your data!<\/span><\/p>\n

      Editor’s note: This post was edited 10\/2020 to reflect system updates.<\/em><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

      Overview of Azure Search for knowledge mining, recognizing and extracting text and identity from images<\/p>\n","protected":false},"author":21,"featured_media":13938,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[330,319],"class_list":["post-15801","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-knowledge-mining","tag-machine-learning-ai","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15801","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15801"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15801\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/13938"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15801"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15801"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15801"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}