{"id":15792,"date":"2019-07-22T13:34:35","date_gmt":"2019-07-22T20:34:35","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/knowledge-mining-showcase-using-apis-to-manage-azure-search-3\/"},"modified":"2024-01-08T15:17:36","modified_gmt":"2024-01-08T23:17:36","slug":"knowledge-mining-showcase-using-apis-to-manage-azure-search","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/knowledge-mining-showcase-using-apis-to-manage-azure-search\/","title":{"rendered":"Knowledge Mining Showcase: Using APIs to Manage Azure Search"},"content":{"rendered":"

In this installment of 3Cloud’s Knowledge Mining Showcase, we are going to learn how to create an Azure Search Service and perform searches using application programming interfaces (APIs).<\/p>\n

Even though the Azure Portal gives us a reasonable user interface to create and manage our searches, APIs give us more flexibility and can be saved and rerun many times.<\/p>\n

Moving forward in the Knowledge Mining Showcase, we will primarily be using APIs, rather than the Azure Portal user interface.<\/p>\n

<\/p>\n

\"knowledge<\/p>\n

Before we begin, you will need a few things,<\/p>\n

    \n
  1. An API development environment. I use Postman, which you can get here<\/a>.<\/li>\n
  2. A familiarity with RESTful APIs and how to use them. If you are new to APIs, you can get a good start and learn about the different API methods here<\/a>.<\/li>\n
  3. A familiarity with Azure Search<\/a> and an understanding of how to create an Azure storage account, and an Azure Search Service. Both of these are covered extensively in the Azure Search post<\/a> of the Knowledge Mining Showcase.<\/li>\n
  4. A storage account with data to index and search.<\/li>\n<\/ol>\n

    Getting Started<\/h2>\n

    Details on creating an Azure storage account with blob containers are provided in the Azure Search post<\/a> of the Knowledge Mining Showcase. We will be working with a common dataset used in Microsoft\u2019s Knowledge Mining Bootcamp, which is also an excellent introduction to Azure Search. You can go to the Bootcamp. To get the dataset, go to the GitHub repository at https:\/\/github.com\/Azure\/LearnAI-KnowledgeMiningBootcamp.git<\/a> and clone the repository. You\u2019ll find the sample data in the resources\/dataset\/<\/strong> folder.<\/p>\n

    \"Knowledge<\/p>\n

    Upload this data into your blob container if you wish to follow along and get the same results that I get. It\u2019s fine to use your own data if you prefer. Try to use a combination of PDF, Word, Excel, HTML, and graphic files.<\/p>\n

    Once you have your data loaded, you will need to create a Search Service. Again, details on how to do this can be found in the Azure Search post<\/a> of the Knowledge Mining Showcase.<\/p>\n

    Have your blob container loaded with data and your Search Service created? Let\u2019s move forward.<\/p>\n

    Creating a Dataset<\/h2>\n

    We need to give a link to our data to the Search Service. We do this by creating a data source in the Search Service. The format for the API call to create a data source is:<\/p>\n

    POST<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/datasources?api-version=[api-version]<\/span><\/p>\n

    Where [Search_Service_Name]<\/span> is the name of your Search Service and [api-version]<\/span> is the version of the API you are using. \u201c2019-05-06\u201d is the most current production version as the writing of this post.<\/p>\n

    I\u2019ve created a Search Service named \u2018bgblogsearchservice<\/strong>\u2019, so my call would be:<\/p>\n

    POST https:\/\/bgblogsearchservice.search.windows.net\/datasources?api-version=2019-05-06<\/p>\n

    But we\u2019ve got to add information about the blob container we are using for our data source. We do that in the body of the POST call. The body for my call looks like this:<\/p>\n

    \"code<\/p>\n

    Where:
    \nname <\/strong>\u2013 The name of your new data source
    \ndescription<\/strong> \u2013 A description of your data source
    \ntype<\/strong> \u2013 Can be one of \u00a0the following: azureblob, azuretable, azuresql, or documentdb
    \ncredentials<\/strong> \u2013 Contains the connection string to your storage account
    \ncontainer <\/strong>\u2013 The name of the container where your blobs are stored<\/p>\n

    You can get the connection string to your storage account by opening up your storage account, selecting Access Keys<\/strong> under Settings<\/strong> and then copying the key1 Connection string<\/strong>.<\/p>\n

    \"Knowledge<\/p>\n

    One last thing \u2013 to complete our Create Data source API call, we\u2019ve got to add the headers to our call. We need to add the Content-Type<\/strong> and the api-key<\/strong>.<\/p>\n

    \"Knowledge<\/p>\n

    You can get the API Key from your Search Service. Select your Search Service<\/strong> and then select Keys<\/strong> under Settings<\/strong> and copy the Primary admin key<\/strong>.<\/p>\n

    \"Knowledge<\/p>\n

    Once you copy the Search Service key<\/strong> into the header of your API call, you should be all set. Here\u2019s what my call looks like in Postman:<\/p>\n

    \"Knowledge<\/p>\n

    Just hit Send <\/strong>and you should see a result something like this:<\/p>\n

    \"Knowledge<\/p>\n

    If you don\u2019t trust the API response, you can go to your Search Service in the Azure Portal and verify that that data source has been created.<\/p>\n

    \"Knowledge<\/p>\n

    In case you ever need it, you can delete your data source via an API call using the following format:<\/p>\n

    DELETE<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/datasources\/[Data_Source_Name]<\/span>?api-version=[api-version]<\/span><\/p>\n

    Creating an Index<\/h2>\n

    Think of the Index<\/strong> as the definition of the data in our data source; how it will be searched, and how results will be returned. The format for the API call to create an Index is:<\/p>\n

    POST<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexes?api-version=[api-version]<\/span><\/p>\n

    Where [Search_Service_Name]<\/span> is the name of your Search Service, [Index_Name]<\/span> is the name of your Index, and [api-version]<\/span> is the version of the API you are using. \u201c2019-05-06\u201d is the most current production version as the writing of this post.<\/p>\n

    Using my Search Service named \u2018bgblogsearchservice<\/strong>\u2019 and an Index named \u2018bgblogindex<\/strong>\u2019, my call would be:<\/p>\n

    POST<\/strong> https:\/\/bgblogsearchservice.search.windows.net\/indexes?api-version=2019-05-06<\/p>\n

    Note that you can update an existing Index by using the following PUT format. PUT will create the given Index if it does not already exist:<\/p>\n

    PUT<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexes\/[Index_Name]<\/span>?api-version=[api-version]<\/span><\/p>\n

    The body for our Index POST is a bit longer than that of the Data source POST, but it\u2019s not too bad. We need to name our Index and define each field that will be in our Index. The format of each field definition will be:<\/p>\n

    \"knowledge<\/p>\n

    This is an example subset of commonly used field definitions; there are many more. For a full reference of available Index field definitions, see here<\/a>.<\/p>\n

    A pretty simple and standard definition of an Index may look like this:<\/p>\n

    \"code<\/p>\n

    We can use the above JSON as our Create Index POST<\/strong>. Don\u2019t forget to add the Headers with the Content-Type<\/strong> and the api-key<\/strong> values.<\/p>\n

    Here\u2019s what mine looks like in Postman:<\/p>\n

    \"Knowledge<\/p>\n

    Hit the Send<\/strong> button and you should get a result like this:<\/p>\n

    \"Knowledge<\/p>\n

    We know that with this response, the Index was created, but we can verify that by going to our Search Service on the Azure Portal and finding the Index:<\/p>\n

    \"Knowledge<\/p>\n

    In case you ever need it, you can delete your Index via an API call using the following format:<\/p>\n

    DELETE<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexes\/[Index_Name]<\/span>?api-version=[api-version]<\/span><\/p>\n

    Creating an Indexer<\/h2>\n

    The Indexer is utility that moves the raw data from your data source into your Index. The format for the API call to create an Indexer is:<\/p>\n

    POST<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexers?api-version=[api-version]<\/span><\/p>\n

    Where [Search_Service_Name]<\/span> is the name of your Search Service and [api-version]<\/span> is the version of the API you are using. \u201c2019-05-06\u201d is the most current production version as the writing of this post.<\/p>\n

    Using my Search Service named \u2018bgblogsearchservice<\/strong>\u2019 and an Indexer named \u2018bgblogindexer<\/strong>\u2019, my call would be:<\/p>\n

    POST<\/strong> https:\/\/bgblogsearchservice.search.windows.net\/indexers?api-version=2019-05-06<\/p>\n

    Note that you can update an existing Indexer by using the following PUT format:<\/p>\n

    PUT<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexers\/[Indexer_Name]<\/span>?api-version=[api-version]<\/span><\/p>\n

    The body for our Indexer POST is basically a mapping from our raw data to the Index. We only need to do this if we want the name of the field in the Index to be different than the raw data. We\u2019ll also need to use the Indexer to perform mapping functions.<\/p>\n

    Here is a basic Indexer body for our example:<\/p>\n

    \"knowledge<\/p>\n

    We need to name our Indexer (\u201cname<\/strong>\u201d) and then identify our Data source (\u201cdataSourceName<\/strong>\u201d) and our Index (\u201ctargetIndexName<\/strong>\u201d). The rest is field mapping.<\/p>\n

    Remember in our Index definition we had a field named \u201cmetadata_storage_path<\/strong>\u201d? This is a metadata field that Search gives us automatically. Common practice is to make this the unique identifier in our index, as it is guaranteed to be unique. Since \u201cmetadata_storage_path<\/strong>\u201d is an odd name for an ID field, let\u2019s rename it to \u201cid<\/strong>\u201d. We do this with the field mapping above. We also want to encode it into a number rather than a text field. We do that with the \u201cmappingFunction<\/strong>\u201d section and the \u201cbase64code<\/strong>\u201d encoding function provided to us by Search.<\/p>\n

    Let\u2019s go back to our Index definition:<\/p>\n

    \"knowledge<\/p>\n

    Since we\u2019re mapping the \u201cmetadata_storage_path<\/strong>\u201d field to \u201cid<\/strong>\u201d with our Indexer, we need to update our Index, or we will get an error that the \u201cid<\/strong>\u201d field does not exist on the target Index when we run our Indexer.<\/p>\n

    We must DELETE<\/strong> our existing Index and then recreate it with a POST <\/strong>call. You cannot change field names on an existing Index with a PUT call. Using the Index created earlier, we make the following API call to delete it:<\/p>\n

    DELETE<\/strong> https:\/\/bgblogsearchservice.search.windows.net\/indexes\/bgblogindex?api-version=2019-05-06<\/p>\n

    We can then make the same POST <\/strong>call we made earlier to create the Index, but replace the name \u201cmetadata_storage_path<\/strong>\u201d with \u201cid<\/strong>\u201d in the body of the call. This is what my call looks like in Postman:<\/p>\n

    \"Knowledge<\/p>\n

    Now we are set to run our Indexer using the mapping we defined earlier. Don\u2019t forget to add the api-key <\/strong>and the Content-Type<\/strong> to the header. This is what my call looked like in Postman:<\/p>\n

    \"Knowledge<\/p>\n

    Not only does the create Indexer call<\/strong> create the Indexer, it also runs it. This can take some time. You can check the status of the Indexer from the Azure Portal Search Service Overview<\/strong> page, or you can make the following API Call:<\/p>\n

    GET<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexers\/[Indexer_Name]<\/span>\/status?api-version=[api-version]<\/span><\/p>\n

    \"Knowledge<\/p>\n

    Once the \u201cstatus<\/strong>\u201d comes back as \u201csuccess<\/strong>\u201d, your Indexer has finished.<\/p>\n

    Performing A Search<\/h2>\n

    We now have our Index populated with the data from our data source using the rules we defined in our Indexer.<\/p>\n

    The Index is now ready to Search!<\/p>\n

    The format of our API call to perform a search is:<\/p>\n

    GET<\/strong> https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexes\/[Index_Name]<\/span>\/docs?[query paramaters]?api-version=[api-version]<\/span><\/p>\n

    There are a lot of potential [query parameters] <\/strong>we can use for our search. You can find a full list here<\/a>. For this exercise we\u2019re only going to use a few:<\/p>\n

    search<\/strong>=[string]<\/p>\n

    $count<\/strong>=[true|false]<\/p>\n

    \u201csearch<\/strong>\u201d is the term that we will be searching for and \u201c$count=true<\/strong>\u201d will give us back the count of the records found with our search term. If we use this API call:<\/p>\n

    https:\/\/[Search_Service_Name]<\/span>.search.windows.net\/indexes\/bgblogindex\/docs?search=microsoft&$count=true&api-version=2019-05-06<\/p>\n

    If you used the files from the Knowledge Mining Bootcamp, you should return 10 records back from your Search and it should look something like this:<\/p>\n

    \"Knowledge<\/p>\n

    Congratulations! You\u2019ve now created a data source, index, and indexer, as well as performed an Azure Search, all using APIs.<\/p>\n

    More to Come<\/h2>\n

    In the coming weeks I\u2019ll be exploring the many ways to use Azure Cognitive Search to more easily unearth knowledge from once-difficult-to-mine data sources. Be sure to\u00a0subscribe to our blog\u00a0so that you don’t miss a tutorial, or contact us<\/a> today to discover the many ways 3Cloud can make the most of your data!<\/p>\n

    Reference and Resources<\/h2>\n

    Indexer Overview<\/a><\/p>\n

    API Call Reference<\/p>\n

    Create a Data Source<\/a><\/p>\n

    Delete a Data Source<\/a><\/p>\n

    Create an Index<\/a><\/p>\n

    Delete an Index<\/a><\/p>\n

    Create an Indexer<\/a><\/p>\n

    Delete an Indexer<\/a><\/p>\n

    Run an Indexer<\/a><\/p>\n

    Reset an Indexer<\/a><\/p>\n

    Perform a Search<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

    In this installment of 3Cloud\\’s Knowledge Mining Showcase, we are going to learn how to create an Azure Search Service and perform searches using application programming interfaces (APIs).<\/p>\n","protected":false},"author":21,"featured_media":13308,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[260],"tags":[330],"class_list":["post-15792","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai","tag-knowledge-mining","topics-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15792","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/comments?post=15792"}],"version-history":[{"count":0,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/posts\/15792\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media\/13308"}],"wp:attachment":[{"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/media?parent=15792"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/categories?post=15792"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/3cloudsolutions.com\/wp-json\/wp\/v2\/tags?post=15792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}