Create Azure Search Index, Load Data And Query From .NET SDK

This is part 3 of a 3 article series about Azure search on the below topics

• Getting started with Azure search and provisioning Search service.
• Loading sample data from Azure portal
• Using .NET SDK to perform operation on Azure search

If you are interested in watching video on the same topic, please find it here. This article is in continuation from previous articles at this link.

In this article, we will learn how to create Azure search index, load document/data to it and query the search results. Before going into a step by step tutorial, let us understand what Azure search is from official Azure Search documentation:

Azure Search is a search-as-a-service cloud solution that gives developers APIs and tools for adding a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. Your custom code invokes data ingestion (indexing) to create and load an index. On the other side, your application code issues query requests and handles responses. The search experience is defined in your client using functionality from Azure Search, with query execution over a persisted index that you create, own, and store on Azure Search. 

Use Case

• Suppose your source applications holding data will expose it in XML format (say in ftp or any path)
• We will create console application utility which will run on scheduled time and read xml and load data to Azure Search Index.
• It will also create Index (this can be done from Azure portal UI as well) and delete Index if already created (this is only for demo purposes, in a real business case deleting index everytime is not recommended)
• Once index is created, data is loaded, we will use some sample queries to get search results

Pre-requisites

• Azure Search Service is running (Refer to this link to create one), you can use a free service for this quickstart. // CHANGE THIS TO MY ARTICE LINK,
• Azure search service admin key (Refer to this link on how to get one)
• Visual Studio with Internet connection(to download NuGet packages)

Step 1

Create a console application (I have used C#)

Step 2 - Install NuGet packages

Go to Tools-> NuGet Package Manager-> Package manager console. Run the below command.

Note
We are using a package at this link

Install-Package Microsoft.Azure.Search

Troubleshoot- If you get the below error while installing, follow the steps mentioned below the error.

Install-Package - Failed to add reference to 'System.Runtime'. Please make sure that it is in the Global Assembly Cache.

• Close Visual studio
• Go to project folder and open .csproj file in notepad.
• Add below entry in <ItemGroup> references. You will find other references as well here.
• <Reference Include="System.Runtime" />
• Save file and close it from notepad
• Open project again in visual studio
• Try command again ‘Install-Package Microsoft.Azure.Search’

If everything goes well, you should see the below message,

Step 3 

Add below DLL references to the project.

System.ComponentModel.DataAnnotations.dll

Add reference to below DLL if not already added (in my case, it added automatically for first time but was missing the second time).

Project Path\AzureSearchConsole\packages\Microsoft.Rest.ClientRuntime.Azure.3.3.19\lib\net452\Microsoft.Rest.ClientRuntime.Azure.dll

Step 4. Create sample source XML

Create s sample XML file and save it in some physical folder,

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>    
<root>    
    <Persons>    
        <Person>    
            <ID>1</ID>    
            <NAME>Bruce Wayne</NAME>    
            <ScreenName>Batman</ScreenName>    
            <Power>strength, stamina, endurance, intellect</Power>    
        </Person>    
        <Person>    
            <ID>2</ID>    
            <NAME>Clark Kent</NAME>    
            <ScreenName>Superman</ScreenName>    
            <Power>can fly, freeze breath, durablity</Power>    
        </Person>    
        <Person>    
            <ID>3</ID>    
            <NAME>Barry Allen</NAME>    
            <ScreenName>Flash</ScreenName>    
            <Power>fastest man, durablity, metahuman</Power>    
        </Person>    
        <Person>    
            <ID>4</ID>    
            <NAME>Princess Diana</NAME>    
            <ScreenName>Wonder woman</ScreenName>    
            <Power>athlete, acrobat, fighter and strategist</Power>    
        </Person>    
    </Persons>    
</root> 

Step 5

Create a entity class Person (to hold above xml data).

using Microsoft.Azure.Search;  
// other default references here  
namespace AzureSearchConsole    
{    
    class Person    
    {    
        [System.ComponentModel.DataAnnotations.Key]    
        [IsFilterable]    
        public string ID { get; set; }    
    
        [IsSearchable, IsFilterable, IsSortable, IsFacetable]    
        public string NAME { get; set; }    
    
        [IsSearchable, IsFilterable, IsSortable, IsFacetable]    
        public string ScreenName { get; set; }    
    
        [IsSearchable, IsFilterable, IsSortable, IsFacetable]    
        public string Power { get; set; }    
    }    
}

Step 6

Replace main method in program.cs with below,

static void Main(string[] args)  
{  
    string searchServiceName = "azsearchdemofree";  
    string queryApiKey = "MYADMINAPIKEYFROMAZUREPORTAL";  // Replace this with actual key
    string indexName = "superheroes";  
    SearchServiceClient serviceClient = CreateSearchServiceClient(searchServiceName, queryApiKey);  
    Console.WriteLine("{0}", "Deleting index...\n");  
    DeleteIndexIfExists(indexName, serviceClient);  
    Console.WriteLine("{0}", "Creating index...\n");  
    CreateIndex(indexName, serviceClient);  
    ISearchIndexClient indexClient = serviceClient.Indexes.GetClient(indexName);  
    Console.WriteLine("{0}", "Uploading documents...\n");  
    UploadDocuments(indexClient);  
    ISearchIndexClient indexClientForQueries = CreateSearchIndexClient(indexName, searchServiceName, queryApiKey); 
    RunQueries(indexClientForQueries);  
    Console.WriteLine("{0}", "Complete.  Press any key to end application...\n");  
    Console.ReadKey();  
}

Step 7. Add supporting methods called in above main method

Below methods are to create index and search client, using this two object type we will get a different method available in Search SDK to performs operations on Azure Search.

private static SearchIndexClient CreateSearchIndexClient(string indexName, string searchServiceName, string queryApiKey)  
{  
    SearchIndexClient indexClient = new SearchIndexClient(searchServiceName, indexName, new SearchCredentials(queryApiKey));  
    return indexClient;  
}  
private static SearchServiceClient CreateSearchServiceClient(string searchServiceName, string adminApiKey)  
{  
    SearchCredentials creds = new SearchCredentials(adminApiKey);  
    SearchServiceClient serviceClient = new SearchServiceClient(searchServiceName, creds);  
    return serviceClient;  
}

// Delete an existing index to reuse its name  
private static void DeleteIndexIfExists(string indexName, SearchServiceClient serviceClient)  
{  
    if (serviceClient.Indexes.Exists(indexName))  
    {  
        serviceClient.Indexes.Delete(indexName);  
    }  
}

Below method will create index and create field based on Person class we defined  

// Create an index whose fields correspond to the properties of the Person class.  
// The fields of the index are defined by calling the FieldBuilder.BuildForType() method.  
private static void CreateIndex(string indexName, SearchServiceClient serviceClient)  
{  
    var definition = new Index()  
    {  
        Name = indexName,  
        Fields = FieldBuilder.BuildForType<Person>()  
    };  
    serviceClient.Indexes.Create(definition);  
}

Next method is to load document(records) in Azure Search Index (database/table). We would be parsing our source XML data to get actual person data and add it to IndexClient in batch. And finally call Index method by passing batch object to actually index our data.

// Upload documents as a batch  
private static void UploadDocuments(ISearchIndexClient indexClient)  
{  
    XmlDocument xmlDoc = new XmlDocument();  
    xmlDoc.Load(@"F:\Sid\AzureSearchConsole\AzureSearchConsole\sourcedata.xml");  
    XmlNodeList nodeList = xmlDoc.DocumentElement.SelectNodes("/root/Persons/Person");  
    List<Person> personCollections = new List<Person>();  
    var actions = new IndexAction<Person>[4];  
    int i = 0;  
    foreach (XmlNode node in nodeList)  
    {  
        Person person = new Person();  
        person.ID = node.SelectSingleNode("ID").InnerText;  
        person.NAME = node.SelectSingleNode("NAME").InnerText;  
        person.ScreenName = node.SelectSingleNode("ScreenName").InnerText;  
        person.Power = node.SelectSingleNode("Power").InnerText;  
        personCollections.Add(person);  
        actions[i] = IndexAction.Upload(person);  
        i++;  
    }  
    var batch = IndexBatch.New(actions);  
    try  
    {  
        indexClient.Documents.Index(batch);  
    }  
    catch (Exception e)  
    {  
        // When a service is under load, indexing might fail for some documents in the batch.   
        // Depending on your application, you can compensate by delaying and retrying.   
        // For this simple demo, we just log the failed document keys and continue.  
        Console.WriteLine("Failed to index some of the documents: {0}",  
            String.Join(", ", e.Message));  
    }  
    // Wait 2 seconds before starting queries  
    Console.WriteLine("Waiting for indexing...\n");  
    Thread.Sleep(2000);  
}

Next add method which will run queries.

// Add query logic and handle results  
private static void RunQueries(ISearchIndexClient indexClient)  
{  
    SearchParameters parameters;  
    DocumentSearchResult<Person> results;  

    // Query 1   
    Console.WriteLine("Query 1: Search for term 'batman', returning the full document");  
    parameters = new SearchParameters();  
    results = indexClient.Documents.Search<Person>("batman", parameters);  
    WriteDocuments(results);  

    // Query 2  
    Console.WriteLine("Query 2: Search on the term 'fly', returning selected fields");  
    Console.WriteLine("Returning only these fields: NAME, ScreenName, Power:\n");  
    parameters =  
        new SearchParameters()  
        {  
            Select = new[] { "NAME", "ScreenName","Power" },  
        };  
    results = indexClient.Documents.Search<Person>("fly", parameters);  
    WriteDocuments(results);  

    // Query 3  
    Console.WriteLine("Query 3: Search for the terms 'acrobat' and 'metahuman'");  
    Console.WriteLine("Returning only these fields: NAME, ScreenName, Power:\n");  
    parameters =  
        new SearchParameters()  
        {  
            Select = new[] { "NAME", "ScreenName", "Power" },  
        };  
    results = indexClient.Documents.Search<Person>("acrobat, metahuman", parameters);  
    WriteDocuments(results);      
}  

Next add method which will display search results

// Handle search results, writing output to the console  
private static void WriteDocuments(DocumentSearchResult<Person> searchResults)  
{  
    foreach (SearchResult<Person> result in searchResults.Results)  
    {  
        Console.WriteLine("ID:--" + result.Document.ID);  
        Console.WriteLine("Name:--" + result.Document.NAME);  
        Console.WriteLine("ScreenName:--" + result.Document.ScreenName);  
        Console.WriteLine("Power:--" + result.Document.Power);  
    }  
    Console.WriteLine();  
}

Step 7

Run the console application, we should see the below output in console.

Notes on output

• Id column is empty for query 2 and 3, this is because in this query we have specified to return specific fields and ID is not mentioned.
• For multiple terms specified like acrobat and metahuman, we got 2 results which is Flash and Wonder woman which are powers assigned to individuals.
• We can also specify search query based on specific column like Screen name = 'Wonder woman' etc.

Now let us go to Azure portal to see indexes created and data which is being loaded by our console application. 

Select Azure Search Service,

In overview tab, click on Indexes as marked in below screenshot.

We can see 4 as document count

Click on superheroes, we will get the below screen, without entering any query just click on Search

This concludes our article on how to create console utility application which can load data to Azure Search index. This would be a real use case because, mostly in business cases, data would be exposed by multiple external applications or datasources and can be used to display search results in  enterprise search portal.

You can download a sample attached with this article or clone it from this github repo.

Hope you enjoyed reading..!!