Browse Images By Tag Using C# And Azure Cognitive Services

Introduction

In the following article, we'll see how to develop a program that uses the powerful Azure Cognitive Services to get a list of tags from an image, and stores them to further create an application capable of searching images through keywords. The presented source code will be written in C#, and split in two applications: The first one will process a directory, searching for JPEG files and analyzing them through Azure. Cognitive Services results will be saved to an XML file. The second program will simply allow the user to enter a keyword, and scan the XML file to find matches. When a match is found, the images will be shown in a WPF DataGrid.

Before we begin, we must create a Cognitive Service resource on Azure. The following section will cover how to do that.

Create an Azure Computer Vision API resource

Log in to the Azure portal, then click the "Create a resource" link. In the search bar, enter "Computer Vision API", and confirm. Now our browser will show something like the following screenshot. Click "Create" to proceed.

Now we can assign a name, a geographic location (select the nearest for less latency), and a pricing tier. For the purposes of the article, the free F0 tier will be a good choice.

In creation confirming, Azure will assign two API keys, which applications need to know and use to access the services functions.

To get the keys, click on the newly created resource, then "Keys" link under "Resource Management"

Save one of those keys (a single key will be required) for the following steps.

Write a C# class to use Azure resource

As stated in the Introduction, our source code will be composed of two parts. Here we'll see how to write a simple console app to query Azure Cognitive Services, feeding them with single images we wish to be "seen". The Azure AI will return a set of tags, that will be saved in an XML file for future reference, like a catalog of images along with their descriptive texts.

To achieve that feature, we need first of all a class to communicate with Azure, and some prerequisites.

Let's open Visual Studio, and create a new solution. Add a first project, which will be a Console App. Then, save it and go to Nuget Manager, where two additional packages need to be installed, being them Microsoft.ProjectOxford.Visionand Newtonsoft.Json

The following class is used to send a Stream to Azure Computer Vision, and to read the returned values:

using Microsoft.ProjectOxford.Vision;
using Microsoft.ProjectOxford.Vision.Contract;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
namespace ivDataGather {
    class MicrosoftCognitiveCaptionService {
        private static readonly string ApiKey = "<INSERT YOUR API KEY HERE>";
        private static readonly string ApiEndpoint = "https://westus.api.cognitive.microsoft.com/vision/v1.0";
        private static readonly VisualFeature[] VisualFeatures = {
            VisualFeature.Description,
            VisualFeature.Tags
        };
        public async Task < string > GetTagsAsync(Stream stream) {
            var client = new VisionServiceClient(ApiKey, ApiEndpoint);
            var result = await client.AnalyzeImageAsync(stream, VisualFeatures);
            return ProcessAnalysisResult(result);
        }
        private static string ProcessAnalysisResult(AnalysisResult result) {
            string message = result ? .Description ? .Captions.Count() > 0 ? string.Join(", ", result ? .Description ? .Captions.Select(x => x.Text)) : "";
            string tags = result ? .Tags ? .Count() > 0 ? string.Join(", ", result ? .Tags ? .Select(x => x.Name)) : "";
            var list = new List < string > ();
            message.Split(' ').ToList().ForEach(x => {
                list.Add(x.Trim());
            });
            tags.Split(',').ToList().ForEach(x => {
                list.Add(x.Trim());
            });
            return string.Join(",", list.Distinct().ToList());
        }
    }
}

The first async method, GetTagsAsync, will create a new VisionServiceClient object, given the service API key and the endpoint to use (which is the URL of the service, which could be extracted from Azure, and its based on the chosen geographical location). Here we will query Azure on two features: VisualFeature.Description and VisualFeature.Tags. Those enum values will inform the service we are interested in, reading back only that information.

Once we have received a result, GetTagsAsync will call on a second method, ProcessAnalysisResult. The method will return a string object, based on the Description and Tags properties, which will be parsed and taken as a distinct list, comma separated, transformed in a one-line response.

Develop a console app to process an images folder

With those premises, developing the further steps is trivial. Here follows the console part, which simply parse a given directory for JPEG files, sending them to Azure and awaiting results. When they come back to us, those results will be stored in a TaggedImage list (a custom class we'll see in a moment), to be saved in a serialized XML at the end of the procedure.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Xml.Serialization;
namespace ivDataGather {
    class Program {
        static async void ParseDirectory(string directory, string catalogOutp) {
            var itemList = new List < TaggedImage > ();
            var cc = new MicrosoftCognitiveCaptionService();
            foreach(var f in Directory.GetFiles(directory, "*.jpg")) {
                var r = "";
                var ms = new FileStream(f, FileMode.Open);
                r = await cc.GetTagsAsync(ms);
                itemList.Add(new TaggedImage() {
                    FilePath = f,
                        Tags = r.Split(',').ToList()
                });
                Console.WriteLine(f + "\n" + r + "\n");
                ms.Dispose();
            }
            cc = null;
            XmlSerializer serialiser = new XmlSerializer(typeof(List < TaggedImage > ));
            TextWriter fileStream = new StreamWriter(catalogOutp);
            serialiser.Serialize(fileStream, itemList);
            fileStream.Close();
        }
        static void Main(string[] args) {
            ParseDirectory(@ "C:\<input_folder>\images", @ "C:\<any_output_folder>\ivcatalog.xml");
            Console.ReadLine();
        }
    }
}
using System.Collections.Generic;
using System.IO;
using System.Windows.Media.Imaging;
namespace ivDataGather {
    public class TaggedImage {
        public string FilePath {
            get;
            set;
        }
        public string FileName {
            get {
                return GetFileAttributes() ? .Name;
            }
        }
        public string FileExtension {
            get {
                return GetFileAttributes() ? .Extension;
            }
        }
        public string FileDirectory {
            get {
                return GetFileAttributes() ? .DirectoryName;
            }
        }
        public List < string > Tags {
            get;
            set;
        }
        public BitmapImage Image {
            get {
                return new BitmapImage(new System.Uri(this.FilePath));
            }
        }
        private FileInfo GetFileAttributes() {
            return new FileInfo(this.FilePath);
        }
    }
}

As it can be seen, the TaggedImage class simply contains the found file name and path reference and tags (returned by Azure), plus an Image property which returns a BitmapImage object, i.e. the graphical format for the file. That property will be useful for the second part of the program.

Executing the console app

Let's create a sample folder with some images in it.

Running our console app with the path containing the above image will result in something like the following screenshot:

In other words, for every found JPG file, the method GetTagsAsync will return a list of words from Azure image analysis. The console app receives that list, then proceed in saving them to an XML file through an XMLSerializer object. The saved file, from the above analysis, will be as follows:

That file will be used as a catalog by the WPF application, which will be covered in the next section

Develop a WPF app to search tags and show results

The idea behind that part of the solution is really simple: a TextBox in which to write our search term, a button to start searching, and a DataGrid to show all the found results. Obviously, the search involves the deserialization of our XML catalog. Here we will develop using WPF, to quickly implement - through XAML - an image column for our Datagrid.

The XAML code of our MainWindow will be

<Window x:Class="ivImageFinder.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:local="clr-namespace:ivImageFinder" mc:Ignorable="d" Title="Image finder" Height="350" Width="525">
    <Grid>
        <DockPanel HorizontalAlignment="Stretch" LastChildFill="true" VerticalAlignment="Stretch" Background="LightGray">
            <DockPanel Height="50" LastChildFill="True" VerticalAlignment="Top" DockPanel.Dock="Top" Background="WhiteSmoke">
                <Button x:Name="button" Content="Search" Width="75" DockPanel.Dock="Right" FontSize="16" Click="button_Click" />
                <TextBox x:Name="searchBox" TextAlignment="Center" VerticalContentAlignment="Center" HorizontalAlignment="Stretch" Text="TextBox" DockPanel.Dock="Left" FontSize="16" /> </DockPanel>
            <DataGrid x:Name="dataGrid" DockPanel.Dock="Bottom">
                <DataGrid.Columns>
                    <DataGridTemplateColumn Header="Image" Width="300" IsReadOnly="True">
                        <DataGridTemplateColumn.CellTemplate>
                            <DataTemplate>
                                <Image Source="{Binding Image}" /> </DataTemplate>
                        </DataGridTemplateColumn.CellTemplate>
                    </DataGridTemplateColumn>
                </DataGrid.Columns>
            </DataGrid>
        </DockPanel>
    </Grid>
</Window>

As you can see, the image column is easily created by accessing the DataGrid Columns property, customizing a column to contain an Image control. Here we will bind its contents to a property named "Image" (the same property we saw above in TaggedImage class - so, the ItemSource property of the DataGrid will be a List<TaggedImage>).

That layout will result in the following window:

The code behind is trivial and consists mainly of the button click handler:

using ivDataGather;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Windows;
using System.Xml.Serialization;
namespace ivImageFinder {
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow: Window {
        private string catalogPath = @ "C:\<path_to_catalog_file>\ivcatalog.xml";
        public MainWindow() {
            InitializeComponent();
        }
        private void button_Click(object sender, RoutedEventArgs e) {
            List < TaggedImage > itemList;
            using(var reader = new StreamReader(catalogPath)) {
                XmlSerializer deserializer = new XmlSerializer(typeof(List < TaggedImage > ));
                itemList = (List < TaggedImage > ) deserializer.Deserialize(reader);
            }
            dataGrid.ItemsSource = itemList.Where(x => x.Tags.Contains(searchBox.Text));
        }
    }
}

We have a private variable which specifies the path to our XML catalog. At each button click, the program deserializes the XML content to a List<TaggedImage>, using it as the ItemSource of DataGrid. LINQ is here used to filter only those items which corresponds to the digited text. The Image property will be bound to the Image control of the DataGrid column, showing the desired results.

Here are some examples of searching for keywords

Source code

The source code used in the article is freely downloadable  here.

Bibliography

• Azure Cognitive Services
• Azure Computer Vision API
• Examples of XML serialization
• WPF DataGrid customization