Learn About Azure Cognitive Services

Introduction

Microsoft Cognitive Services (earlier known as Project Oxford) provide us the ability to build intelligent applications just by writing a few lines of code. These applications or services are deployed on all major platforms like Windows, iOS, and Android.  All the APIs are based on machine learning APIs and enable the developers to easily add intelligent features – such as emotion and video detection; facial, speech and vision recognition; and speech and language understanding – into their applications.

Look into the below URLs for reference of Microsoft Cognitive Services.

• https://azure.microsoft.com/en-in/services/cognitive-services/
• https://docs.microsoft.com/en-us/azure/cognitive-services/welcome

NOTE

Microsoft announced the preview version of Microsoft Cognitive Services on March 30, 2016.

I would recommend you to read the articles mentioned below.

• Microsoft Cognitive Services – Computer Vision API Version 1.0
• Microsoft Cognitive Services – Face API
• Microsoft Cognitive Services – Custom Speech Service
• Microsoft Cognitive Services – Speaker Recognition API
• Microsoft Cognitive Services – Learn about Language Understanding Intelligent Service (LUIS)
• Microsoft Cognitive Services – Bing Speech API
• Microsoft Cognitive Services - Academic Knowledge API
• Microsoft Cognitive Services - Emotion API

In this article, I will walk you through steps of exploring FACE and EMOTION APIs of Microsoft Cognitive Services. Henceforth, this article will cover the following three parts and I would suggest you go through this article in the below-given order only.

1. Create Azure Account
2. FACE API
3. EMOTION API

So, below are the prerequisites to work with Microsoft Cognitive Services API,

1. Visual Studio 2015 (Community, Enterprise, or Professional edition)
2. Microsoft Azure Account 

Now, let’s get started.

Part 1 – Create Azure Account

• Sign in to the Microsoft Azure Portal.
• You will be asked to log in with a Microsoft Account. You can take a free subscription of one month or can choose among different plans available on the Azure portal, as per your requirement or business needs. (In my case, I took free subscription of one month).
  
  
  • You will be asked for your phone number and credit card details.
    
    
• You will be given some credit points after successful creation of your account based on your selected country and zone.
• After successful creation of Azure Account, you will see the dashboard as shown below.
  
  
  

Part 2 – FACE API

Face API, provided by Microsoft Cognitive Services, helps to detect, analyze, and organize the faces in a given image. We can also tag faces in any given photo.

Face API provides the advanced face algorithms and it has two main functions:-

1. Face Detection with Attributes 
• Face API detects up to 64 human faces with high precision face location in an image.
• The image can be specified by file in bytes or valid URL.
  
  
2. Face Recognition

• Face Verification 
  
  
• Performs an authentication against two detected faces or authentication from one detected face to one-person object.

• Finding Similar Face 
  
  
• It takes target face or query face as input and finds a small set of faces that looks most similar to the target face. It has two modes – matchFace and matchPerson.

• matchFace returns similar faces, as it ignores the same-person threshold
• matchPerson returns similar faces after applying a same-person threshold

• Face Grouping
  
  
• It takes a set of unknown faces as input
• Face Grouping API divides these unknown faces into several groups based on similarity.
  
  
• Personal (Face) Identification
  
  
• It identifies people based on a detected face and people database. This database needs to be created in advance and can be edited later.

Assuming you have Azure portal account, follow the below steps to implement FACE API.

Step 1 

Click on “+” or “New” link in the Azure portal on left-hand side.

Step 2 

Once you click on “AI + Cognitive Services”, you will see the list of APIs available in Cognitive Services.

Step 3 

Choose Face API to subscribe to Microsoft Cognitive Services Face API and proceed further with subscription steps. After clicking on Face API, it will show the legal page. Read it carefully and then click on "Create".

Step 4 

After clicking on Create, there are two possibilities (happened in my case).

• You will be given a form to fill, as shown below, in which you have to fill the below details (This option might be visible after a few hours)
  
• Name
• Subscription (In my case, I choose Free trial; you can choose F0 or S0 subscription types)
• Resource Group
  
  
  
  
• You will see the below image and it will ask you for a new subscription, in which case you can generate the Subscription Keys and Endpoint URL from a different URL (https://azure.microsoft.com/en-us/try/cognitive-services/).
  
  
  
  

Below are the generated Keys and Endpoint URL.

Step 5 

Create a WPF Application in Visual Studio 2015 (Visual C# > Windows Desktop > WPF Application). I have named the application as “Face Tutorial”.

Step 6 

Add a button with the name as “Browse” on MainWindow.xaml, using designer or code. Here, I prefer adding the button using code.

<Window x:Class="Face_Tutorial.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="700" Width="960">
    <Grid x:Name="BackPanel">
        <Image x:Name="FacePhoto" Stretch="Uniform" Margin="0,0,0,50" MouseMove="FacePhoto_MouseMove" />
        <DockPanel DockPanel.Dock="Bottom">
            <Button x:Name="BrowseButton" Width="72" Height="20" VerticalAlignment="Bottom" HorizontalAlignment="Left"
                    Content="Browse..."
                    Click="BrowseButton_Click" />
            <StatusBar VerticalAlignment="Bottom">
                <StatusBarItem>
                    <TextBlock Name="faceDescriptionStatusBar" />
                </StatusBarItem>
            </StatusBar>
        </DockPanel>
    </Grid>
</Window>

Step 7 

Now, go to MainWindow.xaml.cs. The below-given directives are required in the solution to access the Face APIs.

1. using Microsoft.ProjectOxford.Face;
2. using Microsoft.ProjectOxford.Face.Contract;

To add the above references, browse for two dll’s and click on "Install".

1. Newtonsoft.JSON
2. Microsoft.ProjectOxford.Face
   
   
   
   
   

Once you add these two dll’s, these will be shown in the solution.

 

Step 8

Add the lines of code given below to click event of Browse button.

private async void BrowseButton_Click(object sender, RoutedEventArgs e)
        {
            // Get the image file to scan from the user.
            var openDlg = new Microsoft.Win32.OpenFileDialog();

            openDlg.Filter = "JPEG Image(*.jpg)|*.jpg";
            bool? result = openDlg.ShowDialog(this);

            // Return if canceled.
            if (!(bool)result)
            {
                return;
            }

            // Display the image file.
            string filePath = openDlg.FileName;

            Uri fileUri = new Uri(filePath);
            BitmapImage bitmapSource = new BitmapImage();

            bitmapSource.BeginInit();
            bitmapSource.CacheOption = BitmapCacheOption.None;
            bitmapSource.UriSource = fileUri;
            bitmapSource.EndInit();

            FacePhoto.Source = bitmapSource;

            // Detect any faces in the image.
            Title = "Detecting...";
            faces = await UploadAndDetectFaces(filePath);
            Title = String.Format("Detection Finished. {0} face(s) detected", faces.Length);

            if(faces.Length > 0)
            {
                // Prepare to draw rectangles around the faces.
                DrawingVisual visual = new DrawingVisual();
                DrawingContext drawingContext = visual.RenderOpen();
                drawingContext.DrawImage(bitmapSource, new Rect(0, 0, bitmapSource.Width, bitmapSource.Height));
                double dpi = bitmapSource.DpiX;
                resizeFactor = 96 / dpi;
                faceDescriptions = new String[faces.Length];

                for (int i = 0; i < faces.Length; ++i)
                {
                    Face face = faces[i];

                    // Draw a rectangle on the face.
                    drawingContext.DrawRectangle(
                        Brushes.Transparent,
                        new Pen(Brushes.Red, 2),
                        new Rect(
                            face.FaceRectangle.Left * resizeFactor,
                            face.FaceRectangle.Top * resizeFactor,
                            face.FaceRectangle.Width * resizeFactor,
                            face.FaceRectangle.Height * resizeFactor
                            )
                    );

                    // Store the face description.
                    faceDescriptions[i] = FaceDescription(face);
                }

                drawingContext.Close();

                // Display the image with the rectangle around the face.
                RenderTargetBitmap faceWithRectBitmap = new RenderTargetBitmap(
                    (int)(bitmapSource.PixelWidth * resizeFactor),
                    (int)(bitmapSource.PixelHeight * resizeFactor),
                    96,
                    96,
                    PixelFormats.Pbgra32);

                faceWithRectBitmap.Render(visual);
                FacePhoto.Source = faceWithRectBitmap;

                // Set the status bar text.
                faceDescriptionStatusBar.Text = "Place the mouse pointer over a face to see the face description.";
            }
        }

Add a new await function named UploadAndDetectFaces(), which accepts imageFilePath as an object parameter.

// Uploads the image file and calls Detect Faces.
        private async Task<Face[]> UploadAndDetectFaces(string imageFilePath)
        {
            // The list of Face attributes to return.
            IEnumerable<FaceAttributeType> faceAttributes =
                new FaceAttributeType[] { FaceAttributeType.Gender, FaceAttributeType.Age, FaceAttributeType.Smile, FaceAttributeType.Emotion, FaceAttributeType.Glasses, FaceAttributeType.Hair };

            // Call the Face API.
            try
            {
                using (Stream imageFileStream = File.OpenRead(imageFilePath))
                {
                    Face[] faces = await faceServiceClient.DetectAsync(imageFileStream, returnFaceId: true, returnFaceLandmarks: false, returnFaceAttributes: faceAttributes);
                    return faces;
                }
            }
            // Catch and display Face API errors.
            catch (FaceAPIException f)
            {
                MessageBox.Show(f.ErrorMessage, f.ErrorCode);
                return new Face[0];
            }
            // Catch and display all other errors.
            catch (Exception e)
            {
                MessageBox.Show(e.Message, "Error");
                return new Face[0];
            }
        }

Below is the code which appends the string as output in the status bar.

// Returns a string that describes the given face.
        private string FaceDescription(Face face)
        {
            StringBuilder sb = new StringBuilder();

            sb.Append("Face: ");

            // Add the gender, age, and smile.
            sb.Append(face.FaceAttributes.Gender);
            sb.Append(", ");
            sb.Append(face.FaceAttributes.Age);
            sb.Append(", ");
            sb.Append(String.Format("smile {0:F1}%, ", face.FaceAttributes.Smile * 100));

            // Add the emotions. Display all emotions over 10%.
            sb.Append("Emotion: ");
            EmotionScores emotionScores = face.FaceAttributes.Emotion;
            if (emotionScores.Anger >= 0.1f) sb.Append(String.Format("anger {0:F1}%, ", emotionScores.Anger * 100));
            if (emotionScores.Contempt >= 0.1f) sb.Append(String.Format("contempt {0:F1}%, ", emotionScores.Contempt * 100));
            if (emotionScores.Disgust >= 0.1f) sb.Append(String.Format("disgust {0:F1}%, ", emotionScores.Disgust * 100));
            if (emotionScores.Fear >= 0.1f) sb.Append(String.Format("fear {0:F1}%, ", emotionScores.Fear * 100));
            if (emotionScores.Happiness >= 0.1f) sb.Append(String.Format("happiness {0:F1}%, ", emotionScores.Happiness * 100));
            if (emotionScores.Neutral >= 0.1f) sb.Append(String.Format("neutral {0:F1}%, ", emotionScores.Neutral * 100));
            if (emotionScores.Sadness >= 0.1f) sb.Append(String.Format("sadness {0:F1}%, ", emotionScores.Sadness * 100));
            if (emotionScores.Surprise >= 0.1f) sb.Append(String.Format("surprise {0:F1}%, ", emotionScores.Surprise * 100));

            // Add glasses.
            sb.Append(face.FaceAttributes.Glasses);
            sb.Append(", ");

            // Add hair.
            sb.Append("Hair: ");

            // Display baldness confidence if over 1%.
            if (face.FaceAttributes.Hair.Bald >= 0.01f)
                sb.Append(String.Format("bald {0:F1}% ", face.FaceAttributes.Hair.Bald * 100));

            // Display all hair color attributes over 10%.
            HairColor[] hairColors = face.FaceAttributes.Hair.HairColor;
            foreach (HairColor hairColor in hairColors)
            {
                if (hairColor.Confidence >= 0.1f)
                {
                    sb.Append(hairColor.Color.ToString());
                    sb.Append(String.Format(" {0:F1}% ", hairColor.Confidence * 100));
                }
            }

            // Return the built string.
            return sb.ToString();
        }

Finally, add a function, which returns the string on mouse hover over the image.

private void FacePhoto_MouseMove(object sender, MouseEventArgs e)
        {
            // If the REST call has not completed, return from this method.
            if (faces == null)
                return;

            // Find the mouse position relative to the image.
            Point mouseXY = e.GetPosition(FacePhoto);

            ImageSource imageSource = FacePhoto.Source;
            BitmapSource bitmapSource = (BitmapSource)imageSource;

            // Scale adjustment between the actual size and displayed size.
            var scale = FacePhoto.ActualWidth / (bitmapSource.PixelWidth / resizeFactor);

            // Check if this mouse position is over a face rectangle.
            bool mouseOverFace = false;

            for (int i = 0; i < faces.Length; ++i)
            {
                FaceRectangle fr = faces[i].FaceRectangle;
                double left = fr.Left * scale;
                double top = fr.Top * scale;
                double width = fr.Width * scale;
                double height = fr.Height * scale;

                // Display the face description for this face if the mouse is over this face rectangle.
                if (mouseXY.X >= left && mouseXY.X <= left + width && mouseXY.Y >= top && mouseXY.Y <= top + height)
                {
                    faceDescriptionStatusBar.Text = faceDescriptions[i];
                    mouseOverFace = true;
                    break;
                }
            }

            // If the mouse is not over a face rectangle.
            if (!mouseOverFace)
                faceDescriptionStatusBar.Text = "Place the mouse pointer over a face to see the face description.";
        }

Step 9 

Build the application and run it. You will see the output as below.

Summary

Therefore, in this part, we saw that by writing very less code, we can use Microsoft Cognitive Services Face API.

Part 3 – EMOTION API

Emotion API provides the advanced Emotion algorithms and it has two main functions:-

1. Emotion Recognition
   
   
   1. It takes image as an input and returns the confidence of the emotions for each face in the image.
   2. The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.
   3. Emotion scores are normalized to sum to one.
      
      
2. Emotion in Video
   
   
   1. It takes video as an input and returns the confidence across a set of emotions for the group of faces in the image over a period.
   2. The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.
   3. It returns two types of aggregates,
      
      
      • windowMeanScores gives a mean score for all of the faces detected in a frame for each emotion.
      • windowFaceDistribution gives the distribution of faces with each emotion as the dominant emotion for that face.

Now, let’s create a C# Console application to test Microsoft Cognitive Services Emotion API.

Step 1 

We have already created the Endpoint URL and subscription keys in Azure for Emotion API. We will create a Console Application in Visual Studio 2015.

Step 2 

Replace the Program.cs with the following code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;

namespace Emotion_API_Tutorial
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Enter the path to a JPEG image file:");
            string imageFilePath = Console.ReadLine();

            MakeRequest(imageFilePath);

            Console.WriteLine("\n\n\nWait for the result below, then hit ENTER to exit...\n\n\n");
            Console.ReadLine();
        }

        static byte[] GetImageAsByteArray(string imageFilePath)
        {
            FileStream fileStream = new FileStream(imageFilePath, FileMode.Open, FileAccess.Read);
            BinaryReader binaryReader = new BinaryReader(fileStream);
            return binaryReader.ReadBytes((int)fileStream.Length);
        }

        static async void MakeRequest(string imageFilePath)
        {
            try
            {
                var client = new HttpClient();
                // Request headers
                client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "<Enter Your Key Value>");

                //Endpoint URL
                string uri = "https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize?";

                HttpResponseMessage response;
                string responseContent;

                // Request body.
                byte[] byteData = GetImageAsByteArray(imageFilePath);

                using (var content = new ByteArrayContent(byteData))
                {
                    // This example uses content type "application/octet-stream".
                    // The other content types you can use are "application/json" and "multipart/form-data" and application/octet-stream.
                    content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                    response = await client.PostAsync(uri, content);
                    responseContent = response.Content.ReadAsStringAsync().Result;
                }

                //A peak at the JSON response.
                Console.WriteLine(responseContent);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error occured:= " + ex.Message + ex.StackTrace);
                Console.ReadLine();
            }
        }
    }
}

Step 3 

Build and run the application. A successful call will return an array of face entries and their emotion scores. An empty response indicates that no faces were detected. An emotion entry contains the following fields.

• faceRectangle - Rectangle location of face in the image.
• scores - Emotion scores for each face in the image.

Below is the output of my trial run. Provide the path of an image from your local folder and hit Enter.

[
  {
    "faceRectangle": {
      "height": 44,
      "left": 62,
      "top": 36,
      "width": 44
    },
    "scores": {
      "anger": 0.000009850864,
      "contempt": 1.073325e-8,
      "disgust": 0.00000230705427,
      "fear": 1.63113334e-9,
      "happiness": 0.9999875,
      "neutral": 1.00619431e-7,
      "sadness": 1.13927945e-9,
      "surprise": 2.365794e-7
    }
  }
]

Summary

Therefore, in this part, we saw that by writing much less code, we could use Microsoft Cognitive Services Emotion API.