Multiple People Emotions In UWP With Cognitive Service Emotion API, Azure, XAML And C#

Before reading this article, please go through the following article.

1. Introduction To Universal Windows Platform (UWP) App Development Using Windows 10 And Visual Studio 2015
2. How To Create Microsoft Cognitive Service Emotion API On Azure Portal
   
3. Cognitive Service Emotion API In UWP With Azure, XAML And C#

Microsoft Cognitive Services lets you build apps with powerful algorithms, using just a few lines of code. They work across devices and platforms such as iOS, Android, and Windows, keep improving, and are easy to set up.

The emotions detected are anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise. These emotions are understood to be cross-culturally and universally communicated with particular facial expressions.

Reading this article, you can learn how to use Multiple People Emotions in Universal Windows apps development with Cognitive Service Emotion API, Azure, XAML and Visual C#. Here, input is a captured image and Emotion API returns different emotions like anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise in different people.

The following important tools are required for developing UWP,

1. Windows 10 (Recommended)
2. Visual Studio 2015 Community Edition (It is a free software available online).
3. Emotion API Key - (How To Create Microsoft Cognitive Service Emotion API On Azure Portal)

Now, we can discuss step by step app development.

Step1

Open Visual Studio 2015 -> Start -> New Project-> Select Universal (under Visual C#->Windows)-> Blank App -> Give a suitable name to your app (UWPCogGroupEmot) ->OK.



After choosing the Target and Minimum platform version that your Windows Universal application will support the Project creates App.xaml and MainPage.xaml.



Step 2

Add a Grid with two columns - for photo and for emotion results.



Add a Stack Panel and Border Control. Inside the Border Control, add a Canvas Control for displaying a photo.



Add a Button Control and set the name for View Emotions.



Add another Grid for Column 2 and add the Listview Control for Emotion Results.



Add the "Click Event" method for the Button Control.



Add the TextBlock Control for Emotion Result title.



Add the Rectangle image for drawing a rectangle on each face.



Step 3

Open (double click) the file MainPage.xaml in Solution Explorer and add the following references in the project.

1. Microsoft.ProjectOxford.Face
2. Newtonsoft.Json

For adding Microsoft.ProjectOxford.Emotion Reference, right click your project (UWPCogGroupEmot) and select "Manage NuGet Packages".

Choose Browse and search Microsoft.ProjectOxford.Emotion. Select the package and install it.

Reference is now added to your project. For adding Newtonsoft.Json Reference, choose Browse and search Newtonsoft.Json. Select the package and install it.



Reference is added to your project



Automatically, the following code will be generated in XAML code view, while we are done in the design view.

<Page x:Class="UWPCogGroupEmot.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:local="using:UWPCogGroupEmot" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d">
    <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        <TextBlock x:Name="tblTitle" HorizontalAlignment="Left" Margin="313,88,0,0" TextWrapping="Wrap" Text="UWP Coginitive Service Muliple People Emotions Demo" VerticalAlignment="Top" FontSize="24" FontWeight="Bold" />
        <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}" BorderThickness="5" Margin="82,125,50,50">
            <Grid.ColumnDefinitions>
                <ColumnDefinition Width="*" />
                <ColumnDefinition Width="*" />
            </Grid.ColumnDefinitions>
            <StackPanel Grid.Column="0" VerticalAlignment="Center">
                <Border x:Name="BorPhoto" BorderBrush="Black" BorderThickness="1" HorizontalAlignment="Left" Height="437" Margin="51,0,0,0" Width="495">
                    <Canvas x:Name="CANPhoto" Width="598" Height="598" />
                </Border>
            </StackPanel>
            <Button x:Name="btnEmotions" Content="View Emotions" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Height="39" Margin="374,0,10,506" Click="btnEmotions_Click" RenderTransformOrigin="0.989,0.406" FontWeight="Bold" />
            <Grid Grid.Column="2">
                <ListView x:Name="lstEmotionRes" HorizontalAlignment="Stretch" Margin="248,63,10,52" VerticalAlignment="Stretch" Height="420" />
            </Grid>
        </Grid>
        <TextBlock x:Name="tblEmotion" HorizontalAlignment="Left" Margin="959,140,0,0" TextWrapping="Wrap" Text="Emotions are," VerticalAlignment="Top" Height="30" Width="187" FontWeight="Bold" FontSize="22" />
    </Grid>
</Page>

Step 4

Add the following namespaces in Mainpage.xaml.cs for capturing a photo and getting emotions.

using Microsoft.ProjectOxford.Emotion;
using Microsoft.ProjectOxford.Emotion.Contract;
using Windows.UI.Xaml.Media.Imaging;
using System.Threading.Tasks;
using Windows.Graphics.Imaging;
using Windows.Storage.Streams;

Step 5

Add the EmotionServiceClient key, use Azure service, and generate it (For More Information, please refer the article How To Create Microsoft Cognitive Service Emotion API On Azure Portal.

Add the following code with Azure generated key for EmotionServiceClient.

const string APIKey = "Your Emotion Key";
EmotionServiceClient emotionServiceClient = new EmotionServiceClient(APIKey);
Add the code
for Multiple People Emotion Details from the image url,
const string APIKey = "Your Emotion Key";
string urlStr = "https://pbs.twimg.com/media/CtrrCSYUMAEanQI.jpg";
Uri uri;
BitmapImage bitMapImage;
public MainPage() {
    this.InitializeComponent();
    CANPhoto.Children.Clear();
    try {
        uri = new Uri(urlStr, UriKind.Absolute);
    } catch (UriFormatException ex) {
        return;
    }
    bitMapImage = new BitmapImage();
    bitMapImage.UriSource = uri;
    ImageBrush imgBrush = new ImageBrush();
    imgBrush.ImageSource = bitMapImage;
    CANPhoto.Background = imgBrush;
}
private async void btnEmotions_Click(object sender, RoutedEventArgs e) {
    Emotion[] emotionResult = await PhotoUploadAndDetectEmotions();
    resultdisplay(emotionResult);
    DrawFaceRectangle(emotionResult, bitMapImage);
}
private async Task < Emotion[] > PhotoUploadAndDetectEmotions() {

    EmotionServiceClient emotionServiceClient = new EmotionServiceClient(APIKey);
    try {
        Emotion[] emotionResult = await emotionServiceClient.RecognizeAsync(urlStr);
        return emotionResult;
    } catch (Exception exception) {
        return null;
    }
}
private void resultdisplay(Emotion[] resultList) {
    int index = 0;
    foreach(Emotion emotion in resultList) {
        lstEmotionRes.Items.Add("\nFace #" + index +
            "\nAnger: " + emotion.Scores.Anger +
            "\nContempt: " + emotion.Scores.Contempt +
            "\nDisgust: " + emotion.Scores.Disgust +
            "\nFear: " + emotion.Scores.Fear +
            "\nHappiness: " + emotion.Scores.Happiness +
            "\nSurprise: " + emotion.Scores.Surprise);
        index++;
    }
}
public async void DrawFaceRectangle(Emotion[] emotionResult, BitmapImage bitMapImage) {
    if (emotionResult != null && emotionResult.Length > 0) {
        IRandomAccessStream stream = await RandomAccessStreamReference.CreateFromUri(new Uri(urlStr)).OpenReadAsync();
        BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);
        double resizeFactorH = CANPhoto.Height / decoder.PixelHeight;
        double resizeFactorW = CANPhoto.Width / decoder.PixelWidth;
        foreach(var emotion in emotionResult) {
            Microsoft.ProjectOxford.Common.Rectangle faceRect = emotion.FaceRectangle;
            Image Img = new Image();
            BitmapImage BitImg = new BitmapImage();
            Windows.Storage.Streams.IRandomAccessStream box = await Windows.Storage.Streams.RandomAccessStreamReference.CreateFromUri(new Uri("ms-appx:///Assets/rectangle.png", UriKind.Absolute)).OpenReadAsync();
            BitImg.SetSource(box);
            var maxWidth = faceRect.Width * resizeFactorW;
            var maxHeight = faceRect.Height * resizeFactorH;
            var origHeight = BitImg.PixelHeight;
            var origWidth = BitImg.PixelWidth;
            var ratioX = maxWidth / (float) origWidth;
            var ratioY = maxHeight / (float) origHeight;
            var ratio = Math.Min(ratioX, ratioY);
            var newHeight = (int)(origHeight * ratio);
            var newWidth = (int)(origWidth * ratio);
            BitImg.DecodePixelWidth = newWidth;
            BitImg.DecodePixelHeight = newHeight;
            Thickness margin = Img.Margin;
            margin.Left = faceRect.Left * resizeFactorW;
            margin.Top = faceRect.Top * resizeFactorH;
            Img.Margin = margin;
            Img.Source = BitImg;
            CANPhoto.Children.Add(Img);
        }
    }
}

Step 6

Deploy your app in a Local Machine. The output of the UWPCogGroupEmot app is shown below.



After clicking the View Emotions button.



Summary

Now, you have successfully tested Multiple People Emotions with Cognitive Service Emotion API, Azure, XAML AND C# in UWP environment.