Cortana Integration In Universal Windows Platform Apps

Introduction

This article is all about integrating Cortana (Virtual Voice-activated Personal Assistant) with our app.This means, if a user gives any instructions/asks question to Cortana related to our app anything an app can fulfil, Cortana communicates with our app and responds back to the user accordingly. To use Cortana, your device must have proper region set with the language pack installed, which Cortana understands.

Step 1 : Create Windows Universal Project

Open Visual Studio 2015 -> New Project -> Visual C# ->Windows -> Universal -> Blank App.



Step 2 : Add Windows Runtime Component

Now, let's add Windows Runtime Component, which acts as a background AppService for this app and activates whenever the user asks anything to Cortana with the name of our app. We'll see it later.

Right click on solution -> Add new project -> Add Windows Runtime Component.



Step 3 : Add VCD file to project

Now, add a Voice Command Definition (VCD) file into our app, which we need to install on app launch.

A voice command is a single utterance with a specific intent, defined in a Voice Command Definition (VCD) file, directed at an installed app through Cortana.

A VCD file defines one or more voice commands, each with a unique intent.

CortanaUWPDemo -> Add ->New Item ->XML File,



Copy paste the code, given below, in your VCD file.

<?xml version="1.0" encoding="utf-8"?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2 ">
    <!--For English US language-->
    <CommandSet xml:lang="en-us" Name="examplevcd">
        <AppName>Assistant</AppName>
        <Example>Ask your query to assistant regarding travels</Example>
        <Command Name="YourKeyword">
            <Example>I want to travel near areas</Example>
            <ListenFor>{UserCommand}</ListenFor>
            <Feedback>In progress...</Feedback>
            <VoiceCommandService Target="CortanaBackgroundTask" /> </Command>
        <PhraseTopic Label="UserCommand" Scenario="Natural Language"> </PhraseTopic>
    </CommandSet>
    <!--For English-India Language-->
    <CommandSet xml:lang="en-in" Name="examplevcd">
        <AppName>Assistant</AppName>
        <Example>Ask your query to assistant regarding travels</Example>
        <Command Name="YourKeyword">
            <Example>I want to travel near areas</Example>
            <ListenFor>{UserCommand}</ListenFor>
            <Feedback>In progress...</Feedback>
            <VoiceCommandService Target="CortanaBackgroundTask" /> </Command>
        <PhraseTopic Label="UserCommand" Scenario="Natural Language"> </PhraseTopic>
    </CommandSet>
</VoiceCommands>

Here, I added multiple command sets for en-us and en-in., where:

AppName : Cortana will listen to the user only with this app name as a prefix of the sentence (Example :Assistant, I want to travel near areas.).
Feedback : Text shows to the user, while working on creating a user response after a user queries/asks something.
VoiceCommandService : The name of an app service, which activates on calling an AppName or CommandPrefix by the user.
PhraseTopic : You can add multiple phrases inside this PhraseTopic for better response and understandin of the user.

Step 4 : Install VCD file

Let's write a code in our app to install this VCD file.

public async Task InstallVCD()
{
    try {
        StorageFile vcdStorageFile = await Package.Current.InstalledLocation.GetFileAsync(@ "VCD.xml");
        await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(vcdStorageFile);
    } catch (Exception ex) {}
}

Call the method, shown above, from OnLaunched() of app.xaml.cs file.

Step 5 : Add reference of Windows Runtime Project

Now, we are done with making VCD file, installing VCD file from app, added a Runtime component into a solution. It's time to give reference of the background Service to our app.

CortanaUWPDemo -> References -> Right Click ->Add Reference -> Project -> Solution -> Check CortanaUWPDemoBgTask -> click OK.

Step 6 : Add Background task to Windows Runtime Project

Now, the reference of Background Service has been added into your UWP app. Now, we need to add BackgroundTask into Windows Runtime component project.

CortanaUWPDemoBgTask -> Add New item ->Class -> CortanaBackgroundTask implements a IBackgrountTask Interface

public sealed class CortanaBackgroundTask: IBackgroundTask {
    public void Run(IBackgroundTaskInstance taskInstance) {}
}

This background task(Run()) will be called on every voice command by the user.

Now, add some code for Voice Command Connection and get the text out of it. It is what the user said --  here is the updated code for the Background task.

public sealed class CortanaBackgroundTask: IBackgroundTask
{
    BackgroundTaskDeferral serviceDeferral;
    VoiceCommandServiceConnection voiceServiceConnection;
    public async void Run(IBackgroundTaskInstance taskInstance) {
        serviceDeferral = taskInstance.GetDeferral();
        taskInstance.Canceled += OnTaskCanceled;
        var triggerDetails = taskInstance.TriggerDetails asAppServiceTriggerDetails;
        // This should match the uap:AppService and VoiceCommandService references from the
        // package manifest and VCD files, respectively. Make sure we've been launched by
        // a Cortana Voice Command.
        if (triggerDetails != null && triggerDetails.Name == "CortanaBackgroundTask") {
            try {
                voiceServiceConnection = VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails);
                voiceServiceConnection.VoiceCommandCompleted += OnVoiceCommandCompleted;
                // GetVoiceCommandAsync establishes initial connection to Cortana, and must // be called prior to any
                // messages sent to Cortana. Attempting to use // ReportSuccessAsync, ReportProgressAsync, etc
                // prior to calling this will produce undefined behavior.
                VoiceCommand voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync();
                var text = voiceCommand.SpeechRecognitionResult.Text;
                if (text.Equals("I")) {
                    RespondTouser("Hi, How are your doing ?");
                } else {
                    //Here you call up any api's to create a meaning full response to user/call // services like Bot/Luis etc... to create a meaningful response.
                    RespondTouser("You Said ," + text);
                }
            } catch (Exception ex) {}
        }
    }
    private async void RespondTouser(string text) {
        var userMessage = new VoiceCommandUserMessage();
        string keepingTripToDestination = text; //How can i Help you?
        userMessage.DisplayMessage = userMessage.SpokenMessage = keepingTripToDestination;
        VoiceCommandResponse response = VoiceCommandResponse.CreateResponse(userMessage);
        await voiceServiceConnection.ReportSuccessAsync(response);
    }
    private void OnVoiceCommandCompleted(VoiceCommandServiceConnection sender, VoiceCommandCompletedEventArgs args) {
        if (this.serviceDeferral != null) {
            this.serviceDeferral.Complete();
        }
    }
    private void OnTaskCanceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason) {
        if (this.serviceDeferral != null) {
            this.serviceDeferral.Complete();
        }
    }
}

Step 7 : Define AppService in Package Manifest file

Now, we are done with our Background app Service. Let our UWP app know about it.

Add the code, given below, in package.appmanifest file.

<Extensions>
    <uap:Extension Category="windows.appService" EntryPoint="CortanaUWPDemoBgTask.CortanaBackgroundTask">
        <uap:AppService Name="CortanaBackgroundTask" /> </uap:Extension>
    <uap:Extension Category="windows.personalAssistantLaunch" />
</Extensions>

Now, we are done with all the stuff. It's time to run our app. 

Output Video

Check out here : 1tsgENcMPk

Source Code

Download the full source code from here.