Chatbot Image Recognization

Introduction

 

In my previous article, I discussed the language understanding tool for the chatbot. We will continue here by adding some services of LUIS to our ChatBot, by using Visual Studio. Also, we will see how to add visual responses using adaptive cards with the chatbot. All the lab tools are remaining the same, we will just move with the coding section using Microsoft Visual Studio as an IDE. Natural interfaces are the same as to understand the voice, text, and touch. In this article, I'll show you a user clicking on an item from an image carousel. Adaptive cards enable a developer to define the user interface element using JSON by the host platform in order to build an intelligent bot.

 

Before moving towards the visual response, we have to add a service to our bot.

 

Install LUIS package

 

Step 1

 

Click FILE from Visual Studio. 

 

Step 2

 

Click Open Project/ Solution.

 

Step 3

 

Select the Solution file Downloads\buildanintelligentbot\src\TalkToMyBot.sln wait until it's loaded.

 

Step 4

 

Click the Solution and then click Restore NuGet Packages. 

 

Check out this image:

 

 

Now, we will...

 

Install the LUIS package from NuGet

 

Step 1 

 

Right-click on the ChatBot project and click Manage NuGet Packages.

 

 

Step 2

 

Select the Browse tab and search for Microsoft.Bot.Builder.AI.Luis

 

Step 3

 

Click on the NuGet package. Select Version 4.4 and click install. Higher or latest versions might raise errors.

 

These steps are both shown in this image:

 

 

Add the LUIS recognizer to our Bot

 

LUIS is a cognitive service that is accessible through RESTful endpoints. This Bot Builder SDK has inbuilt middleware that integrates the code processing for this intent, not on natural language.

 

We have to follow these steps:

 

Step 1 

 

Go to Visual Studio, Open the appsettings.json file

 

Step 2

 

Update the following configuration values.

 

Replace <your_luis_app_id> and <your_luis_api_key> with the values that you had stored in Notepad previously from the LUIS Portal.

 

Step 3

 

Now open the Startup.cs file.

 

Step 4

 

Add the following namespace at the top of this file. 

1. using Microsoft.Bot.Builder.AI.Luis;

Step 5

 

Find the comment // and create Luis recognizer around line 67, then replace it with this code snippet:

var luisApplication = new LuisApplication(
    Configuration["LuisAppId"],
    Configuration["LuisAPIKey"],
    "https://" + Configuration["LuisAPIHostName"]);
services.AddSingleton(new LuisRecognizer(luisApplication));

Adjust our Bot to process the Results

 

Do some modifications in the code and get the results from LUIS. 

 

Step 1

 

Open EchoBot.cs

 

Add the following namespace at the top of this file.

using Microsoft.Bot.Builder.AI.Luis;

Step 2

 

Go to this comment:  // Add Luis Recognizerand replace it with this line:

protected LuisRecognizer _luis;

Step 3

 

Constructor header is replaced with:

public EchoBot(EchoBotAccessors accessors, IOptions<MySettings> config, LuisRecognizer luisRecognizer)

Step 4

 

Goto the comment line: // Initialize the LUIS recognizer and replace it with this line:

_luis = luisRecognizer;

Step 5

 

Check for the comment: // Handle LUIS intents recognition and replace it with this code snippet:

// Perform a call to LUIS to retrieve results for the current activity message.
var luisResults = await _luis.RecognizeAsync(turnContext, cancellationToken).ConfigureAwait(false);
var topScoringIntent = luisResults?.GetTopScoringIntent();
var topIntent = topScoringIntent.Value.score > 0.5 ? topScoringIntent.Value.intent : string.Empty;

// Your code goes here

Step 6

 

Recheck Step 5 again, you will find a line // Your code goes here  and replace it with this code snippet:

switch (topIntent)
{
    case "TodaysSpecialty":
        await turnContext.SendActivityAsync($"For today we have the following options: {string.Join(", ", BotConstants.Specialties)}");
        break;
    default:
        await turnContext.SendActivityAsync("Sorry, I didn't understand that.");
        break;
}

Note

This switch statement sends the user's message to the right handler based on LUIS intent.

 

Test LUIS Configuration

 

Step 1

 

Run the ChatBot by clicking on the IIS Express button in Visual Studio. 

 

Step 2

 

Once the page has loaded in Web Browser, copy the site host from the address bar to Notepad. (Like, http://localhost:XXXX)

 

 

Step 3

 

Come to the Bot Framework Emulator.

 

Step 4

 

Click file -> New Bot Configuration

 

Step 5

 

Paste the URL and add /api/messages it at the end of the URL. (Like, http://locslhost:XXXX/api/messages)

 

 

Step 6 

 

Click on the Save and Connect button.

 

Step 7 

 

Type what is the specialty for today? Then press Enter. 

 

LUIS is processing the input and the bot can handle it accordingly. 

 

 

Step 8

 

Stop Debugging by clicking the stop button in the Visual Studio toolbar.

 

Add a Visual Response To Your Bot: Carousel

 

In Visual Studio open EchoBot.cs 

 

Go to line 54:

private async Task TodaysSpecialtiesHandlerAsync(ITurnContext context)
{
    var actions = new[]
    {
        new CardAction(type: ActionTypes.ShowImage, title: "Carbonara", value: "Carbonara", image: $"{BotConstants.Site}/carbonara.jpg"),
        new CardAction(type: ActionTypes.ShowImage, title: "Pizza", value: "Pizza", image: $"{BotConstants.Site}/pizza.jpg"),
        new CardAction(type: ActionTypes.ShowImage, title: "Lasagna", value: "Lasagna", image: $"{BotConstants.Site}/lasagna.jpg"),
    };

    var cards = actions
      .Select(x => new HeroCard
      {
          Images = new List<CardImage> { new CardImage(x.Image) },
          Buttons = new List<CardAction> { x },
      }.ToAttachment())
      .ToList();
    var activity = (Activity)MessageFactory.Carousel(cards, "For today we have:");

    await context.SendActivityAsync(activity);
}

Add this method.

 

Let's modify the OnTurnAsync method, so replace this line: 

await turnContext.SendActivityAsync($"For today we have the following options: {string.Join(", ", BotConstants.Specialties)}");

With:

await TodaysSpecialtiesHandlerAsync(turnContext);

TEST THE VISUAL RESPONSE

 

Follow these steps:

 

Step 1

 

Run the app, click on IIS Express button(green play icon) in Visual Studio.

 

Step 2 

 

Open the Models/BotConstants.cs file.

  

Step 3

 

Check Site constant. If there is a different URL, then stop the app, update it with the right port, and run this app again.

 

Step 4

 

Open the Bot Framework Emulator and click on Restart Conversation. 

 

Step 5

 

Type this query line: What is today's specialty? Press enter.  

 

You can see the set of images in a Carousel.

 

At last, Stop debugging in Visual Studio. 

 

 

In this image, there is a kind of online food ordering app, all the recommended options are in front of the viewer and according to the visual recognition, it also looks too good comparatively text reverts from the ChatBot.

 

Chatbot has many modules such as; foram module: QnA,  Speech Recognization, Voice Recognization, Face Recognization. By using different tools and technologies, we can explore the variety of new techniques to invent something unique and a worthy product for the end-users as well as the experts.   

 

Reference

 

https://aischool.microsoft.com/en-us/conversational/learning-paths/building-an-intelligent-bot

 

For any programming file, DM me, please.

 

Summary

 

In this article, we discussed recognition with a ChatBot using text and visual effects with the help of LUIS and Visual Recognization respectively. There are many different types of modules and usage is going to be included with the chatbot. It all depends on the curiosity and the invention technics of the developer. Kindly check the attachments or links with this article.