Power Apps With Azure Cognitive Services

Introduction 

 

This article is to show a step-by-step procedure on how to integrate Azure Cognitive Services (Computer Vision API) into Power Apps.

 

Agenda

1. Create an Azure Cognitive Services (Computer Vision API)
2. Creating a connector to the created API above in the power apps portal. 
3. Create a new app using power apps and adding the connector created above.
4. Power app functions and methods on how to connect to the different functionality of the APIs.
5. Leveraging the use of Power automate, wherever required.

Creating Azure Cognitive services (Computer Vision API)

 

Step 1

 

If you have a paid Azure paid subscription it's all and well, if not, get a trail version for using this services. Please find the details of this service here 

 

Step 2

 

Go to https://portal.azure.com/, click on Create a Resource, under AI + Machine Learning, select "Computer Vision".

 

 

Step 3

 

Fill in the required fields and click on "Review + Create".

 

 

Step 4

 

Once the services are created, click the "Manage Keys" link to get the keys for the created resource.

 

 

Get the information about the keys and the endpoints, which is required later.

 

 

Please go through the functionality of this API here. 

 

Creating a connector in the power apps portal

 

Step 1

 

Go to https://make.powerapps.com/, Under Data -> Connections -> New Connection.

 

Step 2

 

Search for Computer Vision API and click on create button.

 

Step 3

 

Enter the key of the created computer vision API and the endpoint URL in the dialog box and click on create to create a new connection.

 

 

Create a new app in the powerapps portal using a newly created connector.

 

Step 1

 

Go to https://make.powerapps.com/, Create -> Canvas app from blank -> Phone layout -> Click Create.

 

Step 2

 

In the power apps studio, Go to View -> Data Source and Add the newly created API to this app.

 

 

Creating different Power functions using the newly added API

 

Step 1

 

Go to Insert -> Media -> Camera. The camera control will be added. In the Camera onSelect Control added the following "ClearCollect(ImageColl, Camera1.Photo)"

 

Step 2

 

Go to Insert -> Media -> Image. The image control will be added. In the Image control, add the following "First(ImageColl).Url"

 

What we are doing here is, displaying the image capture from the Camera control into the Image control.

 

Step 3

 

Add a button control, OnSelect function paste the following "Set(ScannedResult, ComputerVisionAPI.DescribeImageContentV2(First(ImageColl).Url,{language:"en",maxCandidates:5}).description)"  

 

What happens here is, we are using the Computer vision API to describe the image we capture using camera and storing the result in ScannedResult variable.

 

 

In Power Apps for the computer vision API, some functionality of the API is limited certain functionalities. Example on the Analysis of the image, OCRText, OCRJSON, etc.

 

In the above image, you can see the file as an input, but the mentioned functions above do not have that input parameter inside power apps. Please refer to the image below.

 

 

To handle this, we need to use Power automate as a workaround to solve this issue.

 

Using Power Automate to leverage the functionality of the OCR and other limited functions of the computer vision API in Power Apps.

 

Step 1

 

Create an MS Flow with the power apps as the trigger.

 

Step 2

 

Create a test file in OneDrive from the input received from PowerApps, which is the base64 value of the captured image.

 

A detailed explanation of the above step:

 

The image we captured in the Camera control and stored in the Image control, when displayed an label will look something like "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAYAAA......" ie an base64 file format.

 

Keep in mind, the input value to be sent to the computer vision parameter should be in binary format, so we have to convert it into a binary value in Power Automate and the base64 should not have the initial "data:image/png;base64," while sending it to power automate.

 

In Power Automate:

 

 

 

In Power Apps on any button click event:

 

 

 

The test file is created to make sure, the image captured is replicated exactly before sending it to the Computer Vision API.

 

Step 3

 

Send the file content of the recreated image to the Computer vision API and send back the response to PowerApps.

 

Entire Power automate flow:

 

 

I hope that this article serves you well. Please find the attached Power App and Power Automate project files, explained above.