Post
Ask Question

Using Google Vision API With ASP.NET MVC

In this article, we are going to learn how to Use Google Vision API with ASP.NET MVC in a step by step way.

Message it’s for Content Editors

Make this code available for download [the size of the source code is large that's why I cannot upload it from the portal].

Make this link available for download:

https://www.dropbox.com/s/wism6v175hm6tg2/WebCamAppVision.rar?dl=0
ASP.NET

Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY

Why do we use Google Vision Face API?

Cloud Vision API allows developers to easily integrate vision detection features within applications, including image labeling, face and landmark detection, optical character recognition (OCR), and tagging of explicit content.

Referenced from

https://cloud.google.com/vision/docs/

There are all giant techie companies  and they compete with each other.

As you can see I have already written an article on Azure Face API. 

http://www.c-sharpcorner.com/article/using-azure-face-api-with-asp-net-mvc/   

Note

After downloading the source code, just change the “Key” to make it work.

Process Flow

  • Creating an ASP.NET MVC project.
  • Get an API key for using CLOUD VISION API.
  • Using HTML Canvas to capture photo and detect labels
  • Using HTML Canvas to capture photo and detect faces
  • Finally, displaying Google CLOUD VISION API Response

Creating an ASP.NET MVC project

After opening IDE, next, we are going to create an ASP.NET MVC project. For doing that, just click File - New - Project.

After choosing a project, a new dialog will pop up with the name "New Project". In that, we are going to choose Visual C# Project Templates Web ASP.NET Web Application. Then, we are going to name the project  “WebCamAppVision".

After naming the project, click on the OK button to create the project. A new dialog will pop up with the name “New ASP.NET Project”; from that, we are going to choose “MVC” templates for creating "MVC" applications and we are not going to use any authentication in this application. For that, we are going to choose “No Authentication”. After that, finally, click on the OK button to create the project.

ASP.NET

After completing creating the project ,next, we are going to Get API Key from Google Cloud portal.

Getting an API key for using Google Vision API

For getting an API key, you must register at Google Cloud portal.

Google Cloud is also free for 1 year with rupees credits: 19,060.50

After logging into Google Cloud portal, click on the link below to start with Vision API.

https://cloud.google.com/vision/docs/before-you-begin

ASP.NET

After accessing it,  click on “ENABLE THE API” button and it will take you to where you can enable the API page where you need to create a project, or select an already-created a project to register your Google Cloud Vision API.

We are using Google API for the first time, that’s why we are going to create a project first.

Creating a project

For creating project choose the Create a project drop-down and click on continue button.

ASP.NET

After clicking on continue button it will create a project and it will display notification, as shown below. 

ASP.NET

Next click on “Create Project: My Project” Notification.

After clicking it will take you to Dashboard page of that API.

ASP.NET

Next click on the go to APIS overview link.

After clicking on APIS overview link it will take you to APIs & Services Dashboard.

Next, we are going to enable “Google Cloud Vision API” and for doing that we are going to click on Library menu.

ASP.NET

After clicking on Library Menu it will take you to the API Library page.

ASP.NET

Next, in this API library we are going to search “Google Cloud Vision API”.

ASP.NET

After searching just click on the “Google Cloud Vision API” panel  and it will open “Google Cloud Vision API” service and on the page, you will see Enable button just click on it.

ASP.NET

After clicking on Enable Button an alert will pop up asking for enabling billing, so just click on “Enable billing”.

ASP.NET

After clicking on Enable billing it will ask to set a billing account.

ASP.NET

After enabling API, next it will take you to API dashboard and here we are going to create Credentials.

 Creating Credentials

ASP.NET

On this dashboard you will see the Credentials Menu, just click on it to Create Credentials for API.

After clicking on Credentials you will see Create Credentials Dropdown and after clicing on the Dropdown, you will see various options as shown below.

ASP.NET

Click on API Key, then it will pop up your API KEY.

ASP.NET

After getting the keys, we are going to add “CamGoogle” Controller.

Adding CamGoogle Controller

For adding a controller, just right click on Controller folder and then choose -> Add -> inside that, choose Add New item. A new dialog will pop up for adding a new item. Inside that, choose "MVC Controller Class" and name your controller "CamGoogle" and click on the "Add" button to create a CamGoogle Controller.

ASP.NET

After adding a controller, we are going to add "Capture Action Method" in it for handling the HTTP GET Request.

Adding Capture Action Method

ASP.NET

After adding Capture Action Method, we are going to add Capture View.

Adding Capture View

ASP.NET

After adding Capture view, next, I have added a CamScripts folder which contains the script for displaying html5 canvas object.

ASP.NET

Complete Code Snippet of Capture View

  1. @{  
  2.     Layout = null;  
  3. }  
  4.   
  5. <!DOCTYPE html>  
  6. <html>  
  7. <head>  
  8.     <meta charset="utf-8">  
  9.     <meta http-equiv="X-UA-Compatible" content="IE=edge">  
  10.     <meta name="viewport" content="width=device-width, initial-scale=1">  
  11.     <title>Demo: Take a Selfie With JavaScript</title>  
  12.     <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">  
  13.   
  14.     <link href="~/CamScripts/css/styles.css" rel="stylesheet" />  
  15.     <link href="~/Content/bootstrap.css" rel="stylesheet" />  
  16. </head>  
  17. <body>  
  18.     <h3>  
  19.         Demo: Take a Photo  
  20.     </h3>  
  21.     <div class="container">  
  22.         <div class="col-md-2"></div>  
  23.         <div class="col-md-6">  
  24.             <div class="app">  
  25.                 <a href="#" id="start-camera" class="visible">Touch here to start the app.</a>  
  26.                 <video id="camera-stream"></video>  
  27.                 <img id="snap">  
  28.                 <p id="error-message">  
  29.                 </p>  
  30.                 <div class="controls">  
  31.                     <a href="#" id="delete-photo" title="Delete Photo" class="disabled">  
  32.                         <i class="material-icons">  
  33.                             delete  
  34.                         </i>  
  35.                     </a> <a href="#" id="take-photo" title="Take Photo">  
  36.                         <i class="material-icons">  
  37.                             camera_alt  
  38.                         </i>  
  39.                     </a> <a href="#" id="download-photo" download="selfie.png" title="Save Photo"  
  40.                             class="disabled"><i class="material-icons">file_download</i></a>  
  41.                 </div>  
  42.                 <!-- Hidden canvas element. Used for taking snapshot of video. -->  
  43.                 <canvas width="300" height="400"></canvas>  
  44.             </div>  
  45.         </div>  
  46.         <div class="col-md-2"></div>  
  47.     </div>  
  48.     <div class="container">  
  49.         <div class="row">  
  50.             <div class="col-md-12">  
  51.                 <div id="ResponseTable">  
  52.                 </div>  
  53.             </div>  
  54.         </div>  
  55.     </div>  
  56.     <script src="~/CamScripts/js/CanvasGoogleScript.js"></script>  
  57.     <script src="https://code.jquery.com/jquery-3.2.1.min.js"></script>  
  58.     <style>  
  59.         .contant {  
  60.             border: 1px solid #ddd;  
  61.             border-radius: 4px;  
  62.             width: 500px;  
  63.             padding: 20px;  
  64.             margin: 0 auto;  
  65.             text-align: center;  
  66.         }  
  67.     </style>  
  68. </body>  
  69. </html>  

After adding view, next we are going save the entire application and test capture view by accessing it.

Snapshot of Capture View

ASP.NET

After accessing Capture View, next, we are going to add code for capturing a photo on clicking capture button, on delete we are going to delete a photo which we have captured and on download, you can download your photo.

After adding View, next, we are going to add “Mainrequests” model to send API request.

Adding Mainrequest Model

In this part, we going to add Mainrequest model which we are going to use for sending the API request.

  1. using System.Collections.Generic;  
  2. namespace WebCamAppVision.Models  
  3. {  
  4.     public class Mainrequests  
  5.     {  
  6.         public List<requests> requests { get; set; }  
  7.     }  
  8.   
  9.     public class requests  
  10.     {  
  11.         public image image { get; set; }  
  12.         public List<features> features { get; set; }  
  13.     }  
  14.   
  15.     public class image  
  16.     {  
  17.         public string content { get; set; }  
  18.     }  
  19.     public class features  
  20.     {  
  21.         public string type { get; set; }  
  22.     }  
  23.   
  24. }  

After adding Model, next, we are going to add [HttpPost] method Capture.

Note

Label Detection detects a broad sets of categories within an image, which range from modes of transportation to animals.

Label Detection

Adding [HttpPost] Capture Method

This method will take base64String as input and this string will be converted to bytes and these bytes will be sent for analyzing the photo.

Code Snippet of [HttpPost] Capture method.

  1. [HttpPost]  
  2.      public ActionResult Capture(string base64String)  
  3.      {  
  4.          var imageParts = base64String.Split(',').ToList<string>();  
  5.          byte[] imageBytes = Convert.FromBase64String(imageParts[1]);  
  6.   
  7.          using (var client = new WebClient())  
  8.          {  
  9.              Mainrequests Mainrequests = new Mainrequests()  
  10.              {  
  11.                  requests = new List<requests>()  
  12.                 {  
  13.                      new requests()  
  14.                 {  
  15.                      image = new image()  
  16.                      {  
  17.                      content = imageParts[1]  
  18.                  },  
  19.   
  20.                  features = new List<features>()  
  21.                  {  
  22.                      new features()  
  23.                      {  
  24.                          type = "LABEL_DETECTION",  
  25.                      }  
  26.   
  27.                  }  
  28.   
  29.              }  
  30.   
  31.              }  
  32.   
  33.              };  
  34.   
  35.   
  36.              var uri = "https://vision.googleapis.com/v1/images:annotate?key=" + "AIzaSyCaB0QqD0APn7l1uMYZH8Kj#############";  
  37.              client.Headers.Add("Content-Type:application/json");  
  38.              client.Headers.Add("Accept:application/json");  
  39.              var response = client.UploadString(uri, JsonConvert.SerializeObject(Mainrequests));  
  40.              return Json(data: response);  
  41.          }  
  42.   
  43.      }  

For sending requests we are going to use web client and mainrequests model, to mainrequests model we are going to assign bytes which we have created from base64String, next we are going to send request to API which is https://vision.googleapis.com/v1/images:annotate?key= . To this API we need to pass the Key which we have generated from Google cloud portal, and in  the final step we are going to add headers and then we are going to serialize model and send request.

After adding [HttpPost] Capture method next we are going have a look at CanvasGoogleScript.js which we are using to capture a photo. After capturing it we get base64String.  We are going to post that base64String to Capture method.

Capturing photo code is written in CanvasGoogleScript.js

ASP.NET

Complete Code Snippet of Ajax Post Request

  1. function takeSnapshot()   
  2. {  
  3.     // Here we're using a trick that involves a hidden canvas element.    
  4.   
  5.     var hidden_canvas = document.querySelector('canvas'),  
  6.         context = hidden_canvas.getContext('2d');  
  7.   
  8.     var width = video.videoWidth,  
  9.         height = video.videoHeight;  
  10.   
  11.     if (width && height) {  
  12.   
  13.         // Setup a canvas with the same dimensions as the video.  
  14.         hidden_canvas.width = width;  
  15.         hidden_canvas.height = height;  
  16.   
  17.         // Make a copy of the current frame in the video on the canvas.  
  18.         context.drawImage(video, 0, 0, width, height);  
  19.   
  20.         // Storing Base64String  
  21.         var datacaptured = hidden_canvas.toDataURL('image/jpeg');  
  22.   
  23.         // Ajax Post to Save Image in Folder  
  24.         Uploadsubmit(datacaptured);  
  25.         // Turn the canvas image into a dataURL that can be used as a src for our photo.  
  26.         return datacaptured;  
  27.     }  
  28. }  
  29.   
  30. function Uploadsubmit(datacaptured)  
  31. {  
  32.   
  33.     if (datacaptured != "") {  
  34.         $.ajax({  
  35.             type: 'POST',  
  36.             url: ("/CamGoogle/Capture"),  
  37.             dataType: 'json',  
  38.             data: { base64String: datacaptured },  
  39.             success: function (data) {  
  40.                 if (data == false) {  
  41.   
  42.                     alert("Photo Captured is not Proper!");  
  43.                     $('#ResponseTable').empty();  
  44.                 }  
  45.                 else {  
  46.   
  47.                     if (data.length == 9) {  
  48.                         $('#ResponseTable').empty();  
  49.                         alert("Its not a Face!");  
  50.                     } else {  
  51.                         $('#ResponseTable').empty();  
  52.                         var _faceAttributes = JSON.parse(data);  
  53.                         var _responsetable = "";  
  54.                         var _emotiontable = "";  
  55.                         _responsetable += '<div class="panel panel-default"><div class="panel-heading">Google Face API Response</div>';  
  56.                         _responsetable += "<div class='panel-body'>"  
  57.                         _responsetable += '<table class="table table-bordered"><thead><tr> <th>Description</th> <th>score</th></tr></thead>';  
  58.   
  59.                         for (var i = 0; i < _faceAttributes.responses[0].labelAnnotations.length; i++) {  
  60.   
  61.                             _responsetable += '<tr><td>' +  
  62.                                 _faceAttributes.responses[0].labelAnnotations[i].description +  
  63.                                 '</td><td>' +  
  64.                                 _faceAttributes.responses[0].labelAnnotations[i].score +  
  65.                                 '</td></tr>';  
  66.   
  67.   
  68.                         }  
  69.   
  70.                         _responsetable += "</table></div></div>"  
  71.                         $('#ResponseTable').append(_responsetable);  
  72.                     }  
  73.                 }  
  74.             }  
  75.         });  
  76.     }  
  77.   
  78. }  

After getting to see the CanvasGoogleScript.js code snippet, next, we are going to add a reference to this script on view.

ASP.NET

Now we have completed adding scripts and views; next we are going to save the application and run it to see a demo.

Capturing photo and Getting response from Google Cloud vision API

  1. Wearing glasses.

In this part, we going to click the photo and send it to google cloud vision API for label detection.

ASP.NET

Note
Label Detection detects a broad sets of categories within an image, which range from modes of transportation to animals.

ASP.NET

Now you can see in the above image which we have captured that I was wearing glasses, and the response which came has the labels “eyeware”, “visioncare”.

  1. Wearing headphones

    ASP.NET

ASP.NET

Now you can see in the above image which we have captured, I was wearing headphones and the response which came has the labels “electronic device”, “audio equipment”, “audio”.

 

  1. Mobile phone.

    ASP.NET

    ASP.NET

    Now you can see in the above image which we have captured the response which came has the labels “black”, “mobile phone”, “electronics”, “electronic device”, “gadget” etc.
  1. Glasses

    ASP.NET
    ASP.NET

Now you can see the above image which we have captured was of glasses and the response which came has the  labels “eyewear”, “glasses”, “vision care”, “goggles”, “sunglasses” etc.

As we saw that it detects objects very well and sends an accurate response, this was all about “LABEL_DETECTION”.

Next, we are going to have a look at Face detection, for doing that we have a little change in code.

Face Detection

In this part, we are going use the Google Cloud Vision API to detect faces in an image. To prove to yourself that the faces were detected correctly, you'll then use that data to draw a box around each face.

Reference taken from- https://cloud.google.com/vision/docs/face-tutorial

For detecting faces we are going to make little changes in the API request; we are going to change the type from “LABEL_DETECTION” to “FACE_DETECTION”.

Code Snippet of [HttpPost] Capture method.

  1. [HttpPost]  
  2.       public ActionResult Capture(string base64String)  
  3.       {  
  4.           var imageParts = base64String.Split(',').ToList<string>();  
  5.           byte[] imageBytes = Convert.FromBase64String(imageParts[1]);  
  6.   
  7.   
  8.           using (var client = new WebClient())  
  9.           {  
  10.               Mainrequests Mainrequests = new Mainrequests()  
  11.               {  
  12.   
  13.                   requests = new List<requests>()  
  14.                       {  
  15.                       new requests()  
  16.                       {  
  17.                       image = new image()  
  18.                       {  
  19.                       content = imageParts[1]  
  20.                   },  
  21.   
  22.                   features = new List<features>()  
  23.                   {  
  24.                       new features()  
  25.                       {  
  26.                           type = "FACE_DETECTION",  
  27.                       }  
  28.                   }  
  29.   
  30.               }  
  31.   
  32.               }  
  33.   
  34.               };  
  35.   
  36.               client.Headers.Add("Content-Type:application/json");  
  37.               client.Headers.Add("Accept:application/json");  
  38.               var response = client.UploadString("https://vision.googleapis.com/v1/images:annotate?key=" + "AIzaSyCaB0QqD0APn7l1uMYZH8############ ", JsonConvert.SerializeObject(Mainrequests));  
  39.   
  40.               return Json(data: response);  
  41.           }  
  42.   
  43.       } 

After changing type in post request next we are going to change the script to get facedetection response and display.

Complete Code Snippet of Ajax Post Request

  1. function takeSnapshot() {  
  2.         // Here we're using a trick that involves a hidden canvas element.    
  3.   
  4.         var hidden_canvas = document.querySelector('canvas'),  
  5.             context = hidden_canvas.getContext('2d');  
  6.   
  7.         var width = video.videoWidth,  
  8.             height = video.videoHeight;  
  9.   
  10.         if (width && height) {  
  11.   
  12.             // Setup a canvas with the same dimensions as the video.  
  13.             hidden_canvas.width = width;  
  14.             hidden_canvas.height = height;  
  15.   
  16.             // Make a copy of the current frame in the video on the canvas.  
  17.             context.drawImage(video, 0, 0, width, height);  
  18.   
  19.             // Storing Base64String  
  20.             var datacaptured = hidden_canvas.toDataURL('image/jpeg');  
  21.   
  22.             // Ajax Post to Save Image in Folder  
  23.   
  24.             //Label Detection  
  25.             //Uploadsubmit(datacaptured);  
  26.             //Face Detection  
  27.             UploadFaceDetection(datacaptured);  
  28.   
  29.             // Turn the canvas image into a dataURL that can be used as a src for our photo.  
  30.             return datacaptured;  
  31.         }  
  32.     }  
  33. function UploadFaceDetection(datacaptured) {  
  34.   
  35.         if (datacaptured != "") {  
  36.             $.ajax({  
  37.                 type: 'POST',  
  38.                 url: ("/CamGoogle/Capture"),  
  39.                 dataType: 'json',  
  40.                 data: { base64String: datacaptured },  
  41.                 success: function (data) {  
  42.                     if (data == false) {  
  43.   
  44.                         alert("Photo Captured is not Proper!");  
  45.                         $('#ResponseTable').empty();  
  46.                     }  
  47.                     else {  
  48.   
  49.                         if (data.length == 9) {  
  50.                             $('#ResponseTable').empty();  
  51.                             alert("Its not a Face!");  
  52.                         } else {  
  53.                             var count = 1;  
  54.                             var _faceAttributes = JSON.parse(data);  
  55.   
  56.   
  57.                             var _responsetable = "";  
  58.                             var _emotiontable = "";  
  59.                             _responsetable += '<div class="panel panel-default"><div class="panel-heading">Google Face API Response</div>';  
  60.                             _responsetable += "<div class='panel-body'>"  
  61.                             _responsetable += '<table class="table table-bordered"><thead><tr> <th>Description</th> <th>score</th></tr></thead>';  
  62.   
  63.                             for (var i = 0; i < _faceAttributes.responses[0].faceAnnotations.length; i++)  
  64.                             {  
  65.   
  66.                                 _responsetable += '<tr><td>' + "Face" +'</td><td>' +  
  67.                                   count++ +  
  68.                                   '</td></tr>';  
  69.   
  70.                                 _responsetable += '<tr><td>' + "Joy" + '</td><td>' +  
  71.                                 _faceAttributes.responses[i].faceAnnotations[i].joyLikelihood +  
  72.                                  '</td></tr>';  
  73.   
  74.                                 _responsetable += '<tr><td>' + "Anger" + '</td><td>' +  
  75.                               _faceAttributes.responses[i].faceAnnotations[i].angerLikelihood +  
  76.                                '</td></tr>';  
  77.   
  78.                                 _responsetable += '<tr><td>' + "Sorrow" + '</td><td>' +  
  79.                           _faceAttributes.responses[i].faceAnnotations[i].sorrowLikelihood +  
  80.                            '</td></tr>';  
  81.   
  82.                                 _responsetable += '<tr><td>' + "Surprise" + '</td><td>' +  
  83.                          _faceAttributes.responses[i].faceAnnotations[i].surpriseLikelihood +  
  84.                           '</td></tr>';  
  85.   
  86.                                 _responsetable += '<tr><td>' + "detectionConfidence" + '</td><td>' +  
  87.                    _faceAttributes.responses[i].faceAnnotations[i].detectionConfidence +  
  88.                     '</td></tr>';  
  89.                                 _responsetable += '<tr><td>' + "landmarkingConfidence" + '</td><td>' +  
  90.                  _faceAttributes.responses[i].faceAnnotations[i].landmarkingConfidence +  
  91.                   '</td></tr>';  
  92.   
  93.                                  
  94.                                 for (var j = 0; j < _faceAttributes.responses[i].faceAnnotations[i].landmarks.length; j++)  
  95.                                 {  
  96.                                     _responsetable += '<tr><td>' + "type" + '</td><td>' +  
  97.                   _faceAttributes.responses[i].faceAnnotations[i].landmarks[j].type +  
  98.                   '</td></tr>';  
  99.                                     _responsetable += '<tr><td>' + "X position" + '</td><td>' +  
  100.                  _faceAttributes.responses[i].faceAnnotations[i].landmarks[j].position.x +  
  101.                   '</td></tr>';  
  102.                                     _responsetable += '<tr><td>' + "Y position" + '</td><td>' +  
  103.                  _faceAttributes.responses[i].faceAnnotations[i].landmarks[j].position.y +  
  104.                   '</td></tr>';  
  105.                                     _responsetable += '<tr><td>' + "Z position" + '</td><td>' +  
  106.                  _faceAttributes.responses[i].faceAnnotations[i].landmarks[j].position.z+  
  107.                   '</td></tr>';  
  108.   
  109.                                      
  110.                                 }  
  111.   
  112.                             }  
  113.   
  114.                             _responsetable += "</table></div></div>"  
  115.                             $('#ResponseTable').append(_responsetable);  
  116.                         }  
  117.                     }  
  118.                 }  
  119.             });  
  120.         }  
  121.   
  122.     }   

After making changes next we are going to save the application and run it to see a demo of face detection.

Capturing photo and Getting response of facedetection using Google Cloud vision API

ASP.NET
ASP.NET

We got a face detection response with emotion and also got the position of organs on the face.

Debugging view of facedetection API response

ASP.NET

The response which we got can be used for drawing a square box around the face detected as shown in the below image.

ASP.NET

Referenced from

https://cloud.google.com/vision/docs/face-tutorial

For more details on it, you can visit here.

Conclusion

In this article, we have learned how to use Google Cloud Vision API with MVC applications in simple steps. We started with creating an MVC project then getting a Google API Key.  Further, we have created Controller, Action Methods, and View and finally, captured a photo using HTML 5 canvas and sent it to capture method using Ajax POST. The base64string is sent to Google Vision API for analyzing and getting a response.


Similar Articles