Send JSON Data From ASP.NET Web API To Splunk

Introduction

This is what we are going to see in this article:

 

 

SPLUNK

Splunk (the product) captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations. It is mainly used for Web Analytics.

Here what I am going to do,

Creating the Web API service which is used to get the feedback from the users/consumer and store it in DB this is usual but now we are going to take a next step by sending the data to splunk enterprise which enables you to search, monitor and analyze any machine data for powerful new insights

The article flow as follows,

1. Installing Splunk Enterprise
2. Creating a REST API using ASP.NET WEB API
3. Splunk in picture - HTTP Event Collector
4. Generating a HTTP Event Collector Token
5. Sending a data from ASP.NET WEB API to Splunk
6. Generating the Dashboard in Splunk 

1.Install the Splunk

Download the Splunk Enterprise v6.4.

First it will compute the disk space and the installation process takes place as show in figure,

   

After successful installation, the First time signing page will open in browser,

 

 

Use Username:admin and password: changeme,

Change your password as required, 

2. Creating a WEB API

Create a Web API application using an installed web template in Visual Studio as in the following figures,

  

 

   

Creating a Model Classes:

In the Solution Explorer, right click the model folder, Add-> Class and name it as Feedback

Feedback.cs

public class Feedback
  {

      [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
      public int FeedbackID { get; set; }
      public string Message { get; set; }
  }

Install the Entity Framework

We are using the Entity framework for data access, Install the Enityframework using nuget packager as shown in following figure,

  

Creating a Context Class:

Add one more class in the Model and name it FeedbackContext which is our Entity framework code first context.

FeedbackContext.cs

public class FeedbackContext: DbContext
 {
     public FeedbackContext() : base("name=DefaultConnection")
     {
     }

     public System.Data.Entity.DbSet<Splunk_WEBAPI.Models.Feedback> Feedbacks { get; set; }
 }

Connection String in Web.comfig file

<connectionStrings>
    <add name="DefaultConnection" connectionString="Data Source= server name;Initial Catalog= db_name;Integrated Security=True"
  </connectionStrings>

Seed the Database:

Now, open the Package manager console and run the following commands,

1. Enable-migrations: This will add the folder called migration with code file named configuration.cs.
2. Add-Migration Initial: Check your Web config file and ensure the SQL connection string, 
   
   Open configuration.cs file under migration folder and add the following code in seed method,

Configuration.cs

protected override void Seed(Splunk_WEBAPI.Models.FeedbackContext context)
       {
            context.Feedbacks.AddOrUpdate(new Models.Feedback { Message = "Test" });
}

3. Update-Database

This will run the seed method,

Check you database which is created

  

Feedback Table

 

From the API service you can notice that the user post the feedback and it is going to store in the DB, so now we need to perform analytics on the data send by users with the help of splunk enterprise.

3. SPLUNK IN PICTURE

HTTP EVENT Collector:

The Event Collector is a new way to send data to Splunk Enterprise. Notably, the EC enables you to send data over HTTP (or HTTPS) directly to Splunk Enterprise from your application.

The basics of Splunk HTTP Event Collector are relatively simple,

1. Turn on the Event Collector in Splunk Enterprise by enabling the HTTP input endpoint. It is not enabled by default.
2. From the Splunk Enterprise instance, generate an EC token.
3. On the machine that will log to Splunk Enterprise, create a POST request, and set its authentication header to include the EC token.
4. POST data in JSON format to the EC token receiver. 

4. Jump into Splunk

 Generating a Token

The token for http event collector can be created in following step,

1. Select Settings->Data Inputs
 

2. click on HttpEventCollector in data input page,

 

  

 

Now, it's time to create a HTTPEventCollector Tokens, click on new token in HTTP event collector page,

 

 

 

Give the Token Name and click next, 

  

Select the main in available items and click next,

  

 

Review the token settings,

Click on global setting on HTTPEventCollector page, to get the HTTP setting and make sure that all the tokens are enabled,

 

  

 

5. Sending a data from ASP.NET WEB API to Splunk  

Now open the ASP.NET WEB API project which we have created and replace the code in Feedbackcontroller.cs by following code and download the trace listener and SLAB libraries from nuget packager as shown in following figures,

FeedbackController.cs

FeedbackContext db = new FeedbackContext();
        [HttpPost]
        [Route("Message")]
        [ResponseType(typeof(Feedback))]
        public async Task<IHttpActionResult> SaveFeedback(Feedback feedback)
        {
            if (!ModelState.IsValid)
            {
                return BadRequest(ModelState);
            }

            db.Feedbacks.Add(feedback);
            await db.SaveChangesAsync();



            ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) =>
            {
                return true;
            };

            var middleware = new HttpEventCollectorResendMiddleware(100);
            var ecSender = new HttpEventCollectorSender(new Uri("https://localhost:8088"),  //http port as set in global setting
                "9201EA75-D7DD-40CD-8D1A-6714F66E4E29",  // token
                null,
                HttpEventCollectorSender.SendMode.Sequential,
                0,
                0,
                0,
                middleware.Plugin
            );
            ecSender.OnError += o => Console.WriteLine(o.Message);
            ecSender.Send(Guid.NewGuid().ToString(), "INFO", null, new { Foo = feedback.Message });
            await ecSender.FlushAsync();

            return Ok("Thanks for your feedback");

        }
    }

Creating a HTML page

Create a new HTML page in the project.

Design:

<!DOCTYPE html>
<html>
<head>
    <title></title>
    <meta charset="utf-8" />
    <link href="Content/bootstrap.min.css" rel="stylesheet" />
    <script src="scripts/jquery-1.9.1.js"></script>
    <script src="scripts/Feedback.js"></script>
</head>
<body>
    <form>

        <div class="container">
            <h3 style="font-style:italic">Feedback</h3>
            <br />
            <div class="row">
                <div class="form-group">
              <label>Message:</label>
                <textarea id="TxtFeddback" ></textarea>
                    </div>
            </div>
            <br />

            <button id="btn_feedback" class="btn btn-default">Submit</button>
            <br />
            <br />
            <div id="responediv" class="row" style="color:green">

            </div>
        </div>
    </form>
</body>
</html>

JavaScript:

$(document).ready(function()
{
    $("#btn_feedback").click(function (e) {
        e.preventDefault();

        $.ajax({
            url: "api/Feedback/Message",
            type: "POST",
            data: { Message: $("#TxtFeddback").val() },
            success: function (data) {
                $("#TxtFeddback").val('')
                $("#responediv").html(data);
            }
        })
    })

})

Demo:

 

  
Data in Table:

 

   

Event created in splunk:

 

Hurray! The data are collected as events in the Splunk.

6.Creating a Dashboard

Click on save as and select dashboard panel in the search, the wizard will be open to create a dashboard,

Click save after giving the dashboard name, then select dashboard from the search page to open the dashboard page as shown in below figure,

 

 

Click on Feedback Dashboard to view the events,

Conclusion

From this article we have learned how to transport the data from ASP.NET Web API to Splunk using HTTPEvent Collector. I hope you have enjoyed this article. Your valuable feedback, questions, or comments about this article are always welcomed.

Read more articles on ASP.NET:

• Getting Started With ASP.NET 5 Web Application
• Sending Emails In ASP.NET MVC From Controller Using WebMail Helper