Amazon OpenSearch Service Setup with Kibana

Amazon OpenSearch Service securely unlocks real-time search, monitoring, and analysis of business and operational data for use cases like application monitoring, log analytics, observability, and website search.

For monitoring real-time operational and business data AWS provides a tool for building such kind of application using Amazon OpenSearch service. We can understand via the below architect diagram.

In the diagram below, we will configure the setup for streaming logs from Amazon CloudWatch Logs to the Log Aggregation System.

Assuming that Amazon Open Search Cluster is up and running now we need to configure a Lambda function logs with Amazon Open Search cluster. So let's follow the below steps.

Go to CloudWatch Log and find the Lambda function logs in Cloud Watch > Log group.

In the left-hand menu, under Logs, click on Log Groups.

1. Select the log group beginning with /aws/lambda/ Lambda function log group
2. Create a subscription filter to send the log data to your OpenSearch domain.
3. Click Actions, in the menu that opens, under Subscription filters, click Create Amazon OpenSearch Service subscription filter:
   

The Create Amazon OpenSearch Service subscription filter form will load.

In the Choose Destination section, select the following.

• Select account: Ensure This account is selected
• Amazon OpenSearch Service cluster: Select the cluster you created previously
  

After selecting the Amazon OpenSearchService cluster, the Lambda function section will appear.

Note. If the Lambda Function section does not appear, please wait for the cluster to finish creating.

In the Lambda IAM Execution Role drop-down select LambdaElasticSearch.

If the Lambda function role is not created, create a new lambda function execution role.

In the Configure log format and filters section enter the following.

• Log Format: Select Amazon Lambda
• Subscription filter name: filter name
  

To start sending the logs to ElasticSearch, at the bottom, click Start streaming.

Now go back to the Lambda function and perform some test events.

Return to the Amazon OpenSearch Search Console for the domain you created and click the link under the Kibana URL.

Note. If you encounter a pop-up select Explore on my own.

Now Add data to Kibana.

The log data is stored in OpenSearch, but you need to tell Kibana which index to use for discovering the data.

In the Create an index pattern wizard, enter the following value and click Next step.

Index pattern: xyz-*

The pattern will match the daily CloudWatch Logs (XYZ) indices that are created in Amazon OpenSearch.

In the second step, enter the following value and click Create index pattern.

Index pattern: Select @timestamp

The Time filter field name allows Kibana to determine which attribute represents the timestamp of each event. The confirmation page displays all of the fields in the log data index.

Now that the index settings for Kibana are configured, you can begin using the Log Aggregation system!

Click Discover in the sidebar menu on the left of the page.

Explore the Discover interface.

You see some events and a graph. These are your aggregated log events! The system is online! Notice the search bar up top. It is initially empty so all log events will show up. But what if you only want to see the PUT events for objects containing 12345?

Enter PUT 12345 in the search bar and press enter.

The matching terms in the event show up highlighted, and the bar graph updates to show only the count of PUT 12345 events that you made by clicking Test in the Lambda interface.

Click on the timestamp range in the upper-right corner to display the time filter.

Just as with CloudWatch Logs, you can filter the logs by time. However, in Kibana you can also drag on the bar chart to select a time range visually.

Summary

Amazon Open search Service is a fully managed service and easy to set up and powerful tool for real-time monitoring and analysis. This feature-rich experience supports a variety of use cases, including observability, security analytics, and log workloads.