Optimizing Performance in Azure Cosmos DB: Best Practices

Introduction

When we are working with a database, optimization is crucial and a key factor in terms of application performance and maximizing efficiency. Similarly, in Azure Cosmos DB, optimization is crucial for maximizing efficiency, minimizing costs, and ensuring that your application scales effectively. Here are some best practices with coding examples to optimize performance in Azure Cosmos DB.

1. Selection of Right Partition Key
2. Properly Use Indexing
3. Optimize Queries
4. Consistency Levels Tuning
5. Use Provisioned Throughput (RU/s) and auto-scale Wisely
6. Leverage Change Feed for Efficient Real-Time Processing
7. Utilization of Time-to-Live (TTL) for Automatic Data Expiration
8. Avoid Cross-Partition Queries

1. Selection of Right Partition Key

Choosing an appropriate partition key is vital for distributed databases like Cosmos DB. A good partition key ensures that data is evenly distributed across partitions, reducing hot spots and improving performance. The selection of a partition key is simple but very important at design time in Azure Cosmos DB. Once we select the partition key, it isn't possible to change it in place.

Best Practice

1. Select a partition key with high cardinality (many unique values).
2. Ensure it distributes reads and writes evenly.
3. Keep related data together to minimize cross-partition queries.

Example. Creating a Container with an Optimal Partition Key.

var database = await cosmosClient.CreateDatabaseIfNotExistsAsync("YourDatabase");
var containerProperties = new ContainerProperties
{
    Id = "myContainer",
    PartitionKeyPath = "/customerId" // Partition key selected to ensure balanced distribution
};
// Create the container with 400 RU/s provisioned throughput
var container = await database.CreateContainerIfNotExistsAsync(
    containerProperties, 
    throughput: 400
);

2. Properly Use Indexing

In Azure Cosmos DB, indexes are applied to all properties by default, which can be beneficial but may result in increased storage and RU/s costs. To enhance query performance and minimize expenses, consider customizing the indexing policy. Cosmos DB supports three types of indexes: Range Indexes, Spatial Indexes, and Composite Indexes. Use the proper type wisely.

Best Practice

1. Exclude unnecessary fields from indexing.
2. Use composite indexes for multi-field queries.

Example. Custom Indexing Policy.

{
    "indexingPolicy": {
        "automatic": true,
        "indexingMode": "consistent", // Can use 'none' or 'lazy' to reduce write costs
        "includedPaths": [
            {
                "path": "/orderDate/?", // Only index specific fields like orderDate
                "indexes": [
                    {
                        "kind": "Range",
                        "dataType": "Number"
                    }
                ]
            }
        ],
        "excludedPaths": [
            {
                "path": "/largeDataField/*" // Exclude large fields not used in queries
            }
        ]
    }
}

Example. Adding a Composite Index for Optimized Querying.

{
    "indexingPolicy": {
        "compositeIndexes": [
            [
                { "path": "/lastName", "order": "ascending" },
                { "path": "/firstName", "order": "ascending" }
            ]
        ]
    }
}

Here is the link to read about Indexing types: https://learn.microsoft.com/en-us/azure/cosmos-db/index-overview

3. Optimize Queries

Efficient querying is crucial for minimizing request units (RU/s) and improving performance in Azure Cosmos DB. The RU/s cost depends on the query's complexity and size.

Utilizing bulk executors can further reduce costs by decreasing the RUs consumed per operation. This optimization helps manage RU usage effectively and lowers your overall Cosmos DB expenses.

Best Practice

• Use SELECT * queries in limited amounts and retrieve only necessary properties.
• Avoid cross-partition queries by providing the partition key in your query.
• Use filters on indexed fields to reduce query costs.

Example. Fetch Customer Record.

var query = new QueryDefinition(
    "SELECT c.firstName, c.lastName FROM Customers c WHERE c.customerId = @customerId"
).WithParameter("@customerId", "12345");
var iterator = container.GetItemQueryIterator<Customer>(
    query,
    requestOptions: new QueryRequestOptions
    {
        // Provide partition key to avoid cross-partition query
        PartitionKey = new PartitionKey("12345")
    }
);
while (iterator.HasMoreResults)
{
    var response = await iterator.ReadNextAsync();
    foreach (var customer in response)
    {
        Console.WriteLine($"{customer.firstName} {customer.lastName}");
    }
}

4. Consistency Levels Tuning

The consistency levels define specific operational modes designed to meet speed-related guarantees. There are five consistency levels (Strong, Bounded Staleness, Session, Consistent Prefix, and Eventual) available in Cosmos DB. Each consistency level impacts latency, availability, and throughput.

Best Practice

• Use Session consistency for most scenarios to balance performance and data consistency.
• Strong consistency guarantees data consistency but increases RU/s and latency.

Example. Setting Consistency Level.

var cosmosClient = new CosmosClient(
    "<account-endpoint>",
    "<account-key>",
    new CosmosClientOptions
    {
        	// Set consistency to "Session" for balanced performance
            ConsistencyLevel = ConsistencyLevel.Session      
    });

To read more in detail on the Consistency level, follow the link below: https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels

5. Use Provisioned Throughput (RU/s) and auto-scale Wisely

Provisioning throughput is a key factor in achieving both cost efficiency and optimal performance in Azure Cosmos DB. The service enables you to configure throughput in two ways.

1. Fixed RU/s: A predefined, constant level of Request Units per second (RU/s), suitable for workloads with consistent performance demands.
2. Auto-Scale: A dynamic option that automatically adjusts the throughput based on workload fluctuations, providing scalability while avoiding overprovisioning during periods of low activity.

Choosing the appropriate throughput model helps balance performance needs with cost management effectively.

Best Practice

1. For predictable workloads, provision throughput manually.
2. Use auto-scale for unpredictable or bursty workloads.

Example. Provisioning Throughput with Auto-scale.

// Autoscale up to 4000 RU/s
var throughputProperties = ThroughputProperties.CreateAutoscaleThroughput(maxThroughput: 4000);
var container = await database.CreateContainerIfNotExistsAsync(
    new ContainerProperties
    {
        Id = "autoscaleContainer",
        PartitionKeyPath = "/userId"
    },
    throughputProperties
);

Example. Manually Setting Fixed RU/s for Stable Workloads.

var container = await database.CreateContainerIfNotExistsAsync(new ContainerProperties
{
    Id = "manualThroughputContainer",
    PartitionKeyPath = "/departmentId"
}, throughput: 1000);  // Fixed 1000 RU/s

6. Leverage Change Feed for Efficient Real-Time Processing

The change feed allows for real-time, event-driven processing by automatically capturing changes in the database, eliminating the need for polling. This reduces query overhead and enhances efficiency.

Best Practice

Use change feed for scenarios where real-time data changes need to be processed (e.g., real-time analytics, notifications, alerts).

Example. Reading from the Change Feed.

var iterator = container.GetChangeFeedIterator<YourDataModel>(
    ChangeFeedStartFrom.Beginning(),
    ChangeFeedMode.Incremental
);
while (iterator.HasMoreResults)
{
    var changes = await iterator.ReadNextAsync();
    foreach (var change in changes)
    {
        // Process the change (e.g., trigger event, update cache)
        Console.WriteLine($"Detected change: {change.Id}");
    }
}

7. Utilization of Time-to-Live (TTL) for Automatic Data Expiration

If you have data that is only relevant for a limited time, such as logs or session data, enabling Time-to-Live (TTL) in Azure Cosmos DB can help manage storage costs. TTL automatically deletes expired data after the specified retention period, eliminating the need for manual data cleanup. This approach not only reduces the amount of stored data but also ensures that your database is optimized for cost-efficiency by removing obsolete or unnecessary information.

Best Practice

Set TTL for containers where data should expire automatically to reduce storage costs.

Example. Setting Time-to-Live (TTL) for Expiring Data.

{
    "id": "sessionDataContainer",
    "partitionKey": { "paths": ["/sessionId"] },
    "defaultTtl": 3600  // 1 hour (3600 seconds)
}

In Cosmos DB, the maximum Time-to-Live (TTL) value that can be set is 365 days (1 year). This means that data can be automatically deleted after it expires within a year of creation or last modification, depending on how you configure TTL.

8. Avoid Cross-Partition Queries

Cross-partition queries can significantly increase RU/s and latency. To avoid this:

Best Practice

• Always include the partition key in your queries.
• Design your partition strategy to minimize cross-partition access.

Example. Querying with Partition Key to Avoid Cross-Partition Query.

var query = new QueryDefinition("SELECT * FROM Orders o WHERE o.customerId = @customerId")
    .WithParameter("@customerId", "12345");
var resultSetIterator = container.GetItemQueryIterator<Order>(
    query, 
    requestOptions: new QueryRequestOptions
    {
        PartitionKey = new PartitionKey("12345")
    });
while (resultSetIterator.HasMoreResults)
{
    var response = await resultSetIterator.ReadNextAsync();  
    foreach (var order in response)
    {
        Console.WriteLine($"Order ID: {order.Id}");
    }
}

Conclusion

These best practices and tips are very effective during development. By implementing an effective partitioning strategy, customizing indexing policies, optimizing queries, adjusting consistency levels, and selecting the appropriate throughput provisioning models, you can greatly improve the performance and efficiency of your Azure Cosmos DB deployment. These optimizations not only enhance scalability but also help in managing costs while providing a high-performance database experience.