Modifying CSV Files in Amazon S3 Bucket Using C#

CSV files are a common data format used in various applications for storing tabular data. In this blog post, we'll explore how to read a CSV file from an Amazon S3 bucket, append new lines to it, and re-upload the modified file back to the bucket using C#.

CSV files are a common data format used in various applications for storing tabular data. In this blog post, we'll explore how to read a CSV file from an Amazon S3 bucket, append new lines to it, and re-upload the modified file back to the bucket using C#. This process can be useful in scenarios where you need to update or append data to existing CSV files stored in S3.

Prerequisites

• AWS account with access to an S3 bucket
• Visual Studio or any C# development environment
• Basic understanding of C# programming language

Step 1. Set Up AWS SDK and Credentials

Begin by installing the AWS SDK for .NET via NuGet package manager. Configure your AWS credentials (access key and secret key) to access your S3 bucket securely.

Step 2. Read CSV File from S3 Bucket

Connect to your S3 bucket using the AWS SDK, and specify the bucket name and file key (path) of the CSV file. Download the CSV file from S3 to a temporary location on your local machine.

Step 3. Append Lines to the CSV File

Read the contents of the CSV file into memory and append new lines or modify existing data as needed. Close the file stream once the modifications are done.

Step 4. Upload Modified CSV File to S3 Bucket

Connect to your S3 bucket using the AWS SDK, and specify the bucket name and file key (path) of the CSV file. Upload the modified CSV file to the specified location in the S3 bucket.

Step 5. Cleanup (Optional)

Delete the temporary file created during the process to clean up resources.

using Amazon.S3;
using Amazon.S3.Model;
using System;
using System.IO;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        // Step 1: Set Up AWS SDK and Credentials
        var credentials = new Amazon.Runtime.BasicAWSCredentials("accessKey", "secretKey");
        var config = new AmazonS3Config
        {
            RegionEndpoint = Amazon.RegionEndpoint.USWest2 // Change region as per your S3 bucket
        };
        var s3Client = new AmazonS3Client(credentials, config);

        // Step 2: Read CSV File from S3 Bucket
        string bucketName = "your-bucket-name";
        string key = "your-file-key.csv";
        string tempFilePath = "temp.csv"; // Temporary file to download CSV
        
        using (var response = await s3Client.GetObjectAsync(bucketName, key))
        using (var fileStream = File.Create(tempFilePath))
        {
            await response.ResponseStream.CopyToAsync(fileStream);
        }

        // Step 3: Append Lines to the CSV File
        using (StreamWriter sw = File.AppendText(tempFilePath))
        {
            sw.WriteLine("New line 1");
            sw.WriteLine("New line 2");
        }

        // Step 4: Upload Modified CSV File to S3 Bucket
        using (var fileStream = File.OpenRead(tempFilePath))
        {
            var uploadRequest = new PutObjectRequest
            {
                BucketName = bucketName,
                Key = key,
                InputStream = fileStream,
                ContentType = "text/csv"
            };
            await s3Client.PutObjectAsync(uploadRequest);
        }

        // Step 5: Cleanup (Optional)
        File.Delete(tempFilePath);

        Console.WriteLine("File appended and uploaded successfully.");
    }
}

Conclusion

Modifying CSV files in an Amazon S3 bucket using C# provides a convenient way to update or append data to existing files stored in the cloud. By following the steps outlined in this blog post, you can efficiently manage CSV files in your S3 bucket, ensuring your data remains up-to-date and accessible to your applications.