Upload/Download/Delete Files From AWS S3 Using .NET Core API

Introduction

In this article, we are going to explore about how to upload, download and delete the file(s) from AWS S3 and check if the file(s) exists or not in AWS S3 using .NET Core Web API.

AWS S3

The Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Please refer to the link for more details about the AWS S3 service.

Topics Covered

This article covers the following topics, 

• Create a sample ASP.NET Core Web API to,
  • Upload file to AWS S3
  • Download file from AWS S3
  • Delete files from AWS S3
  • Check If the file is available or not in the AWS S3
• Test the Web API

Prerequisites

• Download and install Visual Studio 2019.
• Download and install the AWS Toolkit for Visual Studio 2019.
• The user should have an active AWS Account.
• The user should have programmatic access keys. Please refer to the link for getting the access keys from AWS account.

Create a sample ASP.NET Core Web API

Follow the below steps to create an ASP.NET Core Web API using Visual Studio 2019.

Step 1

Open Visual Studio 2019, click Create a new project.

Step 2

Select ASP.NET Core Web Application project template and click Next.

Step 3

Enter the project name as DotNetCore_AWS_Demo and Click Next.

Step 4

Select .NET Core 3.1 and Empty project and Click Create.

Step 5

Install AWSSDK.S3 package using the Nuget Package Manager.

Step 6

Create Controller with name AwsS3Controller.

Step 7

Create IAppConfiguration interface and class name as AppConfiguration.

Step 8

Update the class with following code. This class is used to configure the S3 bucket and AWS access key details.

IAppConfiguration.cs

public interface IAppConfiguration
{
	string AwsAccessKey { get; set; }
	string AwsSecretAccessKey { get; set; }
	string AwsSessionToken { get; set; }
	string BucketName { get; set; }
	string Region { get; set; }
}

AppConfiguration .cs

public class AppConfiguration : IAppConfiguration
{
	// Keep the following details in appsettings.config file or DB or Enivironment variable 
	// Get those values from it and assign to the below varibales. Based on the approach , modify the below code.
	public AppConfiguration()
	{
		BucketName = "";
		Region = "";
		AwsAccessKey = ""; 
		AwsSecretAccessKey = "";
		AwsSessionToken = "";
	}

	public string BucketName { get; set; }
	public string Region { get; set; }
	public string AwsAccessKey { get; set; }
	public string AwsSecretAccessKey { get; set; }
	public string AwsSessionToken { get; set; }
}

Step 9

Register AppConfiguration class in ConfigureServices() method in the Startup class.

services.AddSingleton<IAppConfiguration, AppConfiguration>()

Step 10

In the AwsS3Controller controller, add the constructor for injecting the IAppConfiguration dependency in order to fetch the configuration values.

[ApiController]
[Route("documents")]
public class AwsS3Controller : ControllerBase
{
	private readonly IAppConfiguration _appConfiguration;
	private readonly IAws3Services _aws3Services;

	public AwsS3Controller(IAppConfiguration appConfiguration)
	{
		_appConfiguration = appConfiguration;
	}
}

Step 11

Create IAws3Services interface and Aws3Services class and in the constructor, create object for AmazonS3Client to access the AWS S3 bucket.

IAws3Services.cs

public interface IAws3Services
{
	Task<byte[]> DownloadFileAsync(string file);

	Task<bool> UploadFileAsync(IFormFile file);

	Task<bool> DeleteFileAsync(string fileName, string versionId = "");
}

Aws3Services.cs 

public class Aws3Services : IAws3Services
{
	private readonly string _bucketName;
	private readonly IAmazonS3 _awsS3Client;

	public Aws3Services(string awsAccessKeyId, string awsSecretAccessKey, string awsSessionToken, string region, string bucketName)
	{
		_bucketName = bucketName;
		_awsS3Client = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, awsSessionToken, RegionEndpoint.GetBySystemName(region));
	}
}

Download File from S3

Create GetDocumentFromS3 action (GET method) in the AwsS3Controller controller and pass the filename as a parameter so that the filename will check and retrieve the actual file from the S3 bucket. The different results of the GetDocumentFromS3 action method are,

1. If the filename parameter is null or empty, then Bad Request(400) error response will be returned to the user.
2. If the file is available on the S3 then the file will be downloaded from S3 (status code is 200).
3. If the file is not available in the S3 then Not Found (404) error response will be returned.

AwsS3Controller.cs

[HttpGet("{documentName}")]
public IActionResult GetDocumentFromS3(string documentName)
{
	try
	{
		if (string.IsNullOrEmpty(documentName))
			return ReturnMessage("The 'documentName' parameter is required", (int)HttpStatusCode.BadRequest);

		_aws3Services = new Aws3Services(appConfiguration.AwsAccessKey, appConfiguration.AwsSecretAccessKey, appConfiguration.AwsSessionToken, appConfiguration.Region, appConfiguration.BucketName);

		var document = _aws3Services.DownloadFileAsync(documentName).Result;

		return File(document, "application/octet-stream", documentName);
	}
	catch (Exception ex)
	{
		return ValidateException(ex);
	}
}

Create a new function called DownloadFileAsync in the Aws3Services class. We are going to use AmazonS3Client's GetObjectAsync method to download the required file. GetObjectAsync method is used to retrieve objects from Amazon S3. Please refer to the link for more details about GetObjectAsync.  

Create GetObjectRequest object, which has details of S3 bucket and file name. Pass the GetObjectRequest to GetObjectAsync method to download the file. If the file is downloaded successfully, then return the file to the user else return not found exception to the user.

Aws3Services .cs

public async Task<byte[]> DownloadFileAsync(string file)
{
	MemoryStream ms = null;

	try
	{
		GetObjectRequest getObjectRequest = new GetObjectRequest
		{
			BucketName = _bucketName,
			Key = file
		};
			
		using (var response = await _awsS3Client.GetObjectAsync(getObjectRequest))
		{
			if (response.HttpStatusCode == HttpStatusCode.OK)
			{
				using (ms = new MemoryStream())
				{
					await response.ResponseStream.CopyToAsync(ms);
				}
			}
		}

		if (ms is null || ms.ToArray().Length < 1)
			throw new FileNotFoundException(string.Format("The document '{0}' is not found", file));

		return ms.ToArray();
	}
	catch (Exception)
	{
		throw;
	}
}

Upload File to S3

Create UploadDocumentToS3 action (POST method) in the AwsS3Controller controller and pass the actual file (type is IFormFile) as parameter. The different results of the UploadDocumentToS3 action method are,

1. If the file is null or empty, then Bad Request(400) error response will be returned to the user.
2. If the file is valid then uploaded the file to S3 (status code is 200).

AwsS3Controller.cs

[HttpPost]
public IActionResult UploadDocumentToS3(IFormFile file)
{
	try
	{
		if (file is null || file.Length <= 0)
			return ReturnMessage("file is required to upload", (int)HttpStatusCode.BadRequest);

		_aws3Services = new Aws3Services(appConfiguration.AwsAccessKey, appConfiguration.AwsSecretAccessKey, appConfiguration.AwsSessionToken, appConfiguration.Region, appConfiguration.BucketName);

		var result = _aws3Services.UploadFileAsync(file);

		return ReturnMessage(string.Empty, (int)HttpStatusCode.Created);
	}
	catch (Exception ex)
	{
		return ReturnMessage(ex.Message, (int)HttpStatusCode.InternalServerError);
	}
}

Create a new function UploadFileAsync in the Aws3Services class. We are going to use TransferUtility for uploading the file to S3. 

TransferUtility provides a simple API for uploading content to or downloading content from Amazon S3. Please refer to the link for more details about TransferUtility.  

Create TransferUtilityUploadRequest object, which has details of the bucket and the file to be uploaded. Pass the TransferUtilityUploadRequest object to TransferUtility's UploadAsync method to upload the file. 

Aws3Services .cs

public async Task<bool> UploadFileAsync(IFormFile file)
{
	try
	{
		using (var newMemoryStream = new MemoryStream())
		{
			file.CopyTo(newMemoryStream);

			var uploadRequest = new TransferUtilityUploadRequest
			{
				InputStream = newMemoryStream,
				Key = file.FileName,
				BucketName = _bucketName,
				ContentType = file.ContentType
			};

			var fileTransferUtility = new TransferUtility(_awsS3Client);

			await fileTransferUtility.UploadAsync(uploadRequest);

			return true;
		}
	}
	catch (Exception)
	{
		throw;
	}
}

Delete Files in the S3

Create DeletetDocumentFromS3 action (DELETE method) in the AwsS3Controller controller and pass the filename as parameter. The different results of the UploadDocumentToS3 action method are,

1. If the filename parameter is null or empty, then Bad Request(400) error response will be returned to the user.
2. If the file is available in the S3 then deleted the file and return the success response (status code is 200).
3. If the file is not available in the S3 then Not Found (404) error response will be returned.

AwsS3Controlle.cs

[HttpDelete("{documentName}")]
public IActionResult DeletetDocumentFromS3(string documentName)
{
	try
	{
		if (string.IsNullOrEmpty(documentName))
			return ReturnMessage("The 'documentName' parameter is required", (int)HttpStatusCode.BadRequest);

		_aws3Services = new Aws3Services(appConfiguration.AwsAccessKey, appConfiguration.AwsSecretAccessKey, appConfiguration.AwsSessionToken, appConfiguration.Region, appConfiguration.BucketName);

		_aws3Services.DeleteFileAsync(documentName);

		return ReturnMessage(string.Format("The document '{0}' is deleted successfully", documentName));
	}
	catch (Exception ex)
	{
		return ValidateException(ex);
	}
}

Create a new function called DeleteFileAsync in the Aws3Services class. File name and the version id (If S3 bucket is enabled versioning, then pass version id else pass empty) are parameter for this function. 

We are going to use AmazonS3Client's DeleteObjectAsync method to delete the file in the S3. Please refer to the link for more details about DeleteObjectAsync.  

Create DeleteObjectRequest object, which has details of S3 bucket and file name. Pass the DeleteObjectRequest to DeleteObjectAsync method to perform a delete operation. 

Note: Before going to delete, we need to check if the file is available or not. If not, then return "Not Found" exception otherwise delete the file. To check the file availability is an optional one. Depending on the use case, we can add this functionality to it.

Aws3Services.cs

[HttpDelete("{documentName}")]
public IActionResult DeletetDocumentFromS3(string documentName)
{
	try
	{
		if (string.IsNullOrEmpty(documentName))
			return ReturnMessage("The 'documentName' parameter is required", (int)HttpStatusCode.BadRequest);

		_aws3Services = new Aws3Services(appConfiguration.AwsAccessKey, appConfiguration.AwsSecretAccessKey, appConfiguration.AwsSessionToken, appConfiguration.Region, appConfiguration.BucketName);

		_aws3Services.DeleteFileAsync(documentName);

		return ReturnMessage(string.Format("The document '{0}' is deleted successfully", documentName));
	}
	catch (Exception ex)
	{
		return ValidateException(ex);
	}
}

private async Task DeleteFile(string fileName, string versionId)
{
	DeleteObjectRequest request = new DeleteObjectRequest
	{
		BucketName = _bucketName,
		Key = fileName
	};

	if (!string.IsNullOrEmpty(versionId))
		request.VersionId = versionId;

	await _awsS3Client.DeleteObjectAsync(request);
}

public bool IsFileExists(string fileName, string versionId)
{
	try
	{
		GetObjectMetadataRequest request = new GetObjectMetadataRequest()
		{
			BucketName = _bucketName,
			Key = fileName,
			VersionId = !string.IsNullOrEmpty(versionId) ? versionId : null
		};

		var response = _awsS3Client.GetObjectMetadataAsync(request).Result;

		return true;
	}
	catch (Exception ex)
	{
		if (ex.InnerException != null && ex.InnerException is AmazonS3Exception awsEx)
		{
			if (string.Equals(awsEx.ErrorCode, "NoSuchBucket"))
				return false;

			else if (string.Equals(awsEx.ErrorCode, "NotFound"))
				return false;
		}

		throw;
	}
}

Test the Web API

Now let's run the application and test the above functionality through Swagger. Swagger (OpenAPI) is a language-agnostic specification for describing REST APIs. Swagger UI offers a web-based UI that provides information about the service, using the generated OpenAPI specification. 

To enable the swagger we need to follow the below steps.

Step 1 

Install Swashbuckle.AspNetCore package using the NuGet Package Manager

Step 2

Add the Swagger generator to the services collection in the Startup.ConfigureServices method.

public void ConfigureServices(IServiceCollection services)
{
	services
		.AddSingleton<IAppConfiguration, AppConfiguration>()
		.AddSwaggerGen()
		.AddControllers();
}

In the Startup.Configure method, enable the middleware for serving the generated JSON document and the Swagger UI.

app.UseSwagger();

app.UseSwaggerUI(c =>
{
	c.SwaggerEndpoint("/swagger/v1/swagger.json", "DotNet_AWS");
});

Step 3

Hit F5 to run the API locally and To launch the Swagger just hit the http://localhost:<port_number>/swagger/index.html URL in the browser. You will get the below Swagger UI.

Just expand the required operation and click "Try it out" button.

Enter the input values and click the "Execute" button. 

Download the File (GET Method)

Upload the File (Post Method)

Delete the File (Delete Method)

Summary

In this article, you learned about how to implement the following functionality using .NET Core Web API.

1. Download the file from S3
2. Upload the file to S3
3. Delete the file in the S3