Automating File Uploads to Azure Storage with Azure Pipelines

Introduction

Automation plays a pivotal role in modern software development workflows, streamlining processes and reducing manual intervention. Azure Pipelines, a part of Azure DevOps, offers a powerful platform for building, testing, and deploying applications while automating various tasks. In this article, we will explore how to automate the process of uploading files from a Git repository to Azure Storage using Azure Pipelines. By setting up an automated pipeline, you can ensure consistent and reliable file uploads to Azure Storage whenever changes are made to your repository.

Prerequisites

1. An Azure account with Azure DevOps access
2. A Git repository containing the files you want to upload
3. An existing Azure Storage account

Step 1. Set Up an Azure DevOps Project

Log in to Azure DevOps (dev.azure.com) and create a new project or use an existing one.

Step 2. Create an Azure Pipeline

1. Inside your Azure DevOps project, navigate to Pipelines > Pipelines.
2. Click on the "+ New" button to create a new pipeline.
3. Choose your Git repository as the source for the pipeline.

Step 3. Define Pipeline Stages

1. After selecting the repository, you'll be prompted to configure the pipeline YAML file.
2. Define the stages for your pipeline. For this example, let's create a simple pipeline with one stage.

stages:
- stage: UploadFiles
  jobs:
  - job: UploadJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - script: |
        az storage blob upload-batch --source $(Build.SourcesDirectory) --destination <container_name> --destination-path <destination_folder_path> --account-name <storage_account_name> --sas-token "<connection_string>"
      displayName: 'Upload Files to Azure Storage'

Replace placeholders in the script with actual values for <container_name>, <destination_folder_path>, <storage_account_name>, and <connection_string>. This script uses Azure CLI to upload files.

Step 4. Configure Triggers

1. To trigger the pipeline whenever changes are pushed to the repository, set up CI/CD triggers.
2. In the pipeline YAML, add a trigger section to specify the branches that should trigger the pipeline.

trigger: - main - feature/*

In this example, the pipeline will be triggered for changes in the main branch and any feature branches starting with "feature/".

Step 5. Save and Run Pipeline

1. Save the pipeline configuration.
2. Manually run the pipeline or wait for changes to trigger it based on the configured triggers.

Step 6. Monitor Pipeline Execution

Monitor the pipeline's execution in Azure DevOps to ensure the file upload process completes successfully.

Conclusion

By setting up an Azure Pipeline to automate file uploads from a Git repository to Azure Storage, you can ensure that your files are consistently and reliably uploaded whenever changes are made. This streamlined approach enhances collaboration, reduces manual effort, and helps maintain a reliable data storage process. Azure Pipelines empowers developers to focus on building code while the automation takes care of deployment tasks like file uploads.