Post
Ask Question

Create On Demand Job in AWS

Architecture Overview

  1. AWS Services
  2. AWS Lambda
  3. AWS API Gateway
  4. Amazon SQS
  5. Amazon S3

Architecture Diagram

+-------------------+
|    API Gateway    |
+-------------------+
        |
        v
+-------------------+
|  Lambda Function  |
+-------------------+
        |
        v
+-------------------+
|     Amazon SQS    |
+-------------------+
        |
        v
+-------------------+
|     Amazon S3     |
+-------------------+

Components

  1. API Gateway
    • The REST API endpoint is used to trigger the on-demand job.
    • Integrated with Lambda function.
  2. Lambda Function
    • Runs the on-demand job.
    • Processes messages from the SQS queue.
    • Stores output in an S3 bucket.
  3. Amazon SQS
    • Message queue for job requests.
    • The lambda function polls the SQS queue for new messages.
  4. Amazon S3
    • Stores output files from the Lambda function

Steps to integrate with lambda function

Step 1. Create API Gateway.

  1. Log in to the AWS Management Console
  2. Navigate to API Gateway
  3. Create REST API
  4. Create a POST method for triggering on-demand job
  5. Integrate with the Lambda function
    API Gateway

Step 2. Create a Lambda Function.

  1. Navigate to Lambda
  2. Create a new function
  3. Choose Node.js or Python runtime
  4. Set handler function
  5. Configure environment variables
  6. Set timeout and memory limits
    Lambda Function

Step 3. Create SQS Queue.

  1. Navigate to SQS
  2. Create a new queue
  3. Configure queue settings
    SQS Queue

Step 4. Configure Lambda Function to Poll SQS Queue.

  1. Navigate to Lambda
  2. Configure event source mapping
  3. Choose the SQS queue
    Poll SQS

Step 5. Create S3 Bucket.

  1. Navigate to S3
  2. Create a new bucket
  3. Configure bucket settings
    S3 Bucket

Step 6. Test On-Demand Job.

  1. Send a POST request to API Gateway
  2. Verify Lambda function execution
  3. Check output files in the S3 bucket
    Demand Job

Lambda Function (Node.js)

const AWS = require('aws-sdk');
const sqs = new AWS.SQS({ region: 'us-west-2' });
const s3 = new AWS.S3({ region: 'us-west-2' });

exports.handler = async (event) => {
    const message = JSON.parse(event.body);
    const jobId = message.jobId;
    const output = await processJob(jobId);

    await s3.putObject({
        Bucket: 'my-bucket',
        Key: `${jobId}.txt`,
        Body: output,
    }).promise();
};

const processJob = async (jobId) => {
    // Job processing logic here
    return `Job ${jobId} completed`;
};
{
  "jobId": "$input.params('jobId')"
}


Similar Articles