Amazon Sqs

Decoupling Applications with Amazon SQS

Point-to-Point Messaging Model

Core Concepts

• Asynchronous messaging model for decoupled architecture implementation
• Producer: Sending application that generates messages and puts them in queue
• Consumer: Receiving application that gets messages from queue and processes them
• Message Queue: Temporary repository for messages waiting to be processed

Message Flow Process

1. Producer puts message in queue
2. Queue stores message until retrieved
3. Consumer gets message from queue using pull mechanism (periodic polling)

Point-to-point messaging enables one producer to send messages to only one specific consumer, with the producer knowing the receiving application.

Amazon Simple Queue Service (Amazon SQS)

Service Overview

Amazon SQS is a fully managed message queuing service that:

• Integrates and decouples distributed software systems and application components
• Provides highly available, secure, and durable message-queueing capabilities
• Processes billions of messages per day at massive scale
• Stores queues and messages across multiple Availability Zones for redundancy

Key Benefits

Fully Managed

• No messaging software management required
• No infrastructure maintenance needed
• Automatic provisioning of compute, storage, and networking

Reliability

• Extremely high message durability
• Messages stored on multiple servers
• Large volume delivery without message loss

Security

• Controlled access to send and receive messages
• Server-side encryption (SSE) with AWS KMS
• Messages decrypted only for authorized consumers

Scalability

• Billions of messages processed daily
• Elastic scaling based on usage
• No capacity planning or preprovisioning needed

Amazon SQS Components

Messages

• Maximum size: 256 KB (XML, JSON, unformatted text)
• Extended payloads: Up to 2 GB using Amazon SQS Extended Client Library for Java
• Retention: Default 4 days, configurable up to 14 days
• Messages remain until explicitly deleted or retention period expires

Queues

Two types available with different characteristics:

Standard Queues:

• At-least-once delivery: Message delivered at least once, occasionally duplicated
• Best-effort ordering: Messages may arrive out of order
• Nearly unlimited throughput: Supports unlimited API calls per second

FIFO Queues:

• First-in-first-out delivery: Messages delivered in exact order sent
• Exactly once processing: Each message processed exactly once
• High throughput: Up to 300 API calls per second (3,000 with batching)

Queue Configuration Parameters

Message Retention Period: Time Amazon SQS retains messages in queue

Visibility Timeout:

• Length of time message is invisible to other consumers after being received
• Default 30 seconds, maximum 12 hours
• Should match maximum processing time for your application
• Prevents duplicate processing during message handling

Receive Message Wait Time:

• Controls polling behavior (short vs long polling)
• Short Polling (default = 0): Queries subset of servers, immediate response
• Long Polling (1-20 seconds): Queries all servers, waits for messages
• Long polling reduces costs by decreasing empty responses

Dead Letter Queues (DLQ)

• Associated with source queue to receive unprocessable messages
• Messages moved to DLQ after exceeding maximum processing attempts
• Standard queue DLQ must be standard, FIFO queue DLQ must be FIFO
• Enables reprocessing of failed messages at later time

SQS Message Lifecycle

Processing Flow

1. Producer sends message to queue, distributed across servers redundantly
2. Consumer retrieves message, visibility timeout period starts (e.g., 40 seconds)
3. Message processed while remaining in queue, invisible to other consumers
4. Consumer deletes message from queue during visibility timeout
5. If timeout expires without deletion, message becomes visible again

Caution

Critical: Amazon SQS doesn’t automatically delete messages. Consumer must explicitly delete after processing to prevent reprocessing.

Use Cases and Applications

Primary Use Cases

• Work Queues: Decouple components processing work at different rates

  • Auto Scaling groups of consumers process messages at their own pace
  • Example: Car navigation system collecting real-time sensor data for map updates

• Buffering and Batch Operations:

  • Smooth out traffic spikes without losing messages
  • Process messages in bulk at optimal times
  • Example: Stock trading capturing transactions during day, processing at night

• Request Offloading:

  • Move slow operations off interactive request paths
  • Immediate user response with background processing
  • Example: Banking bill payment with immediate confirmation

• Auto Scaling Triggers:

  • Use queue metrics to determine application load
  • CloudWatch alarms trigger EC2 Auto Scaling based on message count
  • Example: Scale when NumberOfMessagesSent > 10

When NOT to Use Queues

• Selective Message Retrieval: Need to receive specific messages matching attributes
• Large Message Management: Messages over hundreds of KB better suited for dedicated storage (S3) with reference passing

Decoupling Example: Order Processing

Architecture: Order capture (producer) and order fulfillment (consumer) applications

• Producer: Three load-balanced instances capturing customer orders
• Queue: Customer order queue storing order messages
• Consumer: Two instances polling queue and processing orders
• DLQ: Handles messages that cannot be processed

Benefits:

• Resilient to traffic spikes through queue buffering
• Independent scaling of capture and fulfillment functions
• Cost-effective processing at optimal pace
• Retry capability and dead letter queue for failed messages

Amazon SQS enables building fault-tolerant, scalable applications through reliable point-to-point messaging with configurable durability and processing guarantees.

Messaging Service