Part 2: Resilient Architectures

FCJ Workshop for Resilient Architectures: The RDS Workshop (000005) covers Multi-AZ deployments, Read Replicas, and backup strategies essential for resilient database architectures.

This section covers the reliability pillar of the AWS Well-Architected Framework, ensuring workloads perform their intended function correctly and consistently when expected.

Key Reliability Concepts

Reliability ensures a workload performs its intended function correctly and consistently, with the ability to operate and recover from failures.

Core Metrics to Memorize

MetricDefinitionExample
RTORecovery Time Objective - Maximum acceptable downtime1 hour
RPORecovery Point Objective - Maximum acceptable data loss15 minutes
MTBFMean Time Between Failures - Average uptime between failures720 hours
MTTRMean Time To Recovery - Average time to restore service30 minutes
AvailabilityPercentage of uptime99.99% = 52.56 min downtime/year

Availability Calculations

SLADowntime per YearDowntime per MonthDowntime per Week
99%3.65 days7.2 hours1.68 hours
99.9%8.76 hours43.8 minutes10.1 minutes
99.95%4.38 hours21.9 minutes5.04 minutes
99.99%52.56 minutes4.32 minutes1.01 minutes
99.999%5.26 minutes25.9 seconds6.05 seconds

Content

This comprehensive section covers all aspects of designing resilient architectures for the FCJ Midterm:

  • Multi-AZ RDS Deployments (RPO=0, RTO=60-120s)
  • Disaster Recovery Strategies (Backup & Restore, Pilot Light, Warm Standby, Multi-Site)
  • Auto Scaling and Load Balancing
  • Route 53 Routing Policies (7 types)
  • AWS Backup Strategies

Multi-AZ RDS Deployment

Practice with RDS Workshop: The RDS Workshop (000005) includes hands-on labs for Multi-AZ configuration, failover testing, and Read Replica setup with synchronous vs asynchronous replication concepts.

How Multi-AZ Works

Amazon RDS Multi-AZ provides high availability with automatic failover using synchronous replication to a standby instance in a different Availability Zone.

Architecture:

Primary AZ (us-east-1a)          Standby AZ (us-east-1b)
┌─────────────────────┐          ┌─────────────────────┐
│  Primary RDS        │          │  Standby RDS        │
│  Read/Write         │─────────>│  (Passive)          │
│  10.0.1.100         │Sync Rep  │  10.0.2.100         │
└─────────────────────┘          └─────────────────────┘
         ↑                                ↑
         │                                │
    DNS Endpoint                     (Automatic
 mydb.xyz.rds.amazonaws.com          Failover)

Key Characteristics (FCJ Midterm Critical)

Replication:

  • Synchronous replication - Data written to both instances before commit
  • Zero data loss - RPO = 0 (no data loss during failover)
  • Single DNS endpoint - Applications don’t need to change connection string

Failover:

  • Automatic failover - AWS manages failover, no manual intervention
  • Failover time: 60-120 seconds (typically 1-2 minutes)
  • DNS update - Endpoint automatically points to standby after promotion

Triggers for Failover:

  1. Primary instance failure
  2. Availability Zone outage
  3. Storage failure on primary
  4. Network connectivity loss
  5. Compute unit failure

Important Facts:

  • ✅ Standby is PASSIVE - Cannot serve read traffic
  • ✅ Backups taken from standby (no performance impact on primary)
  • ✅ Synchronous replication (vs async for read replicas)
  • ✅ Different from Read Replicas (which are for read scaling)

Multi-AZ vs Read Replicas

FeatureMulti-AZRead Replicas
PurposeHigh AvailabilityRead Scalability
ReplicationSynchronousAsynchronous
Standby serves trafficNoYes (read-only)
Automatic failoverYesNo (manual promotion)
RPOZeroMinutes (depends on lag)
RTO60-120 secondsManual promotion time
Use caseDR, HARead-heavy workloads
Cost~2x primaryPer replica

You can use both together: Multi-AZ for HA + Read Replicas for scaling

Multi-AZ DB Cluster (New)

AWS introduced Multi-AZ DB Clusters with different characteristics:

Replication: Semi-synchronous (not fully synchronous)

  • Writes committed after at least 1 standby acknowledges
  • “Quorum” mechanism for consistency

Standby Instances: 2 readable standby instances (not passive!)

Failover Time: 25-75 seconds (faster than Multi-AZ instance)

Use Case: Need both HA and read scaling in same deployment

Disaster Recovery Strategies

Four DR Strategies (Memorize This)

Cost →                   Increasing →
RTO/RPO →                Decreasing →
Complexity →             Increasing →

┌────────────┬───────────┬──────────────┬────────────────┐
│ Backup &   │ Pilot     │ Warm         │ Multi-Site     │
│ Restore    │ Light     │ Standby      │ Active-Active  │
└────────────┴───────────┴──────────────┴────────────────┘
   Cheapest                               Most Expensive
   Slowest                                Fastest Recovery

1. Backup and Restore ($ - Cheapest)

Description: Back up data to recovery region, restore when needed

Characteristics:

  • RTO: Hours to 24 hours
  • RPO: Hours (depends on backup frequency)
  • Cost: Lowest (storage only)
  • Complexity: Low

Implementation:

  • Regular snapshots to S3
  • Infrastructure as Code (CloudFormation, Terraform)
  • Automated backup schedules

When to Use: Non-critical workloads, cost-sensitive environments

2. Pilot Light ($$)

Description: Core infrastructure always running in recovery region, scale up during disaster

Characteristics:

  • RTO: 10s of minutes
  • RPO: Minutes
  • Cost: Low-medium
  • Complexity: Medium

What’s Always Running:

  • ✅ Databases (with replication)
  • ✅ Core data storage (S3, EBS snapshots)
  • ❌ Application servers (deployed when needed)
  • ❌ Full-scale resources

Implementation:

  1. Replicate data continuously to recovery region
  2. Maintain minimal core infrastructure
  3. During disaster: Deploy app servers, scale up
  4. Update DNS to point to recovery region

Analogy: Like a pilot light in a furnace - ready to ignite quickly

When to Use: Business-critical apps with moderate RTO/RPO requirements

3. Warm Standby ($$$)

Description: Scaled-down, fully functional version always running

Characteristics:

  • RTO: Minutes
  • RPO: Seconds
  • Cost: Medium-high
  • Complexity: Medium-high

What’s Running:

  • ✅ Full application stack deployed
  • ✅ Reduced capacity (e.g., 1-2 instances instead of 10)
  • ✅ Can handle traffic immediately (at reduced capacity)
  • ✅ Continuous data replication

Implementation:

  1. Deploy full stack at reduced capacity
  2. Continuous data replication
  3. During disaster: Scale up resources (Auto Scaling)
  4. Update Route 53 for traffic routing

Difference from Pilot Light:

  • Pilot Light: Cannot serve requests without additional actions
  • Warm Standby: Can serve requests immediately (at reduced capacity)

When to Use: Mission-critical apps, moderate budget, need fast recovery

4. Multi-Site Active-Active ($$$$)

Description: Full production environment in multiple regions, actively serving traffic

Characteristics:

  • RTO: Near-zero (real-time failover)
  • RPO: Near-zero (or zero with sync replication)
  • Cost: Highest (full duplicate infrastructure)
  • Complexity: High

Implementation:

  • Full production capacity in both regions
  • Route 53 with latency-based or weighted routing
  • DynamoDB Global Tables (multi-active)
  • Aurora Global Database
  • Active-active configuration

Challenges:

  • Data consistency across regions
  • Write conflict resolution
  • Increased operational complexity
  • Higher costs

When to Use: Zero-downtime requirements, global user base, highest SLA

DR Strategy Comparison Table

StrategyRTORPOCostWhen RunningCan Serve Traffic
Backup & Restore24hHours$NothingNo
Pilot Light10minMinutes$$Core onlyNo (needs scaling)
Warm StandbyMinutesSeconds$$$Full stack (scaled down)Yes (limited)
Multi-SiteNear-zeroNear-zero$$$$Full stackYes (full capacity)

Auto Scaling and Load Balancing

Auto Scaling Components

Auto Scaling Group (ASG) automatically adjusts EC2 capacity based on demand.

Key Configuration:

  • Minimum: Instances always running (e.g., 2)
  • Desired: Current target capacity (e.g., 4)
  • Maximum: Upper limit (e.g., 10)

Auto Scaling Policies

Description: Maintain a specific metric at target value

Example:

{
  "TargetValue": 70.0,
  "PredefinedMetricType": "ASGAverageCPUUtilization"
}

Behavior: AWS automatically creates CloudWatch alarms and adjusts capacity

Predefined Metrics:

  • ASGAverageCPUUtilization
  • ASGAverageNetworkIn/Out
  • ALBRequestCountPerTarget

When to Use: Most common use case, simple configuration

2. Step Scaling

Description: Scale in steps based on alarm thresholds

Example:

  • CPU > 80%: Add 2 instances
  • CPU > 90%: Add 4 instances
  • CPU < 30%: Remove 1 instance

When to Use: Need fine-grained control over scaling behavior

3. Simple Scaling

Description: Single adjustment when alarm triggered

Example: CPU > 80% → Add 1 instance

Limitation: Cooldown period before next scaling action

When to Use: Simple, predictable workloads

4. Scheduled Scaling

Description: Scale based on time/date patterns

Example:

  • Scale to 10 instances at 9 AM weekdays
  • Scale to 2 instances at 6 PM weekdays

When to Use: Predictable traffic patterns

CloudWatch Metrics for Auto Scaling

EC2 Metrics (default, 5-minute intervals):

  • CPUUtilization
  • NetworkIn/Out
  • DiskReadOps/WriteOps

Detailed Monitoring (1-minute intervals): Additional cost

Custom Metrics: Application-specific metrics (e.g., queue length, active users)

Load Balancer Types

TypeLayerUse CaseFeatures
ALB7 (HTTP/HTTPS)Web applicationsPath/host routing, WebSocket
NLB4 (TCP/UDP)High performance, static IPMillions of requests/sec
GLB3 (IP)Third-party appliancesTransparent network gateway
CLB4 + 7 (Legacy)Old applicationsNot recommended for new apps

Application Load Balancer (ALB)

Features:

  • Path-based routing: /api/* → API servers, /images/* → Image servers
  • Host-based routing: api.example.com → API, www.example.com → Web
  • HTTP/2 and WebSocket support
  • Target Groups: EC2, Lambda, IP addresses
  • Health Checks: HTTP/HTTPS endpoints

Pricing: $0.0225/hour + $0.008 per LCU-hour

Network Load Balancer (NLB)

Features:

  • Ultra-high performance: Millions of requests per second
  • Static IP per AZ: Elastic IP support
  • Preserve source IP: Client IP visible to targets
  • TLS termination: SSL/TLS offloading
  • TCP/UDP/TLS protocols

When to Use: Real-time streaming, gaming, IoT, extreme performance

Route 53 and DNS

Route 53 Routing Policies (FCJ Midterm Critical)

1. Simple Routing

Description: Single resource, no health checks

Use Case: Single web server

2. Weighted Routing

Description: Distribute traffic by percentage

Example:

  • 80% to Production (weight: 80)
  • 20% to Beta (weight: 20)

Use Case: A/B testing, gradual rollouts

3. Latency-Based Routing

Description: Route to lowest latency region

Example: Users in Asia → ap-southeast-1, Users in US → us-east-1

Use Case: Global applications, multi-region deployments

4. Failover Routing

Description: Primary/secondary for disaster recovery

Implementation:

  • Primary: Active resource (with health check)
  • Secondary: Standby resource (failover)
  • If primary fails health check → Route to secondary

Use Case: Active-passive DR

5. Geolocation Routing

Description: Route based on user’s geographic location

Example: Users in EU → eu-west-1, Users in US → us-east-1

Use Case: Compliance (data residency), localized content

6. Geoproximity Routing

Description: Route based on geographic distance (with bias)

Bias: Adjust routing to prefer certain regions

Use Case: Complex geographic routing requirements

7. Multi-Value Answer Routing

Description: Return multiple IP addresses (with health checks)

Similar to: Simple routing with health checks

Use Case: Simple load distribution

Route 53 Health Checks

Types:

  1. Endpoint Health Checks: Monitor specific endpoint (HTTP/HTTPS/TCP)
  2. Calculated Health Checks: Combine multiple health checks (AND, OR, NOT)
  3. CloudWatch Alarm: Based on CloudWatch alarms

Interval: 30 seconds (standard) or 10 seconds (fast)

Failover: Automatically remove unhealthy endpoints from DNS responses

AWS Backup

AWS Backup provides centralized backup management across AWS services.

Supported Services:

  • EC2 instances and EBS volumes
  • RDS databases
  • DynamoDB tables
  • EFS file systems
  • S3 buckets
  • FSx file systems
  • Storage Gateway volumes

Key Features:

  • Backup Plans: Schedules, retention, lifecycle rules
  • Backup Vaults: Encrypted backup storage
  • Cross-Region Copy: DR capability
  • Cross-Account Backup: Centralized management
  • Point-in-Time Recovery: Restore to specific timestamp

Pricing:

  • Cold storage: $0.01/GB-month (Glacier Deep Archive)
  • Warm storage: $0.05/GB-month (S3)
  • Restore: Variable by service

FCJ Midterm Preparation Tips for Resilient Architectures

  1. Multi-AZ is for HA, Read Replicas are for read scaling
  2. Synchronous replication = Zero RPO (Multi-AZ RDS)
  3. Pilot Light cannot serve traffic, Warm Standby can
  4. Target tracking is the recommended Auto Scaling policy
  5. Latency-based routing for global applications
  6. Failover routing for active-passive DR
  7. RTO = downtime, RPO = data loss
  8. 99.99% availability = 52.56 minutes downtime per year