Migrating Data

Migrating Data into AWS Databases

AWS Database Migration Service (AWS DMS) provides migration and replication solutions to help move existing database and analytics workloads to and within AWS.

AWS DMS Overview

AWS DMS is a web service that you can use to migrate data from a source data store to a target data store. These two data stores are called endpoints.

Key Capabilities

• Managed Migration Service: Helps move existing database and analytics workloads to and within AWS
• Broad Database Support: Supports most widely used commercial and open source databases
• Flexible Replication: Replicates data on demand or on schedule to replicate changes from source
• Minimal Downtime: Source database remains fully operational during migration

Migration Types

Homogeneous Migration

Migrate between source and target endpoints that use the same database engine (e.g., on-premises MySQL to Amazon RDS MySQL)

Heterogeneous Migration

Migrate between source and target endpoints that use different database engines (e.g., Oracle to PostgreSQL)

Note

Important Requirement: One of your endpoints must be on an AWS service. You cannot use AWS DMS to migrate from an on-premises database to another on-premises database.

Supported Databases

Source Databases:

• Oracle, Microsoft SQL Server, MySQL, MariaDB, PostgreSQL
• IBM Db2 LUW, SAP, MongoDB, Aurora

Target Databases:

• Oracle, Microsoft SQL Server, PostgreSQL, MySQL
• Amazon Redshift, SAP ASE, Amazon S3, DynamoDB

Continuous Replication Use Cases

• Replicate from multiple sources to Amazon S3 to build highly available and scalable data lake solution
• Consolidate databases into petabyte-scale data warehouse by streaming data to Amazon Redshift

AWS DMS Homogeneous Migration

Homogeneous data migrations simplify migration of self-managed, on-premises databases or cloud databases to equivalent engine on Amazon RDS or Aurora.

Key Features

• Serverless: AWS DMS automatically scales resources required for migration
• High Performance: Uses native database tools to provide high-performing like-to-like migrations
• Native Tools: Leverages database-specific native tools for optimal performance

Architecture Components

Instance Profile:

• Specifies network and security settings for managed environment where migration project runs
• AWS DMS deploys managed environment when you create migration project with compatible source and target data providers

Migration Process

1. Connect to Source: AWS DMS connects to source data endpoint and reads source data

2. Data Processing: Dumps files on disk using native database tools

3. Data Restoration: Restores data to target using native database tools optimized for that engine

Example: On-premises MySQL to Amazon RDS MySQL migration uses MySQL-specific native tools for optimal performance and compatibility.

Tools for Heterogeneous Database Migrations

Heterogeneous migration requires additional tools for schema analysis, recommendations, and conversion at scale.

AWS DMS Fleet Advisor

Database Discovery Tool

Purpose: Automatically inventories and assesses on-premises database and analytics server fleet

Capabilities:

• Identifies potential migration paths
• Recommends database engine and instance options for migration to AWS
• Delivers results in hours instead of weeks or months
• Eliminates need for third-party tools or hiring migration experts

Schema Conversion Solutions

AWS offers two schema conversion solutions for heterogeneous database migrations:

AWS Schema Conversion Tool (AWS SCT)

Deployment: Download software to your local drive

Features:

• Converts source schema and SQL code into equivalent target schema and SQL code
• Standalone application for local installation
• Works in conjunction with AWS DMS

AWS DMS Schema Conversion

Deployment: Centrally managed service available within AWS DMS workflows

Features:

• Fully managed experience through AWS DMS console
• Integrated workflow for schema assessment and conversion
• No local software installation required

Both options automatically assess and convert source database schema and majority of database code objects to format compatible with target database. Objects that cannot be automatically converted are clearly marked as action items with prescriptive instructions for manual conversion.

AWS DMS Heterogeneous Migration with AWS SCT

The heterogeneous migration process involves schema conversion followed by data migration.

Migration Process

1. Convert Schema: AWS SCT automatically converts source database schemas and most database code objects to format compatible with target database

2. Create Migration Infrastructure: Create AWS DMS migration by setting up:

   • Replication instance (managed EC2 instance that hosts replication tasks)
   • Source and target endpoints
   • Migration tasks to migrate data between source and target data stores

3. Migrate Data: Execute migration tasks to transfer data from source to target with converted schema

Alternative Approach

Instead of AWS SCT, the AWS DMS replication task can use the AWS DMS Schema Conversion feature for a fully integrated experience without separate tool installation.

AWS DMS Replication Instance: Managed EC2 instance that hosts one or more replication tasks for migrating data between source and target data stores in an AWS Region.

Data Lake Replication Use Case

AWS DMS can replicate data from databases into data lakes for analytics purposes.

Higher Education Example

1. Source Data: Student Information System (SIS) data exists in on-premises database requiring ingestion into data lake for analytics

2. Source Endpoint: AWS DMS source endpoint connects to SIS data

3. Replication Task: AWS DMS replication task runs to replicate data from demo SIS database

4. Destination Endpoint: AWS DMS destination endpoint connects replication task to Amazon S3 raw data bucket

5. Analytics Access: Data becomes accessible by analysis and visualization services to derive insights from student information data

This architecture enables:

• Real-time Data Ingestion: Continuous replication from operational database to data lake
• Analytics Enablement: Data readily available for business intelligence and analytics tools
• Scalable Storage: Amazon S3 provides cost-effective, scalable storage for large datasets
• Minimal Impact: Source database continues normal operations during replication

Migration Best Practices

Planning Considerations

• Assessment: Use AWS DMS Fleet Advisor to assess current database fleet and identify optimal migration paths
• Schema Analysis: Utilize schema conversion tools to understand complexity and effort required
• Testing: Perform thorough testing of converted schemas and applications before production migration
• Timing: Plan migrations during maintenance windows to minimize business impact

Performance Optimization

• Right-Sizing: Select appropriate replication instance types based on workload requirements
• Parallel Processing: Use multiple migration tasks for large datasets when appropriate
• Network Optimization: Ensure adequate network bandwidth between source and target environments
• Monitoring: Use CloudWatch metrics to monitor migration progress and performance

Security Considerations

• Encryption: Enable encryption in transit and at rest for sensitive data
• Access Control: Use IAM roles and policies to control access to migration resources
• Network Security: Configure VPC endpoints and security groups appropriately
• Credential Management: Use AWS Secrets Manager for database credentials when possible

AWS DMS provides comprehensive migration capabilities for moving databases to AWS, whether for homogeneous migrations using native database tools or heterogeneous migrations requiring schema conversion.