Database migration to AWS

Author: Igor Royzis

History of AWS relational databases – or how we got here

Early 2000’s, everyone is happily buying servers, installing and configuring on-premise relational databases, both commercial  (SQL Server, Oracle) and open source (Postgres, MySQL). Life is good. Need more storage? Buy more disk space. Need more compute power? Buy more memory. Cannot upgrade anymore? Buy another server.

And then, at the end of 2009, when things are seemingly good, AWS introduces RDS for MySQL. Then RDS for Oracle in 2011, RDS for MS SQL Server in 2012, RDS for PostgreSQL in 2013, Aurora MySQL in 2014, Aurora PostgreSQL in 2017 … What’s happening? We don’t have to buy database servers anymore? We can upgrade storage and memory with a click of a button? What? Multi Availability Zone Support with a click of a button?

This is amazing! But we’re not ready to jump to this new technology yet. Let’s wait and see.

2015, 2016, 2017 .. time goes by. On-premise databases grow in size and complexity. Some applications are being moved to the cloud. Some new databases are being created in the cloud. Is it time to migrate our 15, 20 year old on-premise databases to the cloud? We’d love to, but it’s so complex, so much data, what about all the existing stored procedures and applications, how are we going to do the cutoff?

Sounds familiar? Well, it can all be done. It can be time-consuming, but in most cases it’s more straight forward than it sounds. Sometimes it’s complex, but it can and it’s been done with proper approach, planning and execution.

Use Cases for Database Migration to AWS

Here are a few popular database migration use cases, from simple to more complex. These are high level, just to give an idea of the approach.

E-commerce MySQL database migration to AWS RDS for MySQL

Source: On-premise MySQL

Size: 500GB

Target: RDS for MySQL

Complexity: Low

Approach in a nutshell:

– Use DMS (Database Migration Service) for initial full-migration and on-going replication

– Test/Switch applications to use new database and stop using on-premise MySQL

AWS technologies used:

– AWS DMS (Database Migration Service)

– RDS for MySQL

Multiple MS SQL Server databases migration to AWS RDS for SQL Server

Source: On-premise MS SQL Servers

Size: 30TB

Target: RDS for SQL Servers

Complexity: Medium

Approach in a nutshell:

– Use Snowball Edge, local DMS Agent, for initial full-migrations and on-going replication to S3

– Use DMS for migration from S3 to RDS for SQL Servers

– Test each web and mobile application against target database and switch one database at a time

AWS technologies used:

– Snowball Edge

– Local DMS Agent

– AWS DMS (Database Migration Service)

– S3

– RDS for SQL Server

Oracle data warehouse migration to AWS Redshift

Source: On-premise Oracle

Size: 50TB

Target: Redshift

Complexity: High

Approach in a nutshell:

– Rewrite ETL/ELT jobs to generate data for the new data warehouse

– Use SCT (Schema Conversion Tool) to generate target Redshift schema

– Use Snowball Edge, local DMS Agent, for initial full-migration and on-going replication to S3, followed by DMS for migration from S3 to Redshift

– Test/Switch BI tools to use Redshift and stop using Oracle

– Start running new ETL/ELT jobs to keep populating new data warehouse

AWS technologies used:

– AWS Glue

– SCT (Schema Conversion Tool)

– Snowball Edge

– Local DMS Agent

– AWS DMS (Database Migration Service)

– S3

– Redshift


Migrating databases to AWS is a matter of choosing the right approach depending on the types and sizes of your databases, as well as usage patterns, rate of data change and timing requirements.