Engineering a Successful Database Migration

Like many times before, it was put on my plate to perform a data migration. I was once tasked with migrating a Dynamo Database with a massive size I’d never seen before (TBs).

Luckily, there was an article I used as my guide: https://aws.amazon.com/blogs/database/cross-account-replication-with-amazon-dynamodb

But in this article, we discuss the framework one can use to migrate persistent storage with success. You may find that the parameters of this task may not exactly line up with your application, but I hope you find some insight.

The process was daunting, but at the same time, as you break it down into steps, the anxiety lowers to a manageable level.

In every database migration you have to ask yourself a few questions:

The worst combination of this is No Downtime, and Complete Data Migration. That was my task. Shivers just went down my spine writing that.

I took a step back and realized that out of all the migrations I’ve done before:

All of these are pretty much the same problem, and while scale and the name of the product can change, we can analyze the problem and move forward.

Your toolkit will comprise of a couple of different components

For my example, it was in AWS infrastructure, and the Initial Datastorage was DynamoDB.

While DynamoDB is extremely cost effective for targeted queries, scans actually cause the table to be searched through for the requested object. This means time, and money. We’d have to find a mechanism to make the data accessible for our initial migration.

Firstly, we are going to need to migrate the database from a snapshot. From the moment you start copying the data down, you’ll need to be aware of what that timestamp was, or perhaps the last record. Keep this in mind for later.

Your migration may require transformation of data, so creating some piece of code that takes a chunk of data, transforms it, and places it into the new persistent storage is needed.

Note: If you are going to be pulling data from your Initial Datastorage, be careful you don’t DOS your customers by being the noisiest neighbor.

For AWS, DynamoDB released a Point in Time Recover (PITR) feature a few years back that can help restore a Database in the event of an outage.

Under the hood, it does this via dumping to an S3 bucket. While in an S3 bucket, you can manipulate the data using AWS Glue. This means that we have a replica of the data and accessing it will not use up bandwidth for the customer.

AWS Glue will give us the ability to transform data, and store it elsewhere. You can configure how many hosts are transforming and submitting data.

Let’s assume we finished moving that huge chunk of data over. But in the time it took between our migration, and now, we’ve missed a lot of data. How can we solve for that?

From the moment you kick off the migration, you will need to accumulate all changes somewhere, so that you can replay all of the traffic onto the new database.

In AWS, this is pretty easy, we can use Dynamo Streams. Dynamo Streams are a way to capture changes from one table, and with the help of an AWS Lambda, we can play these events onto the new Table and be good to go.

Dynamo Streams also provides multiple benefits:

In other cases, you will need some Queue/Buffer you can write to in order to ensure that the migration is successful, then you play that buffer after you migrate.

Now that our database is fully migrated over, our application still is using the old dataset. In order for us to start using the new Data Persistence, we need to break down our plan in sets of terms:

In the plan above, you can potentially coalesce T2, and T3 however I believe that the Point of No Return should be placed at the end of the process, to minimize potential impact. Some other benefits include:

It may seem daunting to make these changes, but we can use an old saying my Dad said “Code is guilty until proven innocent” to build confidence in our methodology.

This means we test one small change at a time, and we use data to prove each test.

Firstly, examine what pieces of data we believe we need to move from each step to the next, and build a hypothesis. For example:

In these I recommend you identify where your writes are happening, and travel one step up your class hierarchy to build a level of indirection to manage the migration. For example:

My DynamoDBDao.java is used to write to DynamoDB. I will travel up one layer to the class to DynamoDBController.java and make my changes here.

If you are lucky and your Databases are the same and use the same Datamodel, you can easily construct a new Instance of your DynamoDBDao pointing at the new table and begin writing your changes. If not, you have a little work to do to properly submit your records into your new DB.

You can easily take the plan we worked through in the previous section to build out your metrics and your application logic for the migration. The application doesn’t need to know about the code migration, and neither does DynamoDBDao, we can just use composition to handle this.

At this point you should feel ready to tackle a migration at any scale, and even be able to handle it when you’re afforded no downtime.

In the next part of this series, we will discuss best practices for the discovery and actual running of the migration.

Thanks,

Ben