Dan Iftodi

For more than seven years, I’ve been working on software systems where downtime is not an option—not even for a second. Deployments, upgrades, and migrations all have to happen seamlessly, while the system continues to serve users without interruption.

This requirement changes the way you think about engineering. Simple operations like adding, renaming, or removing a database column suddenly become complex. Modifying the structure of events in an event-driven system, evolving REST APIs, proxying requests, or even moving data between tables or entire databases all require careful planning.

Even in today’s AI-assisted coding era, migrations remain a critical challenge. Doing them right means thinking several steps ahead, anticipating risks, and designing strategies that ensure smooth transitions.

In this article, I’ll walk through some of the most common migration scenarios I’ve faced repeatedly in recent years—and the strategies that helped keep systems running without downtime.

Summary

In this article, I’ll focus on two common types of migrations:

1. Database migrations

Schema changes: modifying structures such as columns, data types, and indices.

Data migrations: moving data within the same table, across different tables, or even between databases.

Note: Handling migrations on very large datasets is a separate challenge—I’ll cover that in a dedicated article.

2. Message schema migrations

Updating message formats in event-driven systems.

Ensuring backward and forward compatibility to avoid breaking producers or consumers.

Database migrations

Database migrations are usually needed when business requirements change, when software becomes more generic, or when existing data no longer fits the old structure. Sometimes, data also needs to be moved between tables or even across databases for performance, scalability, or organizational reasons.

Data structure migrations

A data structure migration involves changes such as adding or removing columns, introducing or dropping an index, or modifying the data type of an existing column.

If downtime is acceptable, the process is simple:

1. Stop the application.
2. Run the SQL migration.
3. Restart the application.

However, when downtime is not an option, things become more complicated. A few of the main challenges include:

• Database locks: schema changes may lock tables or rows, blocking reads and writes.
• Application failures: deserialization of data can break if the new data type or structure doesn’t match what the application expects.

Strategies for zero-downtime migrations

To overcome these issues, the industry relies on a number of proven strategies. These techniques allow schema evolution and data movement while keeping systems online and users unaffected.do changes smoothly here are some strategies that industry is using:

Adding a column

Adding a new column is usually one of the simplest schema operations:

1. Execute the SQL script.
2. Deploy the new version of the software.

Important: Always apply the database change first, then deploy the application. Don’t combine these steps or reverse the order.

Removing a column

Removing a column is also straightforward, but the order of operations is reversed:

1. Deploy the new version of the software that no longer references the column.
2. Execute the SQL script to remove the column.

Note: If you drop the column before updating the application, it will fail at runtime.

Renaming a column

Renaming a column is more complex—it cannot be done by simply renaming it in the database without coordinating with the application. A safe approach is:

1. Add a new column with the desired name.
2. Copy data from the old column into the new column.
3. Keep both columns in sync until the application is updated (e.g., via triggers, dual writes, or background jobs).
4. Deploy the new version of the software that uses the new column.
5. Remove the old column.

Data migrations

Data migrations are about preserving the structure but moving the data elsewhere—whether to another database, a different service, or even across cloud providers.

Doing this without downtime is far more challenging than simple schema changes. The main issue is the gap period: after you’ve copied the data to the new location, you must keep it in sync until the application fully switches to the new source. Otherwise, writes to the old source will be lost.

A common approach

T0: Initial copy – perform a full copy of the data from source to the target, table A -> B (mysqldump, pg_dump/pg_restore)

T1: Change Data Capture (CDC) – every new data from A needs to be duplicated to B

• CDC Tools: Debezium, Maxwell Daemon – they are based on WAL/binlog of the DB
• OR Trigger-based CDC – listen on INSERT/UPDATE/DELETE and insert on the DB B

T2: Verify data consistency – check row count, incremental sampling

SELECT COUNT(*) FROM table_a;
SELECT COUNT(*) FROM table_b;

SELECT floor(id/10000) as bucket, count(*), md5(string_agg(col1 || col2, ','))
FROM table_a GROUP BY bucket;

SELECT floor(id/10000) as bucket, count(*), md5(string_agg(col1 || col2, ','))
FROM table_b GROUP BY bucket;

T3: Deploy the new software that uses B.

T4: Monitor

T5: Stop syncing, retire A.

Message schemas

By message schemas, I refer to the structure of messages that are produced or consumed in an event-driven architecture. Conceptually, you can think of them as similar to database schema migrations.

In most real-time systems, schema evolution must be handled carefully so that producers and consumers remain compatible while changes are rolled out. This becomes especially challenging if events are persisted for years and then later re-consumed—for example, for analytics or replay.

One common technique is live schema migration: when a message is read, its schema is transformed into the version expected by the application. This allows producers and consumers to evolve independently while maintaining compatibility.

Adding a new property

Adding a new property to a message schema is generally straightforward:

1. Update the schema – Add the new property and increment the schema version.
2. Deploy the producer – Release the new version of the software that produces messages using the updated schema.
3. Deploy the consumer – Release the version of the software that can read and handle the new property.

In more complex setups, schema registries (like Confluent Schema Registry) and serialization frameworks such as Apache Avro can help enforce compatibility and versioning, making it easier to evolve schemas safely.

Removing a property

Removing a property from a message schema is similar to column removal in a database—the key is the order of operations:

1. Deploy the updated software – Release the version that no longer relies on the property.
2. Update the schema – Remove the property and increment the schema version.

Note: Removing the property before updating the software can lead to runtime errors, so always update the consumer first.

Renaming a column

Renaming a property in a message schema is more complex than simply changing its name. A safe approach is:

1. Add the new property – Increment the schema version.
2. Start producing messages with both properties – Keep the old property for backward compatibility.
3. Deploy the consumer update – Release the new version of the software that reads the new property.
4. Remove the old property – Once all consumers are compatible, the old property can be safely removed.

Note: This approach works for simple scenarios. In more complex systems, additional migration logic may be required to transform messages before they reach the part of the software that actually consumes them.

Example:

interface Message {
    int getVersion();
    .....
}
interface Migrator {
    Message migrate(Message inputMessage);
}
Map<SchemaVersion, Migrator> migrators;
schema1:migrator1
schema2:migrator2

> When Message is received:
migrators.get(message.getVersion).migrate(message);

^ do this until you reach the latest available version.