Changing the message shape (Schema Evolution)

The problem: one small change, four broken services

In our online shop, four services read the OrderCreated event: reports, SMS, warehouse and accounting. Each service has its own team and its own release time. Today the message looks like this:

{ "orderId": "o-1", "amount": 250000 }

The shop wants to support several currencies. The order team turns the amount into an object:

{ "orderId": "o-1", "amount": { "value": 250000, "currency": "IRR" } }

What happens, step by step?

1. The SMS service still has the old code. In its code, the amount is a number.
2. The new message arrives and the amount is an object. JSON conversion fails with an error.
3. The consumer tries again and fails again. In Kafka, order matters, so the next messages also get stuck behind this message.
4. The problem also happens the other way. If a service moves to the new code earlier, it does not understand the old messages that are still in the topic.
5. Should we all release together? Making four teams release at the same moment is not possible in practice. Old messages also stay in the queue.

The main idea: a message is a contract

Think of a public API. You cannot change the type of a field one day and expect all clients to work correctly the same day. A message is exactly the same. Only harder, because:

• A message stays. A message written today may be read again next week or next year.
• The sender does not know the receivers. Maybe a service you do not know about also reads this message.

So the main rule is: old code and new code must work side by side for some time.

Two directions of compatibility

1. Backward compatibility (Backward). New code understands an old message. It is needed because old messages are still in the queue or topic.
2. Forward compatibility (Forward). Old code understands a new message. It is needed because all consumers are not updated at the same time.
3. Full compatibility (Full). Both together. The safest case for events that several teams read.

Which change is safe?

The right way: expand, migrate, contract

We split a big change into three small steps. Each step alone is safe.

1. Expand. Add a new field with a new name. The old field is still filled too.
2. Migrate. Each team moves its code to reading the new field whenever it is ready.
3. Contract. When everyone has moved and the old messages are not needed anymore, remove the old field.

The message in the first step:

{ "orderId": "o-1", "amount": 250000, "money": { "value": 250000, "currency": "IRR" } }

The sender side in C#:

public sealed record Money(decimal Value, string Currency);

public sealed record OrderCreated(
    string OrderId,
    [property: Obsolete("Use Money. Will be removed after all consumers move.")]
    decimal Amount,
    Money Money);

var message = new OrderCreated(order.Id, order.Total.Value, order.Total);

The receiver side: Tolerant Reader

The sender must be careful. But the receiver must also be tolerant:

1. Read only the fields you need. The SMS service needs only the order id and the amount. Do not define the other fields in its class.
2. Ignore unknown fields. The System.Text.Json library does this by default. Do not turn this behavior off.
3. For a new field, accept that it may be missing. An old message does not have the new field. So define it as nullable and have a fallback.

// The SMS service reads only what it needs.
public sealed record OrderCreatedForSms(string OrderId, decimal? Amount, Money? Money)
{
    // Works with old messages (only Amount) and new ones (Money).
    public decimal Total => Money?.Value ?? Amount
        ?? throw new InvalidOperationException("Message has no amount.");
}

Reading old messages again (Replay)

A new service wants to read all orders of the past year from the start of Kafka. The messages of one year have several different shapes. We have two ways:

1. The new service understands all shapes. This is possible for a few versions, but the code gets messy.
2. A conversion layer (Upcaster). Before the message reaches the main code, turn each old version into the new version. The main code sees only one shape.

For this, each message must carry its version. For example in the message Header, or with a schema id.

public sealed record OrderCreatedV2(string OrderId, Money Money);

public static class OrderCreatedUpcaster
{
    public static OrderCreatedV2 Upcast(int version, JsonElement json) => version switch
    {
        1 => new OrderCreatedV2(
                json.GetProperty("orderId").GetString()!,
                new Money(json.GetProperty("amount").GetDecimal(), "IRR")), // v1 was always IRR
        2 => json.Deserialize<OrderCreatedV2>(JsonSerializerOptions.Web)!, // camelCase names
        _ => throw new NotSupportedException($"Unknown OrderCreated version {version}.")
    };
}

Stopping it before Production with a Schema Registry

People forget the rules above. A tool can check them automatically.

1. The message shape is defined in an exact language. Usually Avro or Protobuf, and sometimes JSON Schema.
2. Each new version is registered in the Schema Registry. The sender registers the schema before sending.
3. The compatibility rule is checked. If the new version does not match the rule (for example Backward), the registration is rejected. So the error is seen in CI or during release, not in Production.
4. The message carries only the schema id. With this id, the receiver gets the exact message shape from the Registry.

Two points about the formats:

• In Protobuf, each field has a number. Never use the number of a removed field again. Mark it as reserved.
• In Avro, a new field must have a default value. Otherwise reading an old message with the new schema is not possible.

Common mistakes

How strict do you need to be?