Vivian Voss

Event Sourcing: The Archaeology Tax

architecture sql performance

The Invoice ■ Episode 5

“Complete audit trail! Time travel! Never lose data!”

Marvellous. Let us examine the invoice.

Event sourcing is one of those architectural patterns that sounds unobjectionable in a conference talk and becomes progressively less charming once it meets production data. The pitch is elegant: instead of storing the current state, store every event that led to it. The system becomes a ledger. You can reconstruct any past state by replaying the event history. Audit trails emerge for free. Time travel is built in.

The pitch is not wrong. It is merely incomplete. Rather dramatically so.

The Storage Invoice

In a traditional database, a shopping cart with ten items is one row. Ten columns, or a JSONB array, or a normalised set of line items. The state is the state. When the customer removes item seven, you update the row. Storage cost: constant.

In an event-sourced system, that same shopping cart is a sequence of events: CartCreated, ItemAdded (ten times), ItemRemoved, QuantityChanged, CouponApplied. Thirteen events to represent a cart that currently contains nine items. The customer’s browsing indecision is now a permanent part of your storage infrastructure.

Shopping Cart: Storage Cost Traditional (PostgreSQL) 1 row Current state. Done. Event Sourced 13+ events CartCreated, ItemAdded ×10, ItemRemoved, CouponApplied … At Scale 1 row per entity, constant 3 TB replay for one balance

Scale this to a real system. Every user action, every state transition, every correction, all preserved, all immutable, all growing. A real-world case study reports 3 TB of events to reconstruct a single account balance. Query times measured in minutes, not milliseconds. The database is not slow. It is doing exactly what you asked: reading three terabytes of history to answer a question about the present.

The Replay Invoice

The recovery story is the centrepiece of every event sourcing talk. System corrupted? Replay from the event log. State lost? Rebuild from events. It sounds like a superpower until you do the arithmetic.

Millions of events. Sequential replay. No shortcuts. Events must be applied in order, because that is the entire point. A production replay of a moderately sized system: ten hours. Your system is down. Your customers are waiting. Your on-call engineer is watching a progress bar that moves with the urgency of a civil servant on a Friday afternoon.

The solution, inevitably, is snapshots. Periodically capture the current state so you only need to replay events since the last snapshot. Sensible enough. But consider what you have just done: you have introduced a second storage system (one that stores current state) to compensate for the fact that your primary storage system does not.

Snapshots are an admission that the architecture does not scale. If storing every event were sufficient, you would not need periodic photographs of the result. The snapshot is not a feature of event sourcing. It is a workaround for event sourcing.

The Schema Evolution Invoice

Events are immutable. That is the contract. Once written, they never change. Splendid, until your domain model evolves, which it will, because domains evolve, because the business evolves, because the world is not a versioned API.

Version 1 of your OrderPlaced event:

{ price: 100 }

Version 2, six months later:

{ price: 100, currency: “EUR” }

Every event written before version 2 lacks a currency field. They cannot be updated (immutability is the rule). They cannot be ignored (the replay depends on them). They must be handled, forever, by every consumer that reads the event stream.

Five patterns exist to manage this: versioning, upcasting, weak schemas, in-place migration, and copy-and-transform. Each adds complexity. Each introduces its own failure modes. Each requires that every developer on the team understands not just the current schema, but the entire history of every schema that has ever existed.

The Event Sourcing Iceberg waterline The Pitch Immutable events Complete history Time travel The Invoice Schema versioning (5 patterns) Upcasting old events forever Snapshot maintenance Replay testing Eventual consistency (200 ms lag) Doubled onboarding time Debugging as archaeology The top sells the pattern. The bottom is the invoice.

Greg Young, the person who popularised event sourcing, wrote an entire book about versioning alone. When the inventor needs a book-length treatment for a single sub-problem of the pattern, that is not documentation. That is a warning label.

The Consistency Invoice

Event sourcing typically separates the write model (event store) from the read model (projections). Commands produce events. Projections consume events and build queryable views. The read store is “eventually consistent”, industry shorthand for “not consistent yet.”

The typical lag: 200 milliseconds. In conference talks, this sounds negligible. In production, it means a customer places an order, the confirmation page loads, and the order is not there. The customer refreshes. Still not there. They place the order again. Now you have two orders and a support ticket.

“The read store is 200 ms behind.” Try explaining that to a customer who has just entered their credit card details. Try explaining it to the product owner. Try explaining it to yourself at two in the morning when the pager goes off and the answer is “the system is working as designed.”

The Team Invoice

Every developer must think in events. Not in state, not in entities, not in the mental model that every database textbook, every ORM, and every line-of-business application has used for the past forty years. Events. Sequences of facts that, when replayed, produce the current state, provided you handle every schema version correctly, apply the upcasters in order, and remember that the read model is 200 milliseconds behind.

Onboarding doubles. Debugging becomes archaeology. A bug report does not say “the balance is wrong.” It says “somewhere in the 47,000 events for this account, an event was projected incorrectly, possibly by an upcaster that was written eighteen months ago by someone who has since left.”

AWS itself warns in its own prescriptive guidance: event sourcing “should not be the default choice due to its complexity.” When the platform that would directly profit from you using more infrastructure tells you to reconsider, the polite thing to do is listen.

When It Earns Its Keep

Event sourcing is not universally wrong. It is specifically right in domains where the event history is the business value. Banking: every transaction must be auditable, reversible, and reconstructable by regulation. Accounting: the ledger is literally a sequence of events, and has been since double-entry bookkeeping was formalised in 1494. Regulated industries where Martin Fowler’s original description applies: the audit trail is not a feature. It is the product.

In these domains, the cost is justified because the cost of not having a complete event history is higher: regulatory fines, legal exposure, compliance failures. The complexity is the same. The calculus is different.

The Alternative

PostgreSQL. An audit table. A trigger that logs every INSERT, UPDATE, and DELETE with the old values, the new values, the timestamp, and the user. Five lines of SQL. No event store. No projections. No eventual consistency. No replay. No snapshots. No five versioning patterns. No book-length treatment of a sub-problem.

You get a complete audit trail. You get the current state in one query. You get consistency measured in microseconds, not the 200-millisecond prayer of eventual convergence. You get a system that every developer on the team already understands, because it is a database doing what databases do.

The audit table is not as theoretically elegant. It does not permit time travel or state reconstruction from first principles. It does, however, answer the question that actually matters, “who changed what, and when?”, without requiring a specialist team, a snapshot strategy, and a versioning book.

The Verdict

Event sourcing stores every state transition to avoid storing state. It introduces snapshots to compensate for the cost of not storing state. It adds projections to provide the queryable state it declined to store in the first place. It demands five versioning patterns to handle the immutable events that inevitably need to change. It splits consistency into “eventual,” which is a polite way of saying “not yet.”

The alternative is a database and an audit table.

The invoice is not the event store. It is everything you build around it to make it usable.