Moving invoicing out of a monolith without losing a record

At Zoloo, a SaaS platform for bol.com sellers, I'm leading the migration of our core business domains out of the monolith into microservices. This one is about invoicing: generating, numbering, storing and delivering every invoice and credit note, including the upload to bol.com.

How we ended up starting with invoicing

The first plan was actually to migrate the big Orders functionality. Invoicing was one sub-functionality living inside it. When we went deeper into orders, invoicing was the piece that made the most sense to move first, into its own single-responsibility service. What we were aiming for was clear: better issue tracking, no downtime and more reliability.

It also had to go before orders. If orders moved first, the new orders service would have inherited invoice data it should never own. Getting invoicing out first gives us a cleaner code structure for future updates and for other shop integrations.

What I had to take care of

Invoices are fiscal documents, and the upload to bol.com is important for our users. So two things were non-negotiable from day one: no maintenance window, and no invoice number skipped or repeated.

What worried me the most was losing events during the partial migration of users. That's why we started with the accounts that receive the fewest orders, and only moved bigger ones once everything held.

On top of that:

• The new service had to print exactly what the old one printed, to the cent, rounding included. Customers already had those documents.
• Numbering had to move per account without gaps or duplicates, even with an invoice being generated at the exact moment of the switch.
• A small team. One backend engineer, me, working with an AI pair, while another migration (orders) was waiting to use the same events.

The plan, step by step

I went with the strangler fig pattern, moving users in cohorts. The order side publishes a snapshot of the order, and the new service only owns the fiscal documents.

1. Fix the monolith first. Numbering, scheduling and monitoring had to be right inside the monolith before moving anything. Otherwise you migrate the bugs together with the data, and later you can't tell old problems from new ones.
2. Transactional outbox. Every invoice request is saved as an event in the same database transaction as the change that caused it, and a relay publishes it with broker confirms. Nothing counts as sent until the broker acknowledged it.
3. Shadow mode. For me, shadow mode is basically executing everything the system needs to work in production, but without actually using that data in production. The new service generated every invoice next to the monolith, stored it apart, didn't assign numbers and didn't send anything. A job compared both versions (number, order, totals, VAT per rate), and the gate to continue was zero differences on a large sample. This way we could make sure everything worked as expected before it mattered.
4. Backfill the history while the shadow was running, with ids that make every batch safe to run again.
5. Reads behind a flag. Downloads and integrations ask the new service first and fall back to the old path if the document isn't there yet. A circuit breaker makes sure a deploy of the service can never block the monolith.
6. Writes by cohort, smallest first. A list of accounts plus a percentage decides who is migrated. Numbering is handed over per account: the new counter starts exactly where the old one stopped, and moving an account back is one command.
7. Send the results back. The service publishes events when a document is generated or delivered, and the monolith keeps a small reference, so the screens that list orders keep working without owning invoice data anymore.

Moving 40M+ records

• Simple reads only. The old rows were read in primary-key order, in fixed batches, with a read-only database user and a pause between batches. The live database only saw cheap, predictable queries.
• Safe to run again. Every migrated document gets an id calculated from its old key, so running a batch twice only inserts what's missing.
• Checkpoints. A watermark saves the last key copied with every batch, so if the backfill stops it continues exactly where it was, without copying anything twice.
• Daily deltas inside the service. The rows created after the big copy are picked up every day. That first ran as a cron job on the VM where our containerized services and workers live. It wasn't deployed correctly and silently didn't run, so we moved it into the service's own scheduler, where a failure shows up on its dashboard.

What zero data loss means to me

In this case, zero data loss means we don't lose order events, and as a consequence we don't lose invoice generation or anything that comes after it: the emails, the upload to bol.com, and so on. And if something fails, it gets rescheduled, so users don't have headaches tracking it later, doing the task by hand or contacting customer support.

To be sure about it, we checked on several levels:

• Counts per period, old against new, for each document type.
• Samples from storage. We fetched random migrated documents with the new service's own credentials. A row count tells you the record exists; only fetching the file tells you users can actually get it.
• Side by side. The shadow comparison ran for weeks before any account switched, and the last rounds compared tens of thousands of documents with zero differences.
• Live monitoring. A completeness check (orders that should have an invoice and don't) kept running during the whole cutover.

These checks caught real problems before any customer did: a bug in the verifier itself, an access issue with older files, and rounding differences between the old and the new code.

Mathematically correct isn't always correct for the customer

This one surprised me. The new code calculated VAT the mathematically correct way, and the old one rounded a bit differently. But customers compare what they see with their own data, and any difference makes them ask questions. As long as the data matches the fiscal rules and what they already had, sometimes the decision has to be about the business and not only the technical side. So we made the new service match the old output exactly, and left any "improvement" for a separate, deliberate change later.

A lesson about query shapes

The first night, the backfill got slower with every batch. The check for duplicate invoice numbers was written like account_id IN (...) AND number IN (...), and MySQL ran it as a scan of every row of every account in the batch. The more data we copied, the slower it got.

I noticed by reading the watermark twice, one hour apart, and then measured six versions of the query with EXPLAIN on a scratch database with fake data. The tuple version, (account_id, number) IN (...), also did a full scan. What worked was an index on account and number plus one OR group per account, and every batch went down to milliseconds. On a big table, how you write the query matters as much as the index.

How I worked with AI on this

A year ago I wasn't working with AI at all. On this migration it was part of my daily work, with clear rules:

• First I analyse with the help of AI, to go faster.
• Then I confirm with the team, especially with people who have been longer in the company and know the monolith better than any model.
• Then I execute progressively, always thinking about the final user.

Some things I never delegated: queries, data checks, running scripts, business models and new properties. The AI could copy an existing pattern, but I was the one adding them. Anything related to business logic, instrumentation and checking production data, I did myself, also to keep that data confidential. The AI was there to confirm and refine what I did, not to do it for me.

Results

• 40M+ historical invoices and credit notes in the new service with zero records lost, checked by counts, storage samples and side-by-side generation.
• No maintenance window: the whole migration ran with live traffic.
• The new service now issues invoices for the first accounts, covering every type of invoice, numbered exactly where the monolith stopped, and we're moving the rest progressively.
• Invoicing now deploys on its own in minutes, with its own dashboard, instead of going with every monolith release.

If you're about to do something similar

AI can mislead you. If you don't know the codebase well, actively ask the developers who have been longer in the team, who changed the monolith before and know the functionality closely.

And don't underestimate the waiting. Days of checking, progressive cohorts, migrating little by little: as a junior it can feel like too much work, or like a period that doesn't matter, because you want to finish everything as soon as possible. That time is exactly what I would highlight. During those days a lot of small things got tweaked and fixed.

The two things I'd do differently: match the old output exactly from day one, and write the verifier before the backfill, testing it with realistic data. A verifier is production code too.