Invoice Generation Pipeline

The problem

Postpaid billing has an unforgiving property: it is not eventually consistent in any way a customer will accept. A bill that arrives twice is a complaint. A bill that never arrives is lost revenue and, at scale, a regulatory problem. The run happens on a cycle, under time pressure, and the input is a pile of XML large enough that no single process should be holding it in memory.

The system I inherited and then rebuilt had to take that XML, resolve every customer’s charges — including corporate hierarchies where a parent account absorbs its children’s usage — render a PDF per customer, and deliver it. Roughly two million times, reliably, every cycle.

Shape of the system

Five stages, each its own deployable, connected by Kafka topics rather than calls:

invoice pipeline

The parser uses Spring Batch because a billing run is genuinely a batch job — it has chunks, restartability and a natural failure boundary, and pretending otherwise means rebuilding those primitives badly. Everything downstream is stream-shaped, because once a customer’s charges are resolved, that customer is independent of every other customer.

The hard part: a producer that stalls without failing

The failure that cost me the most sleep was not a crash. It was silence.

Under a heavy cycle, the Kafka producer could reach a state where it neither threw nor delivered. Sends returned futures that simply never completed. Nothing in the health checks noticed, because the service was alive, the consumer lag looked plausible, and no exception was ever logged. The run just quietly stopped making progress, and we would find out when someone asked where the invoices were.

Three changes fixed it, and the order matters:

• A timeout backstop on every send. A future that never completes is not an edge case to be tolerated — it is a bug that must become an exception. Bounding it converts an invisible hang into a loud, catchable failure.
• Wedged-producer detection and reset. Once a stall is detectable, the service can tear the producer down and rebuild it rather than sitting in a broken state waiting for a human.
• Explicit backpressure on the render stage. PDF generation is the slowest link. Letting the parser run ahead unbounded just moves the queue from one place to a worse one. Making the fast stage aware of the slow one keeps memory flat and the cycle predictable.

The general lesson I took from it: in an asynchronous system, absence of an error is not evidence of progress. Anything that can hang must have a clock on it.

Corporate hierarchies

The other genuinely hard piece was billing structure. A corporate customer is not one account — it is a parent with children, where charges roll up, some are settled centrally and some are not, and the invoice has to reflect that at both levels. Getting the traversal right, and keeping it correct as accounts move between parents mid-cycle, took more care than any of the infrastructure work. It is also the part that no diagram makes look impressive, and the part that would have hurt most to get wrong.

What I would do differently

The end-to-end test harness came late. For a long time the only way to know the chain worked was to run a cycle and watch. Building the containerised harness — Kafka, mail capture, the whole chain cold-startable with one command — should have come first, not after the pipeline was already carrying real billing runs.