Building the notification platform for a multi-tenant SaaS

The problem

Notifications were sent synchronously, in the middle of business rules. Some fired inside database transactions, so a rollback could leave a message already sent about something that never happened. Every notification had its own way of being sent, and email and SMS libraries were imported all over the codebase. When one failed, there was no record of when or why.

I was about to build a feature that leaned heavily on notifications. On top of that foundation it would have meant tangled code with poor performance and reliability, so I took on the foundation first.

The design

• Business code records intent, nothing more. A module calls one service that writes "this notification must be sent" to the database, in the same transaction as the business change. This is the transactional outbox pattern: the notification is stored if and only if the business change is.
• The event carries everything delivery needs. Each record is a "fat event": the content, the recipients and the channels the business code explicitly asked for. Delivery never has to call back into business code to find out what to send.
• A separate engine delivers. It reads pending events and puts each send on a BullMQ queue. Its workers build the format and destination each channel needs and send through that channel's provider. Because channels only consume the event, the engine is channel agnostic, and swapping the email or SMS provider touches one adapter, not the codebase.
• Recipients stay in control. Opt-in and opt-out work per notification, per customer and per channel.

When delivery fails

Each send gets three retries with growing waits between them. After that it goes to a dead-letter queue. A failure there is final: it is recorded as a failure event on that notification's timeline, and sending it again is a deliberate manual replay, never a silent retry. Every notification now has a timeline, so "why didn't the customer get this?" has an answer.

The tradeoff: polling instead of real time

Real-time updates over Socket.IO stay in the main application, where the connections live. A separate service holding its own socket connections back into the app would have coupled the two again. So the engine finds work by polling the outbox every few seconds instead of reacting instantly.

The cost is a few seconds of latency. In return the engine stays independent, and nothing is lost if it is down: pending events wait in the outbox and go out when it is back.

The outcome

• Any part of the platform can send notifications, not only the web frontend.
• Adding a notification type or a new channel is a contained change.
• Delivery lives in one place with one owner, which makes it easier to maintain.
• It is faster, and the main application got lighter: before, every pod of the app ran its own copy of the notification logic.

How I built it

I used Claude Code throughout: to pressure-test the design and architecture decisions, and most of all to write and review the code. The decisions and the final review stayed mine.

What I would do differently

I would spend more time on the long-term architecture up front. A separate microservice made sense at the time, but the engine turned out to be closely tied to the main application. Running it as a module inside the main app, still isolated from business code, would have kept the same separation and let in-app notifications go out in real time over the app's existing Socket.IO connections.