Distributed Systems: an introduction to Publish-Subscribe (pub/sub)

More and more of the web is moving to microservice architecture, which allows for loosely-coupled services to work together to provide functionality to users. While these services will often communicate with each other via network requests, the publish-subscribe model is another common way for this collaboration to occur.

There are many different technologies that can be used to implement pub-sub, including but not limited to Apache Kafka, Amazon Kinesis, Google Pub/Sub, and Microsoft Event Hubs. The purpose of this post is not to go into the details of implementation or discuss the pros and cons of these specific technologies, but to provide definitions of the words and an overview of how it works.

I talk about two different types of events that can be passed through a pub-sub model, Change Data Capture events and “business events” here, though these messages could of course be anything!

What is decoupling and why is it good?

Before we dive into pub-sub, let’s take a quick detour to decoupling. If we’re not convinced of the value of decoupling, pub-sub won’t do us any good!

Decoupling means reducing the dependency between pieces of a system. This can mean two classes in the same codebase or two services in a system built on microservices, or anything in between.

Ok, but what do we mean by “reducing the dependency”? Obviously the different parts of the system rely on each other — they’re part of the same system! Typically what people mean when they talk about the level of dependency between systems is whether or not one part of the system needs to be changed when the other does, or whether they can be updated independently of the other.

Pub-sub is decoupled because using it, we can change anything we want about a publisher — we could even go so far as to rebuild it using an entirely different language or framework! — but as long as it continues to produce messages in the same way, we don’t need to change anything at all about the downstream subscriber. Same goes in reverse — as long as a subscriber continues to be able to process messages that come to it in the same format, we can change anything we want about it and the publisher doesn’t need to be aware of this change at all.

This will make more sense once we dive into this a bit more, so let’s do that!