Thinking & Building Distributed First

When many of us are first taught to program, we’re told something along the lines of “programming is a set of very specific instructions that a computer follows in a specific order.” And this is true, sort of. It’s true, but it’s incomplete. What that statement ignores is that when we’re programming for the web, we’re usually dealing with more than one computer.

Multiple computers run our application servers. We use microservices. We have databases and background tasks and message busses and scheduled jobs.

These are the high-level things I wish someone had told me when I first graduated from my bootcamp. They’re questions to think about as we’re writing code within a system. I don’t have answers to all of these questions here — they’re highly context-dependent. But it’s important to consider these things early on in the development process — otherwise we might end up with a system that works delightfully on our local machines, but not at all in production.

The target audience here is folks who have at least some understanding of how web applications work and how to build them, but haven’t spent extensive amount of time working on highly-available distributed systems in production.

Does order matter?

If there are multiple side-effects of a given action, does the order in which they occur matter? Order is not guaranteed in many distributed systems:

• If you have multiple servers processing background tasks, they may be processed in a different order than they are produced.
• Similarly, in partitioned pub-sub systems, messages may be consumed in a different order than they were produced, or consumed by multiple different consumers in a different order each time.
• If you make multiple asynchronous requests, they might finish in a different order than they were initiated.

If order does matter, you’ll need to make sure it’s preserved, by putting all related actions in a single background task, using partition keys to make sure each message ends up on the same partition of your message bus, making your requests synchronous, or another mechanism appropriate to your system.

Is the data formatted for a network request?