"Middle-out" system design for effective distributed design
Leverage conceptual and logical models to achieve a distributed yet coherent system design.
I was recently asked how I approach system design, which made me realise there is a spectrum of design approaches, ranging from '(database) schema-first' to 'API-first'. I've seen engineers who work at either extreme (and I have done so at earlier points in my career), but the best engineers I've worked with operate in the middle, in what I'll now label as 'middle-out' design, a nod to my favourite 'documentary' Silicon Valley (unfortunately a not-quite safe for work clip). Where a database or API-first design focuses on concrete schemas (even in pseudocode form), middle-out design starts with conceptual and logical models from which we derive concrete designs. The power of this approach is that it enables tactical engineering designs and produces a shared mental model of the system amongst engineers (and even users) so they can make independent and distributed decisions now and in the future while maintaining cohesion.
In this post, I'll describe the three models and why I've evolved to a middle-out approach.
Schema-first design
When I first began software engineering, I remember starting designs with the system's data model. What tables do I need? What columns and constraints? How can we do a Third Normal Form design? I base these on the data and queries I need to manage.
This is a great place to start for simple CRUD-like form applications. Tutorials for common web app frameworks usually begin with "define your database model" (like Rails). Working on database schema is also a concrete place to start for novice developers; it's all about data types, relationships, and constraints. We are also more inclined to build out an entire database schema at the start, which can help us know where future extensions and features can fit.
The challenges with a schema-first design are:
We risk building APIs that aren't ergonomic for consumers. It is very tempting to replicate the data model in the APIs, especially when we get used to building CRUD form-based apps designed to manipulate database tables. While such systems are simple to make, we expose the inner workings to consumers. This is unnecessary complexity for consumers to understand, especially for sophisticated systems with complex relationships and optimised storage. Consumers need to compose a bunch of APIs and implement a lot of logic themselves to complete a task.
We risk coupling the API with the data model. If we mirror the data model in our API, it will be hard to evolve our system. When we need to change our data model, do we need to make corresponding updates to APIs? At the very least, it is a significant effort to introduce good abstractions to decouple the two and fight against inertia to change the system's architecture.
Consumers interact with our APIs, so they need to wait until our API designs are ready. i.e., they are spinning their wheels until we finally get to designing the APIs. This is a problem for any project with dependencies.
API-first design
In an attempt to compensate for the challenges of schema-first design, I then gravitated towards API-first design. We're building systems for user needs, so what they want must come first. Let's start with (Open)API specifications and how they will integrate with our system. We work through their Jobs To Be Done (JTBDs) and ensure they have APIs to suit. We might even have an API for each JTBD because 'that's what users would want'. Being agile, we incrementally build new APIs to solve JTBD as we learn about them.
This approach is especially nice for unblocking consumers; they can start working towards an API spec while we build out our system. The developer experience can also be pleasant as users can do each of their jobs using a single API rather than composing a bunch of calls.
However, taking an extreme API-first approach also has problems:
API specs don't cover all the constraints and business rules we need in our systems. There are limitations on what we can specify in API specs (e.g., many data and data type constraints that we could cover in a data model), and analysing a system only from a consumer perspective will not surface all the logic required because consumers shouldn't need to know about the inner workings of our system.
If we are not careful with our API planning, we risk an explosion of APIs to match ever-growing JTBDs. Eventually, this will become hard for consumers to comprehend, ironically making it harder for them to complete their jobs.
We risk a system that is hard to maintain and technically scale. This surfaces in a couple of ways:
We don't understand where data should come from. Many JTBDs need to expose and mutate the same data; for example, on your bank mobile app, you're likely to see your account balance repeated on multiple pages because it is helpful in various contexts. Which underlying model called by different APIs is the source of truth for this data? Do we need to synchronise? Our initial implementation likely involves atomically updating the value across multiple data models, requiring transactions and invoking the CAP theorem that limits our system's scalability.
We can confuse relationships between models exposed by APIs. Relationships are directional, and we need to avoid cyclic dependencies; this is why we encourage layered architectures that avoid the data layer having dependencies on the UI layer. However, good developer experiences typically require traversing relationships in both directions; I want to see a list of transactions for my credit card, and I also want to see which credit card a specific transaction was for. If we are too focused on the JTBD, we risk implementing relationships in the wrong direction. For example, maybe I create data models optimised for 'get all transactions for bank account' when it should be one for 'list all transactions, filtered by bank account'. Fixing this later is hard as it involves pulling apart dependencies and data migration.
Middle-out design: the best of both worlds
While API-first design is a significant improvement, it can sometimes miss the 'bigger picture' we're more likely to get to if we start with a comprehensive data model. However, 'middle-out' design offers the best of both worlds. It starts with the conceptual and logical models for your system, and then builds out APIs and physical models (implementation) from that. This approach ensures a comprehensive understanding of the system and allows for user-friendly APIs and a system that is easier to maintain and technically scale.
Rather than trying to directly solve jobs to be done, we ask, 'How does each of my stakeholders think about my system?' My JTBDs might be accepting pay from my employer and paying my bills using a bank account, but I think of my bank account as a bucket of money and a list of transactions in and out. My bank thinks of my account as a bucket of money that should always be positive, and there's likely a ledger concept to track fees and liabilities if I overdraft. When we learn how stakeholders view our system, we can then start defining our conceptual model:
Concepts (bank account, ledger, transactions) with specific purposes and rules.
Relationships between concepts, including direction and cardinality
Actions we perform on each concept or set of concepts.
We then extend this to logical models by defining attributes on concepts and actions (balance, amounts, transaction types). We evolve these models to ensure we can satisfy JTBDs.
How do we identify these concepts? Eventstorming is a great technique. I'll also have a follow-up post on breaking down domains.
How do we use these models?
Conceptual models are a great way to explain our system to developers and users. It is the mental model we want to share. It has just enough detail for those who don't need to understand the implementation.
Logical models are the starting point for API and data model design. Actions are our first-cut APIs; users can minimally do their jobs with these APIs, and we can decide whether to add optimised APIs that compose calls on behalf of users. Concepts with attributes are a first-cut data model that can be optimised to meet non-functional requirements, e.g., denormalisation, and the use of specialised stores such as search indices.
These models represent a North Star and principles for future design. We can determine which concepts are affected when new JTBD or requirements (e.g., new constraints) come in; instead of building new concepts willy-nilly, we're more inclined to try to fit them into our model, resulting in more cohesive APIs and implementation.
Concepts represent natural service and domain boundaries, enabling simple delegation of work. Aligning onconceptual and logical models allows engineers to work and make decisions independently.
Middle-out design gives us a sufficiently comprehensive big picture on which we (engineers and users) can hang detailed designs on and ensure future extensibility while getting many of the benefits of API-first design, including ensuring we provide a good developer experience with a system that meets users needs.
This approach does trade off a little execution speed; many engineers just want to start getting into APIs. However, we get a far more coherent user experience in a system that will be easier to maintain and grow because we're aligning to a North Star. I see it as an example of 'going slow to go fast'.
One of the challenges with middle-out design is that it is abstract and relatively few engineers are comfortable thinking inabstractions. Conceptual and logical models are a weird mix of abstract in the sense of not being constrained by implementation details yet precise enough to express components of a system and their rules. A common failure mode is producing conceptual models that do not have crisp definitions, resulting in 'handwavy' designs that can't be turned into something an engineer can code against. These are the designs that say, 'We could do this or that; it's whatever the user wants'. The problem with this is we do not get that shared understanding, which is the power of middle-out design. We have to be comfortable calling BS on vague designs and sometimes applying concrete examples (e.g. writing out pseudo-API specs or schemas based on JTBDs) to see if there is sufficient detail.
Summary - Middle-out design enables delegated design
One of our goals as senior individual contributors is to enable teams to independently make decisions 'as if we were the ones to make them'. Middle-out design aligns with this goal, which is why I think the best senior engineers I've worked with move towards it. With a shared understanding of a system's conceptual and logical models, many engineers can independently work on the more concrete design and implementation of individual components or features. In addition to enabling this distribution now, it also helps keep the system coherent over time if everyone returns to the same models (which evolve over time).
Moving towards this system design approach takes practice, and pairing, as I described in my last post, is one of the most effective ways to learn this skill. However, knowing this is a valuable skill to learn is an excellent starting point.