Lately I’ve been thinking about productivity and it’s evolutionary rhythm while scaling. Like, how easy the cadence ebbs. My lived experience as a software developer often feels like anfractuous GPS reroutes. For example, we have split teams to tackle different product areas in parallel. Usually this means working separately in our “owned” codebases. But software is never neat. So when the team in Mendoza needs review on a changeset that touches our code area, we’re talking temporal crossings to achieve conciliation.
It’s late afternoon in Argentina wine country when I’m just digging my heals into a morning Philz roast.
The additional burden of a cross-geo async request means that I’ll have to hunker down to decipher what they’re proposed code is doing and what it’s supposed to do – what might take a few minutes pairing could take tens of minutes. While async communication over Slack is ok, closing the mental gap in understanding the code changes is tough. We are facing a kind of Derridean differánce. As if navigating the palimpsest wasn’t enough. Multiple personages reading and explaining the code text, scribbling comments, changing, re-reading the next day; discovering the truth in the spaces/tabs. Reading software is really tough stuff. This impact of distance and time spent to “get up to speed” (pun intended) is a delta that doesn’t always feel satisfyingly addressed by organizational planning.
Does this sound like what you do for work?:
Every software project is like a palimpsest where every developer scratches old text, and writes on top of it their perceived new solution, creating a manuscript where a mix off styles and languages reflect different understandings of the world, different world visions.
And
Différance is the systematic play of differences, of the traces of differences, of the spacing by means of which elements are related to each other. This spacing is the simultaneously active and passive (the a of différance indicates this indecision as concerns activity and passivity, that which cannot be governed by or distributed between the terms of this opposition) production of the intervals without which the “full” terms would not signify, would not function.
Holy complicatedness, richness, depth! No wonder you can’t just throw more devs at the problem. Though apparently that’s what men in the industry used to think.
—
Not too long into my first software gig my team of 5 engineers organized a book club and tackled The Mythical Man Month. That’s when I really learned how dev teams became reduced to fungible “resources.” (It wasn’t long before I was at a happy hour and our head of sales was asking me, “Why can’t we just ship the work you do to India?” So I apparently it’s a historically recurrent theme.) It was also my first introduction to the idea that you can’t just throw more people at software problems to achieve speed gains.
Adding manpower to a late software project makes it later.
If we believe Jessica Kerr and the other cyberneticians of our ilk, and on this point I might – software is a symmathesy: a learning system of learning parts. Brooks must have believed something similar I just don’t remember his exact words.
Nonetheless, I interpolate. Brooks' insights feel situated as early symmathecist thinking. Software is so highly sophisticated, so highly complicated!, that command and control thinking can’t intervene to establish order:
As time passes, the system becomes less and less well-ordered. Sooner or later the fixing cease to gain any ground. Each forward step is matched by a backward one.
So why does this happen? Like, despite Brooks' spacious, ethereal characterization of our work as only slightly removed from pure thought-stuff where we build castles in the air (words I do remember wonderingly)…we definitely live through a tangible stickiness, queues, waiting rooms; especially mid-to-late in the build cycle. Things get really hard. I swear there’s something even metallic about a build failure.
In order to change our system, we need a mental model of it. Each developer has a mental model of the software we work on. Each developer’s mental model is necessarily incomplete and out of date. And, they’re each different from everyone else’s. (This is called Woods’ Law, if I recall correctly.) We spend time reconciling our mental models enough to communicate with each other; this is a coherence penalty.
Well, we all know the drill and can recall with pretty sharp clarity what tends to happen. Making code changes later in a project becomes increasingly difficult because previous design decisions have constrained possibilities. More consistency is demanded; maybe not total, but if the component handlers should call to the reducer, why are you calling setState() directly in the handler? Because…the rush to the finish line produces anxiety that strains communication (within the already polynomial channeling). You shift from pensive philosopher to tactician; code breaker. Code review quality suffers; if only for the obvious reason that you need to double down on crafting and glueing code for more hours in the day. This, in turn, decreasing mental reserves for context switching to someone else’s code. All that great senior leadership out the door. There’s simply less consensus building. Public Slack channels are quieter. Regardless how high-performing the team, I’m convinced there will be some communication degradation in the twilight hours.
—
When I read Coda Hale’s recent piece Work Is Work all of this ugh was loaded into view again; and I really appreciate his analysis of how we end up in slog mode; and the antidote. Maybe the industry hadn’t received a really concise reflection in a while. Maybe it’s the mathematical proofing and formalism that adds a seductive element for the modern SWE struggling in their unwieldy symmathesies; what Karatani would call our “will to architecture”. This piece was going around the Greater than Code sphere for sure. For my philosopher/poet brain for sure. Ok, so similar to Brooks, but for orgs, Hale says:
At some point in time, every organization realizes that it’s slowing down.
Like Brooks, Hale is dealing with that slog. But with the scope zoomed out to help us understand how to manage and overcome the obstacle of big organizations growing at economies of scale. He enriches the idea of the mythical man month with Amdahl’s law:
The work capacity of an organization scales, at most, linearly as new members are added.
Amdahl’s law: I’m still wrapping my head around it! But I believe the gist is: when we try to add more people to parallelize parts of the dev work, we won’t get any dramatic (exponentialized) speed gains over the sequential work of the smaller group. Amdahl observed this behavior while throwing additional processors at a computational task in a computer machine. The “parallel fraction” of speed increase when adding N new members is only relative to the original group size, and eventually tops out and flattens. Rather than the speed boost our command-control leadership wills, we get something a flip side slog:
Coherence costs grow quadratically as new members are added.
Slog = coherence costs! That’s a sweet little buttoned up phrase for it.
Around the same time I read Hale’s piece I was picking up the aforementioned Kojin Karatani book, Architecture as Metaphor. So while I’m sitting with these ahas! about how the frustrating things at work are these somewhat well-understood “coherence costs” that occur on large teams like mine which can be observed with rigor from the outside (etic)…I’m suddenly finding myself in chapter 4 of Architecture as Metaphor called the Natural City where Karatani continues to build his argument for the ways that theory is entrapped again and again by the architectonic. But the thread I want to pull on is simply Karatani’s synopsis of Christopher Alexander’s The City Is Not A Tree. Lol, of course I should meet Alexander here. He is like one of the father’s of Software Patterns – see his 1996 keynote at OOPSLA – that would inspire so much of how our apps are built at scale today.
In the seminal essay The City Is Not A Tree, that would inspire people across fields and perhaps give rise to the concept of bounded rationality, Alexander proposed mathematically based, formal, structural, ideas (just like Hale is doing) to describe the “natural city” as a semi-lattice; a concept from Set theory.
A collection of sets forms a semilattice if and only if, when two overlapping sets belong to the collection, the set of elements common to both also belongs to the collection.
He is thereby able to reveal the “natural city” as something we can think about rationally. Particularly, and importantly, rationally distinct from the nemesis “artificial city” – a command-and-control style approach to designing cities in a treelike fashion from those who don’t appreciate the the richness, subtlety, and complexity in the overlaps that otherwise staves off “coming destruction” and “anarchy.”
When we think in terms of trees we are trading the humanity and richness of the living city for a conceptual simplicity which benefits only designers, planners, administrators and developers. Every time a piece of a city is torn out, and a tree made to replace the semilattice that was there before, the city takes a further step toward dissociation.
And
The semilattice…is the structure of a complex fabric; it is the structure of living things, of great paintings and symphonies.
My brain keyed on semilattices because they seemed to describe the structure of a software system I’ve been coming to know through Kerr and Bateson (opera, camerata, living) – not to mention the cutely convenient lexical overlap of mathematical terms in the definition; eg, a “Set” is a common data structure implemented by many high-level programming languages. Mendoza devs putting their text in our text. Some overlap of time zone. Knowledge overlaps as more tenured developers work in more areas of the codebase. Overlap at the coffee station, at the lunch table. Overlapping documentation, nearly duplicate. Also the incomplete lines of connection. The grassroots partnerships and friendships that emerge across teams. The polymorphic ensemble of patterns and style in our web applications: classes called as functions, state reducers and DOM events, REST with GraphQL, compositional components within hard-bound graphs. Microservices databases implementing Order and User over and over again. The semilattice seems very familiar.
Therefore, in the context of a symathesy, where Alexander might say our bounded rationality (not to mention hubris, or inexperience, or ego, or other ingressive tendencies) limits our ability to observe and study the subtlety and richness of the organization, throwing more men at the problem is short-sighted as fuck. Managers can’t govern from apart, at the top of the edifice. There are superlinear interactions to behold.
Then the advice that Hale starts giving to tackle coherence costs makes a lot of sense:
The only scalable strategy for containing coherence costs is to limit the number of people an individual needs to talk to in order to do their job to a constant factor.
Limit the overlap!
As with heavily layered applications, the more distance between those designing the organization and the work being done, the greater the risk of unmanaged points of contention. Top-down organizational methods can lead to subdivisions which seem like parallel efforts when listed on a slide but which are, in actuality, highly interdependent and interlocking. Staffing highly sequential efforts as if they were entirely parallel leads to catastrophe.
And
As an organization hires more employees, work on productivity improvements must be a constant priority. Internal tooling, training, and services must be developed and fielded to ensure that all members are able to work on problems of continuously increasing impact. The ceaseless pursuit of force multipliers is the only possible route to superlinear productivity improvements as an organization grows.
In an essay I assume Hale drew from, and to which Jessica Kerr’s above quote references, Michael Nygard makes a similar suggestion:
…take a look at your architecture, language, tools, and team. See where you spend time re-establishing coherence when people make changes to the system’s model of the world…take a look at your architecture, language, tools, and team. See where you spend time re-establishing coherence when people make changes to the system’s model of the world.
I love all this. Of course, it’s hard. Alexander tested folks, experimented to discover that the tendency was to “reorganize” the mental model into a treelike structure (though perhaps he only interviewed a group of limited white men, or folks embedded in white supremacist society?) But I suspect some of the hardness has to do with:
-
Our inability to see the costs (ie the complexity, simply constrained by our brain’s inherent limitation to expansively see/hold it all); for leadership looking at the org we can also probably say that their “mental model is necessarily incomplete and out of date”.
It is similarly difficult to visualize the semi-lattice structure – where multiple sets overlap–of the natural city, and thus we tend to reduce it to a tree, the only structure we can visualize.
-
An ignorance that we have a semilattice at all. Who out there amongst leadership are systems thinkers? Embracers and students of variety, incompleteness, ambiguity, multiplicity? Who can embrace these, split things, and double down on rewiring the learning (neuronic) threads? Ensembles over A/agile. More of that interspecies contamination, contaminated diversity that Anna Tsing talks about. Mushrooms and the dying earth can help.
Where a matrix indicates a high-touch relationship between two groups (e.g., a group of engineers working on a feature and the lawyers trying to ensure the legal compliance of that feature), efforts should be made to reduce the cost of that interaction by colocating their members (e.g., embed a lawyer with the engineers).
Management or whatever version of leadership you have must stay on top of it. It’s some serious work. Maybe we need better integration tech; it’s like we are missing some connective tissue. Maybe it’s not tech at all.
I remember being met with a lot of skepticism when I suggested to some engineers at my company that we create a meeting support team. It was a partially-formed thought, but the basic idea would be to make a self-selected group available to take notes for other peoples' meetings. I don’t think many devs grasp the importance of their meeting communication as architectural design records; and similarly how hard it is to accurately and correctly document after the fact; or document in progress for someone who is also trying to participate. Living in entropy is easier than culture change. I also understand that no computer science education ever taught anyone this stuff was important which is why Brooks' observations went unheeded for 50 years.
Postscript:
Why did I keep trudging through Karatani while writing this essay?? I should have known better: a few chapters later I’m cry-laughing because I bump into that hopeful schizo theory which blew my mind like 20 years ago in undergrad. And now I’m like damn because Karatani is saying that the semilattice is probably not the ideal (post)structure I want to imagine upon my software company. Of course, this makes sense in Karatani’s argument because he wants to problematize architectonic thinking.
Despite its appearance, [the semilattice] is orderly and centered.
And
What happens when the semilattice is broken?
By Delueze and Guattari of course. Whatever, I still think the semilattice is a good formal representation of why coherence costs emerge in the indeterminate, rich encounters of the software symmathesy; the gentle swish of overlapping domains; like at Hearst and Euclid. But the rhizomatic network requires consideration soon. I’m urged that way anyway – see my hints above about interspecies contamination and the anthropocene. I think the ensemble moves toward assemblage soon.