Conway’s Law and Technical Debt

Everyone wants to be agile. It is the catch cry of the day, along with cloud computing, big data and payments processing.

Agile software development boils down to do more with less, and communicate more with less. Do more with less is about using fewer but better suited tools to improve productivity. Communicate more with less is about increasing the design discussion and feedback loops without the overhead of maintaining large documents – note the difference between “discussion” and “documenting what we built”, something many projects confuse.

But I’d postulate that all software development methodologies still fundamentally view software development as a single team activity; despite our typical work environment consisting of multiple teams; multiple delivery structures.

Yes, every project manager has the list of external dependencies on other teams and other systems, but it is just a milestone tracking list. It doesn’t help us become more agile. Similarly, the Scaled Agile Framework is more about making the project management office or the enterprise release management teams more agile than addressing the multiple team challenge.

So what’s the problem?

The below diagram illustrates three projects tasks to deliver three solutions. The three teams must leverage existing corporate assets (e.g., an inventory system or general ledger system) to avoid causing trouble, but the trade-off is overlaps between the three solution spaces. Secondly, the trade-off agreement between team “A” and team “B” will in all likelihood impact the possible trade-off agreement between team “B” and team “C”.

Solution overlaps

Conway discussed this problem back in 1968 and framed it as the correlation between organisational structures and software structures. Bass and others have found similar evidence of projects getting into trouble due to “architectural misalignment”; that is the difference between the organisational enabled communication structure versus the communication structure needed to solve complex architectural problems. This “structures correlation” problem is commonly referred to as Conway’s Law, although “structure” does need to be understood more in context of Bass’ design problem discussion than Conway’s original and more literal computer code structure context.

While we probably don’t have a good answer for dealing with Conway’s Law, it struck me that Martin Fowler’s Technical Debt Quadrant is useful for illustrating the four phases of (what I have coined) the Conway’s Law Denial:

  1. Your company’s methodology calls for identification of project interdependencies; dutifully noted as part of the initial planning efforts.
  2. Early design work uncovers overlaps with other projects (hopefully on your list). But we are all polite and work together, as schedules haven’t yet been tightened (too much); aka we send a few emails with draft designs attached.
  3. The true complexity of the “structures correlation” problem arrives and full-blown denials starts
  4. Final stage of full-blown denial

Projects and TD

And why does all of this matter; other than stop annoying colleagues?

Teams disconnected by organisation, location and/or time cannot make appropriate trade-offs and the scene is ripe for piling on the technical debt.

References

Mel Conway, “How do committees invent?“, 1968

Bass, M. et al, “Architectural Misalignment: An experience report“, 2006

Kwan et al., “Conway’s Law Revisited: The Evidence For a Task-based Perspective“, 2012

If “Buy Before Build” is the answer then what was the question?

If you made a list of the “Top Five All Time Favourite” principles, then I’m sure “Buy Before Build” would be on the list. It just seems like one of those obvious statements. Why wouldn’t you buy of the shelf – proven – software to reduce delivery risk, outsource (non-core) software development, and gain incremental improvements through upgrades.

Buy before build
Buy before build

That’s all very good in theory, but it only works well if the off-the-shelf software provides a good functional fit in terms of business processes, data and organisation, as well as a good technical fit in terms of non-functional requirements and existing technology environment.

The Buy-Before-Build Anti-pattern

This anti-pattern exists when an organisation deploys off-the-shelf software with a poor fit. The result is a pile of custom developed code to change the off-the-shelf software (as illustrated below).

Buy before build - the reality
Buy before build – the reality?

Why would this happen? A poor fit exists when:

  • The (assumed) business processes built into the off-the-shelf software requires significant business process re-engineering. A system implementation project can accommodate smaller changes, but larger changes are unlikely without a dedicated effort. And that’s of course assuming it is in the organisation’s interest to align business processes. Off-the-shelf implies commoditisation, and no one would like to commodities their business advantage.
  • Integration with existing IT systems will be difficult and expensive, where the off-the-shelf software implicitly assumes itself to be the data and/or process master. Even if the new system has a perfect functional fit, the custom code required to ensure good synchronisation between two masters is the perfect recipe for cost blow out.
  • Operation management of an off-the-shelf software system designed for business hours only operation may prove impossible to do within an 24 by 7 environment. Gaps are likely to occur due to long-ish maintenance windows, lack of true redundancy and insufficient configuration management.

Buy-Before-Build is the answer, if the off-the-shelf software provides a “good enough” functional, operational and technical fit – and if future upgrades remain feasible and without too much complexity. If it isn’t then the organisation might have bigger problems to tackle (technical debt?) or it’s trying to standardise its competitive advantage – In those cases, build-before-buy might be the better option (at least in the short-term).