Conway’s Law and Technical Debt

Everyone wants to be agile. It is the catch cry of the day, along with cloud computing, big data and payments processing.

Agile software development boils down to do more with less, and communicate more with less. Do more with less is about using fewer but better suited tools to improve productivity. Communicate more with less is about increasing the design discussion and feedback loops without the overhead of maintaining large documents – note the difference between “discussion” and “documenting what we built”, something many projects confuse.

But I’d postulate that all software development methodologies still fundamentally view software development as a single team activity; despite our typical work environment consisting of multiple teams; multiple delivery structures.

Yes, every project manager has the list of external dependencies on other teams and other systems, but it is just a milestone tracking list. It doesn’t help us become more agile. Similarly, the Scaled Agile Framework is more about making the project management office or the enterprise release management teams more agile than addressing the multiple team challenge.

So what’s the problem?

The below diagram illustrates three projects tasks to deliver three solutions. The three teams must leverage existing corporate assets (e.g., an inventory system or general ledger system) to avoid causing trouble, but the trade-off is overlaps between the three solution spaces. Secondly, the trade-off agreement between team “A” and team “B” will in all likelihood impact the possible trade-off agreement between team “B” and team “C”.

Solution overlaps

Conway discussed this problem back in 1968 and framed it as the correlation between organisational structures and software structures. Bass and others have found similar evidence of projects getting into trouble due to “architectural misalignment”; that is the difference between the organisational enabled communication structure versus the communication structure needed to solve complex architectural problems. This “structures correlation” problem is commonly referred to as Conway’s Law, although “structure” does need to be understood more in context of Bass’ design problem discussion than Conway’s original and more literal computer code structure context.

While we probably don’t have a good answer for dealing with Conway’s Law, it struck me that Martin Fowler’s Technical Debt Quadrant is useful for illustrating the four phases of (what I have coined) the Conway’s Law Denial:

  1. Your company’s methodology calls for identification of project interdependencies; dutifully noted as part of the initial planning efforts.
  2. Early design work uncovers overlaps with other projects (hopefully on your list). But we are all polite and work together, as schedules haven’t yet been tightened (too much); aka we send a few emails with draft designs attached.
  3. The true complexity of the “structures correlation” problem arrives and full-blown denials starts
  4. Final stage of full-blown denial

Projects and TD

And why does all of this matter; other than stop annoying colleagues?

Teams disconnected by organisation, location and/or time cannot make appropriate trade-offs and the scene is ripe for piling on the technical debt.

References

Mel Conway, “How do committees invent?“, 1968

Bass, M. et al, “Architectural Misalignment: An experience report“, 2006

Kwan et al., “Conway’s Law Revisited: The Evidence For a Task-based Perspective“, 2012

Why do we have balls of mud?

IT complexity is often described using metaphors such as “the big ball of mud” or the “software hairball syndrome” to illustrate a haphazardly structured software system. What intrigues me about these systems is their stubborn ability to morph into existence and their persistence. Almost all the companies that I’ve worked with over the years have at least one of these systems and many are several systems intertwined. And they are clearly problematic to maintain and change.

While searching for answers, four approaches repeatedly turned up in different disguises:

  • Delegate – typically to the Enterprise Architecture team, so when it doesn’t work, we can blame them (win either way).
  • Let’s move it to the cloud – it is same shape, so it should fit (Mud as a Service?).
  • New tool – where is that mud removing feature?
  • New method – the IT department is incompetent, so they’ll need a bit of extra help.

Leaving the tongue-in-cheek behind, all the approaches seemed to me like going on a diet. Sure, if there is a lot of complexity then we need to remove it. But diets are rarely much fun, which makes it hard to motivate people and easy to stop regardless of achievements. In fact they often make the problem worse, because they don’t stop you from regaining weight. Diets also don’t address the reason behind the excess weight – we need to look at our habits to find (one of) the reasons.

People don’t create complexity on purpose or out of incompetence. Complexity is rarely in people’s own interest. And if we were incompetent then large corporations shouldn’t be able to function; let alone reasonably well. So why do we end up with complexity despite our best efforts?

While I cannot claim to have the answer, I do think the problem is closely related to how an organisation makes design decisions.

Decision Context

Conway observed that the organisation’s communication structure is mirrored in its software systems. That is the organisation’s accumulative set of design decisions agreed through the organisation’s communication structure is what forms the systems. Software architecture is about guiding and constraining design decisions, which architects facilitate through the available communication structures.

But there is obviously more to it than just structure – what (the decision) we communicate are also important.

Tyree and Akerman’s article, “Architecture Decisions: Demystifying Architecture” (PDF) describes the elements (“what”) of an architectural decision well.  Their definition stress the importance of capturing the decision context such as the associated requirements, constraints, related decisions, and rationale (labelled “context D” in below diagram); as well as the consequences (also referred to as implications). We tend to think of consequences as something relative localised, but architectural decisions often have a much wider impact in four different dimensions: project, system, domain, and enterprise. The same decision will have different consequences depending on its context (“C”):

  1. the project; the context for making and implementing decisions
  2. the system; that is the changed system or impacted another system change
  3. the domain; a horizontal aspect such as security, operations, or a single, enterprise wide business function
  4. the enterprise; the synergy of all domains

The third part is decision priority. Some decisions cannot be made before others, while others are closely interlinked. We structure decisions into decision dependency trees (explicitly or implicitly). In a pure architect only world, we’ll expand and prune the decision tree, as we work our way through it while carefully evaluating the available options. In reality, there are many influences on the decision priority including schedule, cost, available information and the organisation itself (labelled as context P).Decision ContextThe above diagram seeks to illustrate the three parts: Priority, Decision, and Consequence with their respective contexts. For smaller teams, context C, D, and P are just “the context”, but this is rarely the case within larger organisations. What is the result of this less than perfect context overlap (between P, D, and C)?

Context Misalignment Creates Complexity

Most architects have experienced the scenario, where project schedule drives the decision priorities. This forces the need to take shortcuts through the decision tree, and we risk making less than ideal decisions, either because we don’t have time to find the required information or have insufficient time to perform proper analysis. Another key factor is the organisational structure itself. Bass and others found that projects run into problems, when the chosen architecture required subsequent communication needs which the organisation structure couldn’t support – the anti-pattern of Conway’s Law if you like. What’s interesting about Bass’ findings is that we give priority to decisions that we can solve with a given organisational structure. We do this rather than focus on changing the organisational structure to support the actual, required decision structure.

Making decisions without consideration for their multi-dimensional impacts will also cause problems. While we tend to blame the project manager for forcing us to consider the schedule, cost and available resources, architectural decisions made without appropriate consideration for all four dimensions also adds to the complexity – for example:

  • A functional focus without a view to operational domain (project vs domain) will add to the system support and maintenance cost
  • Accepting system capabilities without questioning the functionality integrity (system vs domain) will compromise domain level capabilities
  • Driving a strong domain strategy risks compromising core systems and the enterprise itself (domain vs systems vs enterprise), and;
  • Ivory tower syndrome (enterprise vs the rest)

While IT complexity might be one of those impossible problems to solve, I believe that it goes along way to simply acknowledge and attempt manage the decision context.

Is “architecture” the best metaphor?

We often rely on metaphors and analogies to explain software, its structure and function. We cannot see software (except for its user interface), so we use metaphors to illustrate. It is true that we can print out the source code and look at it, but such static representation can mislead as far as the software’s true behaviour (we call these defects).

Metaphors can also mislead. A single metaphor can rarely illustrate all aspects leaving us with an incomplete illustration. Integrating several metaphors is tricky, as it can easily confuse. But without, a single metaphor will be over-interpreted and extended to cover the gap inappropriately. Consider the word ‘architecture’ as a software metaphor.

We use the word ‘architecture’ as a metaphor to illustrate and explain the purpose of ‘software architecture’ (and feel free to replace ‘software’ with ‘enterprise’, solution, technical, etc.). Most people have an intuitive understanding of ‘architecture’ as representing structure, important, and quality. More generally, the word ‘architecture‘ is defined as the “formation or construction of a unifying or coherent form or structure” and can refer to “both the process and the product of planningdesigning, and constructing buildings and other physical structures” (wikipedia).

Software consists of structures including the one found in its source code, although it is not the only structure of concern. Coherence is a desired property of software, that we seek to maintain through the formation and construction of the structure. Without coherence, the software is unlikely to work properly and be maintainable. The building architecture metaphor is frequently used to define ‘software architecture’ and sometimes made explicit – see Clement et al, Spewak, The Open Group, or Perry and Wolf as well as Garlan and Shaw.

There is one problem. Software is not a building.

Philippe Kruchten argues in his paper, “The Nature of Software“, that software has a very low manufacturing costs. In a strict engineering sense, software manufacturing cost is only the cost of creating the distribution media. The act of programming is similar to creating a cast in mechanical engineering.

This changes the purpose of ‘software architecture’ dramatically from ‘building architecture’ – in other words, the metaphor breaks down.

Software architecture is about guiding and constraining subsequent design decisions and not the subsequent construction (aka manufacturing). Architectural documentation, patterns, and styles guide the design process, while architectural decisions such as “we must use Java” constrain subsequent design decisions (as we can no longer use C# and probably not Microsoft application servers). Architectural design is about defining a suitable Design Space to support subsequent design decisions (which in themselves can also be considered an architecture; enterprise vs solution, solution vs technology etc.).

It occurred to me that many aspects of creating a useful architecture relate as much to cartography (map making) as to building architecture. According to wikipedia, the fundamental problems of traditional cartography cover:

  • Map editing: Set the map’s agenda and select traits of the object to be mapped. We must tailor our architecture according to our stakeholder concerns and choose appropriate views.
  • Map projections: Represent the terrain of the mapped object on flat media. We must choose the appropriate software element representations (such as those described in UML or ArchiMate) and how they can be mapped between different views.
  • Generalisation: Eliminate characteristics of the mapped object that are not relevant to the map’s purpose to reduce the complexity of the characteristics that will be mapped. If there ever was a single critical activity for software architects, then this is it – generalisation to reduce complexity.
  • Map design: Orchestrate the elements of the map to best convey its message to its audience. If you cannot communicate your architecture then there is no point.

Maps express structure often with several layers of information. Their intent is to guide a decision process. Cartography works well as a metaphor for software architecture. On the other hand, the (building) architecture metaphor does not encapsulate many of the challenges we face when designing software systems, while implying activities typically associated with IT management methodologies – a consequence of building architecture’s intend to guide construction and not design.

But I guess “software cartographer” does not have the same ring to it – as “software architect” does – even if, as a metaphor, it provides a better representation.

 

The Social Enterprise – what problem are we trying to solve?

Social Computing along with Cloud Computing is one of the hot IT buzz words – i.e., the Social Cloud must then be the ultimate in buzz word compliance. This is in fact what Andrew McAfee from MIT’s Management school and Mike Gotta from Cisco are discussing.

Andrew presents his Enterprise 2.0 the Indian Way in a recent blog post. He describes a project done internally at Tata Consulting Services, where they build a social collaboration tool to rate and share the broad collection of project derived knowledge. It sounds deceptively simple, but on the other hand, I have seen the results from a number of similar projects deploying a very structured, formal approach to knowledge sharing – and none of those worked very well – so why not? The real trick at TCS didn’t seem to be so much about the tool, but what motivated the TCS consultants to engage. You could call it a bottom up approach to the Social Enterprise.

The opposite example is presented by Mike Gotta in his presentation: Build an Architecture of Participation. I have to warn you, it is heavy on models, slides etc. Although he is discussing the same thing, it is probably more of what you’d call a top down approach to the Social Enterprise.
Continue reading “The Social Enterprise – what problem are we trying to solve?”

Enterprise Architecture – the search for a faster horse?

Len Fehskens from The Open Group recently wrote an interesting article for The Open Group blog titled ‘Enterprise Architecture’s Quest for its Identity‘ (and subsequently recycled at ZDNet and SOA World Magazine), where he poses two questions:

  1. Is enterprise architecture primarily about IT or is it about the entire enterprise?
  2. Is enterprise architecture a “hard” discipline or a “soft” discipline?

Len argues in favour of Enterprise Architecture as a soft discipline concerned with the entire enterprise and not just IT. But I would have liked Len to have addressed the ‘why?’ part better – often identity is derived from the purpose of doing it. He makes a few references to ‘descriptions’ of an enterprise. But as Todd Biske pointed out, the relevance of Enterprise Architecture is determined by the whether the Architects of an enterprise are Enterprise Archivist (model focused) or Enterprise Activist (engagement focused). Continue reading “Enterprise Architecture – the search for a faster horse?”

The (real) problem with Cloud Security

A real gap has appeared between how Cloud vendors and their customers perceive security. In a recent survey, that 69% of vendors believe security is primarily a cloud customer responsibility, but only 35 percent of them believe security is their responsibility only. Just 16 percent of cloud providers feel security is a shared responsibility, compared to 33 percent of cloud users.

Although security has repeatedly been highlighted as one of the key concerns with Cloud Computing, only 20 percent of cloud vendors see security as a competitive advantage, and fewer than 27 percent feel their cloud services can protect and secure customer information.

Why is there such a gap? Continue reading “The (real) problem with Cloud Security”

What is Enterprise Architecture and did it really die?

The recent year or so have offered amble blogosphere discussion about what Enterprise Architecture is, whether it died and who to blame. It appears that John Zachman kicked off the debate back in 2009 with his blog post Yes, “Enterprise Architecture is Relative” BUT it is not Arbitrary, where he claims that Enterprise Architecture is dying due to the extensive overuse of ‘architecture’ for everything in IT. This generated a very extensive debate on Linkedin (summarised here) which also extended into what Enterprise Architecture is and what Enterprise Architects do. The debate does seem to get lost in roles and responsibilities, models, and fuzzy terminology.

Rather than offering another summary or yet another random definition, I find it useful to think about an Enterprise, as the combination of an organisation and its ability to use IT – and doing two things to achieve this: planning and delivery. The combination is the above matrix. Organisations use Information Technology in order to achieve an advantage that otherwise wouldn’t be possible – an advantage that might enable the organisation to reduce cost, deliver new products and services or re-design the organisation itself (to create other advantages).

Organisations’ IT deal with:

  • The planning of IT generally consists of two types: 1) a strategy for the IT systems, e.g., technology choice, the roles of each system, etc, and; 2) the management of the portfolio of IT related activities such as projects and operational support.
  • The delivery of IT is the delivery of actual IT systems and the execution of IT projects.

Organisations’ ability to use IT deals how the organisation itself needs to be structured and behave in order to realise the potential benefits offered by a set of available IT:

  • The planning of organisational IT ability covers the identification of appropriate governance models, or organisational capabilities such knowledge management, social architecture or a unique organisational structure.
  • The delivery of organisational IT ability covers what management literature label as organisational change management or transformation.

Now if one then takes the view that Enterprise Architecture is all of the matrix, then EA is far from dead – most organisations would be doing this to some degree.

But if one views architecture as ‘the fundamental organization of a system‘ (as defined by IEEE’s standard for the description of IT systems), then maybe EA is dying because there is so much more to an Enterprise than its fundamental organisation – in other words, EA is dying because the architecture analogy is a poor analogy for today’s Enterprises?