Bespoke Agile

Disclaimer

This book covers speculative and exploratory ideas about Agile and Product Management. With discussion and iteration, some of these ideas may influence project management and agile framework choices - but that is aspirational.

The Agile Alliance and PMI organizations count on members to improve and build on their respective bodies of knowledge. I believe the two groups aim to solve similar challenges. Cross-fertilization will benefit both fields, which is why this is an open science project.

Overview…

Over the years I’ve used phrases such as acyclic graph, and friction to describe properties of feature development, deployment, and support. In hindsight, those terms needed more context. I own that entirely, and a goal of this book is to provide context: ways of thinking about frameworks and process that help develop intuition and practical approaches to getting stuff done. Another goal is to provide mathematical tools using multiway and causal graphs that deliver reasonably precise metrics that all business groups can use for planning. Real-time metrics and well-understood adaptive process can provide a competitive advantage from every perspective. A personal hope is that a study of these graphs will reveal simple rules that underly project complexity.

Systems of the world

This book explores best practices in functional product development. This chapter presents an overview of ideas, vocabulary, and concepts; with a few examples… and is very much a work in progress.

Additional chapters will present case studies, and explore how these concepts can be used to create a system to guide dynamic and adaptive choices in both framework and process. In other words - provide clear paths for individuals and teams to deliver consistently good releases and outcomes, even as the business needs and work environments change over time.

Modeling the processes used by an entire company seems daunting. While there are any number of ways to get from ideation to product delivery, for the most part people are the most complex component. Each business unit within a company along with customer behavior can be viewed as a ‘Production System’. The set of all production systems within a company, along with their lifecycles comprise a ‘System of the world’ from the perspective, or frame of reference of the company as a whole.

Reference frames, Rules, and Meta-project management

Reference frames

The idea that a company itself has an evolving frame of reference is so important that the tagline of this site is ‘adaptive systems of the world.’ Each team has a perspective that is the aggregate of each team-member’s point of view, or frame of reference… Looking at the larger problems as a set of finite smaller perspectives provides the scope to allow meta-analysis of both how people behave, and the processes they really follow. Quantifying individual behavior can be a sensitive topic, but identifying what influences individual choices in the context of process and management technique is functionally useful.

Rules

The term Software Development Life Cycle describes the aggregate systems of the world… As we look at how different companies evolve their life cycles, there are common sets of constraints that conform the choice of frameworks and processes. Individuals and teams choose systems of the world with sets of rules that allow work to get done without having to think about the whole system all the time.

People have an intuitive understanding of which rules to follow. As these rules are identified and explored, conscious choices can be made about process in the context of reference frames. The rules create maps that teams can follow to efficiently choose the right process at the right time.

Meta-analysis (and riverbeds)

In a system of the world for a software company we know what goes in, and we know what comes out. If the system functions well, we know properties of what comes out, such as quality and cost of production. The meta is identifying the constraints that explain how process and behavioral rules change over time. The idea is to consider all possible ‘rules that act on rules’ (meta-rules), and identify the ones that appear to be most useful: If the rules that govern process form a riverbed that both channels water (work done), and is itself changed by the water’s flow; the meta-rules describe how a map of this riverbed changes over time. If businesses identify relevant meta-rules, they can control the speed of flow, and where to strengthen the riverbed and shores.

Maps, computation, pattern matching and substitution

How I learned to stop worrying and love computational irreducibility

Maps are computational shortcuts. They are ways to reduce the amount of work needed to get from point A to point B. A silly example is that without a map while on a journey, you might have to evaluate which direction to head, and how far to travel (angle and length) with every step. This represents a lot of work! Since you may have to detour or backtrack, you probably can’t predict when you will arrive, if at all - until you get there.

The hypothetical example I prefer is a future with self-driving vehicles that receive real-time updates about the road ahead: elevation and turns over the next hundred feet, road condition, speed and location of other vehicles, perhaps as a dynamically updated point cloud… this information reduces the amount of computation required to navigate safely down the road with a minimum of risk. In this context the roadmaps are a shortcut and provide intuition about the best path to follow.

What happens when the information from the road is reduced, or you take your vehicle ‘off the grid’? Well - your self-driving car will take on a greater computational load to create that point cloud from onboard sensors - and very likely slow down. What if road and environmental conditions become more complex? There will be an inflection point where your vehicle’s ability to predict what will happen next at at a specific speed falls below the number of seconds required to respond to typical road hazard. At some level of complexity, that speed limit will approach zero.

A popular example is finding the value of Pi… the only way to find the next digit in Pi is to compute the value. All those digits are ‘there’, but we simply cannot know them any faster than we can compute them…

All of the above are examples of computational irreducibility… and for the first two at least, maps allow us to get some ‘headroom’ in which to be clever about the problems at hand at a level of complexity where they can still be solved.

Pattern matching and substitution

Everyday maps represent topographical data, usually with agreed-upon patterns: icons, legends, or keys… When we look at a map, and see the legend for a river, we know there is a body of water. In our heads we substitute the pattern with the idea of a river. Functionally we need to plan for a crossing, or traveling up or down the river. We take yet another step in pattern matching and substitution by planning “bring a canoe” as a means of conveyance when we are at the river represented by the legend on the map…

Rules as graphs; graphs as maps…

How do these ideas apply to software or more general product development? The classic approach to build a thing is to define requirements, factor them into prioritized epics, with prioritized stories - with a develop <—> deliver continuous delivery strategy… OK - at both a process level, and a product detail level this implicitly uses pattern matching and substitution.

What are these rules: An intuitive look at software development complexity

Think of all the ways we start a software project, do some work, get a product out the door, prove an idea, create customer relationships, and iterate on all the above… An intuitive way to view the set of rules by which these actions evolve is a virtual space filled with graphs that represent all ways this stuff moves from ideation to done. This feels overwhelming because it is: “modeling the processes used by an entire company” is not just complex - at any given time we probably can’t predict exactly what will happen next faster than it actually happens. This is a wonderful example of computational irreducibility.

However, if we instead use Agile distributed collaboration, we can capture useful metrics from the point of view, or reference frame of each person, team, business unit, and of the company (implicitly in the views of board members, perhaps). If everyone takes on a lightweight set of tasks in their day that generate relevant metrics, we can identify a few rules that underly the paths actually taken through this space of all graphs! The maps that emerge from the choice of paths provides shortcuts and intuition that saves enough computation to make conscious choices about how to get stuff done before it happens.

It turns out that a coordinated distributed effort can change the paths from one possible graph to another that follows different rules (we change the riverbed!). Better maps of how to get stuff done emerge from better paths along more efficient graphs. Adapting our resources to keep production systems in balance (develop vs deploy, for example) can reduce computation. A feel for the meta-rules that underly a system of the world begins to satisfy the goal of ‘a run book for the unexpected.’ My claim is that there are a fairly small number of meta-rule patterns (graphs) that describe most product development.

Note: Illustrations will help

The processes and stages used in iterative design, implementation, delivery, and support can be captured by sets, or bundles of related graphs. Units of work being done move from one node to another. Consider a product design team defining a feature, and a dev team implementing and testing that feature as two nodes in a graph. To reflect reality, there are feedback loops between the teams, and so there are two ‘edges’ connecting the nodes: one leading forward and one leading back. That’s it - a simple multigraph… this can be extended to subsequent nodes for merging or aggregating features in code repository branches, iterative build and deploy strategies; and nodes representing comprehensive support and customer behavior.

The develop <—> deploy system can be represented by two graphs whose nodes have a lot of inter-dependancies. Business groups such as Marketing, Finance, IT, and Sales have their own graphs whose nodes are similarly constructed, and nodes from across all the graphs have interactions and dependancies that are relevant.

For completeness - here is some intuition related to thinking about these graphs: There is no going back in time. So the edges that lead back are really connecting an implementation node to a future design node; and onward in time to another implementation node. Nodes can branch (edges to more than one node) and merge. For a feature under development, the nodes all must eventually merge. If they do not - what is accomplished in the earlier nodes is not included in subsequent (in time) nodes that might represent ‘merge to release branch.’ More on this later…

For the dev graph, there are any number of nodes that might take place between design and implementation: the work has to be factored out, tickets created; incremental releases require versioning of features to manage complexity; dev often needs to load balance implementations across teams using concurrent branching strategies; and on top of all of this - effective testing requires essentially a real-time description of expected behavior for each version of a feature in the pipeline… Otherwise what’s the point? You can’t evaluate tests if you don’t know their expected results.

Computation, or ‘the daily grind…’

Another important idea is that each node represents a certain amount of work. For the purposes of measurement - let’s call the work done in a node ‘computation’. Well - we can count the nodes to get from one point in a process to another. If we know the amount of computation needed for each node - we are able to compute how much time a feature will really take to develop. With a few other details such as distance between nodes (how many branches from a shared ancestor), and angle (can it converge to be included in a release) - we might be able to re-order and combine nodes to attain a faster path through the related bundle of graphs.

Are they done yet?

Along with distance and angles - we can create causal graphs that represent all the dependancies in the system. Again - this sounds complex - but compared to the actual work that people have to do - it is the easier problem to solve! A causal graph can help determine the set of nodes that must be completed to achieve a defined outcome. It also helps in choosing which nodes can be pruned to ‘degrade gracefully’ - by which I mean the business consciously chooses the quality and completeness of all code that is deployed, representing the amount of computation available.

On the tools side, these processes are complex, but reasonable to model with sets of directed multiway graphs. For example, the optimal number of teams working on concurrent implementation, or depth of feature branches might be determined by analyzing causally dependent branches in their respective graphs, and within the bundle of dependent graphs.

No boiling the ocean…

An important design goal in constructing these graphs mirrors the approach we take for curating tests cases during development! Start with a clean slate - no graphs other than what can be put together from a few team discussions… and then add nodes to each graph as tickets (units of work) flow through the development graph; add nodes to the design process graph as these same tickets required iteration with designers. As a concrete example of ‘combining nodes’ to increase velocity - have the Dev <—> designer iterations collapsed to one node shared by both the Design and Dev graphs (see the example below).. Updating the graphs, and testing-out process changes becomes another iterative and adaptive daily light-weight task for the program and project managers.

Putting it together…

To sneak in an agile principle - combining nodes between graphs represents a ‘one touch approach.’ Specifically, if ‘reviewed by design’ is added as a definition of done for every relevant ticket in a sprint, developers are then empowered to go to the designer and solve problems during implementation. If a problem can’t be solved, the product manager can call the current ticket a ‘spike’, create new tickets in the backlog, update the road map - and there are no surprises…

If a problem can be solved - versioned test cases are updated right then and there, and are available to any team who touches that feature. Consider the time saved by developers who no longer have to context switch back to complete a feature from a previous sprint to address an insufficiently spec’d design…

The bespoke design goals of transparency often take the form of incrementally updated heads-up displays that can track features and fixes as they move through the ‘pipeline.’ This information can be customized for the reference frames of designers, developers, automators, CI/CD DevOps; as well as marketing, finance, and customer support. Under the hood these displays depend on units of work traversing related nodes in bundles of multiway graphs… but the intuitive story of what is really going on can be customized to satisfy individual frames of reference: for individuals, teams, and executive summaries.

Next steps

This chapter will be expanded and made more readable. Some of the current pictures will be substituted with illustrations. Additional chapters will present case studies and analysis in the context of these ideas. Ideally they will have wolfram notebooks that allow readers to interactively adjust rules to create visual graphs, and see the consequences of different develop <—> deploy paths taken through the graphs to create a system of the world…

Many thanks to Stephen Wolfram’s physics project, which continues to help me intuit the metaphors presented here.

More to come!