Bespoke Agile

Bespoke Agile lives at the intersection of Project Management and Agile.

‘Agile transformation’ is a highly overloaded term, and its meaning has become diluted. Around 2015 the Agile Alliance began updating their documentation to focus on more sustainable models. This is a work in progress and it takes far too long for critical improvements to see more general use. Engineering and company cultures change slowly. A customized agile approach might provide a vehicle to more quickly validate and socialize these improvements.

In 2021, the Project Management Institute also changed their approach to focus on three new domains: People, Process, and Business Environment. This reflects the need of Project Management Professionals to use ‘more diverse skills and approaches than ever before.’ I took and passed the pilot test for the new PMP certification; and actively support the PMI’s commitment to updating the project management body of knowledge.

Bespoke identifies custom patterns that extend both Agile and Project Management concepts; with a focus on individual perspectives: what is the impact of organizational change on each and every company employee over time? How do we hone in on the realistic tasks people are willing to practice each day to synchronize and marshal products from ideation onward?

Everyone in a company has a frame of reference through which they view change; and so how do we develop models that lead towards company-wide adoption of adaptive process that considers every person’s point of view?

To co-opt a quote by E. O. Wilson “No administrative phenomenon can be fully understood without attention to it’s evolutionary history.” This is from ‘The Superorganism: The Beauty, Elegance, and Strangeness of Insect Societies’: I hope the humor is not lost on anyone - and yes, the quote actually reads “No biological phenomenon…”

Problem statements

Problem: The solutions we care about are large in scope

Enterprise projects are complex, and by far the most complicated is the human side. An agile approach facilitates points of contact - but it does very little to provide actionable collaboration driven by the respective department heads. Yes, there are ‘scrums of scrums’, and other attempts at hierarchical solutions - but this runs into ‘computational irreducibility’: there just isn’t enough time for the relevant problems to be surfaced and solved in this way.

Problem: The perspectives from all departments must be taken into consideration

The interactions between like knowledge domains such as Development and Quality Assurance are complex enough; in addition, departments such as customer Support, finance, Sales, and IT are often under-represented within the teams, and so parallel and inefficient process evolves to address their concerns…

Problem: Ownership of solutions is difficult to craft

Implementing the intra- and inter-department adaptive processes that CAN lead to predictable development and deployment is simply more complex than most people can address without going beyond the bounds of their day-to-day responsibilities. There is often disruptive cognitive dissonance when real people grapple with requests from other teams to do extra work, or believe the work at hand is above their pay grade.

Problem: Unexpected events can be impactful to the business

There are serious business consequences if the impact of a problem is not recognized, or once recognized, there is no run-book available to help craft solutions. Events such as stop-ship bugs, or drop-everything-and-fix-a-problem-in-production defects can roller-coaster over the best of develop and deploy expectations, and certainly derail delivery of new feature milestones.

There are upstream and downstream impacts when reported problems require an unexpected amount of development and test time. Upstream - executives may need to adjust financial projections. Downstream - Marketing needs to be told their favorite new feature may not be ready for a scheduled advertising spend.

Customers may require immediate assistance to prevent erosion of brand loyalty. Worse - if the problem relates to security or data leakage - the viability of the company might be at risk.

Design Goals

Goal: Allow specialization within teams, and require greater personal responsibility in completing distributed clerical tasks

A primary design goal of bespoke-style program management is simple: have everyone opt-in to doing just a little bit of extra work in addition to their career focus. Implicitly this assumes that most developers really like to develop, many people with ‘quality’ in their names may not love to code (yes - QA is dead, long live QA): for example, if those who make code changes opt-in to curate and version manual test cases directly related to each pull request for a feature - the world is a better place for quality-focused team members. QA can ask ‘How do we optimize automation in our feature-branch and release testing?’ instead of ‘Why do I have write a bug because this widget is not implemented to product and design team specs, or the coder did little exploratory testing?”

Goal: A single source of truth (in progress)

There are several concepts combined to deliver ‘truthful data.’ The first is that variables are immutable at any point in time. As time moves forward and values change - we can look for rules related to change. From the variable’s perspective, local rewrite rules (closures or lambdas) move it from one state to the next. This idea applies to information in sets and graphs. This concept is related to mutation and redux in functional programming models.

The second concept is that there is one canonical instance of the variable that is rewritten, and all other instance of the variable are just that - immutable instances embedded at a point in time, updated as needed from the canonical instance. This idea also applies to information in sets and graphs. This concept is related to consistency in distributed database models.

The third relates to scope and namespaces. Variables do not stand alone - they take on life and meaning in time and context. They have their own perspectives, indelibly related to the filters used to create reference frames. We generally use names to identify variables, but it’s useful to reuse those names, and copy variables. With more than one variable with the same name, scope is the region of local process space where a name is available. Namespace is how we ‘look up’ the name associate with a unique variable in the region. This concept is related to maps: I live in Cambridge Massachusetts, USA and not in Cambridge, UK…

Goal: Transparency

This covers the agile initiative to have all information available to everyone at any time. Transparency exists in the context of reference frames. A reference frame is a filtered view through which the exact set of information needed for a specific perspective is available. The promise is that all information is available to all frames of references, as close to real-time as possible. For example, anyone who looks at the value of a variable, set, or graph is guaranteed to see the same values in any instance at the same point in time and scope.

Goal: Degrade gracefully

Everyone has been in the situation where there’s more work to be done than can realistically be met with the resources available. The bespoke approach recognizes this is always the reality, and the focus is on best practices to plan collaborative projects. The relevant goals can be stated as:

- Mitigating areas of high risk with incremental and small changes to both deployed products, and the processes themselves used to develop and deploy the products

- Empower product management to adjust the scope of deployable features to keep schedules in balance with available resources

- Create effective plans for unexpected events and practice them: have a process run-book for the unexpected…

Ideally teams can change the scope and focus of current tickets on demand. Historically scrum teams must complete a sprint, bound to those tickets sized and accepted… but this approach can lead to poor outcomes.

To accommodate these business realities, an adaptive approach to sprints allows current work to be put aside, backlog items reprioritized, fires tended to, and then a smooth recovery to the previous stories… The normal flow is interrupted by design for the time needed, and not ad-hoc with all the mayhem that usually entails.

This approach is often labelled ‘ScrumBan’, as a melding of strict Scrum rules for sprints and Kanban, an approach focused on serialized ticket completion. For both, sprint boundaries are useful for metrics and to track story completion. For those who who view deviation from sprint rules heretical: consider the model of a 3-day sprint, with day-1 being a spike; and days 2-3 being reserved for clearing all active tickets above the backlog… rinse and repeat.

In fact, degrading gracefully can be a sustainable approach. It’s a catchy phrase - but the more accurate description is ‘adaptive development.’

Development and Test

Here are a few ideas that demonstrate a custom approach to development and test. From a bespoke agile perspective - a development team is responsible for merging code into a target branch at the right quality and the right time. This includes validating design with product management, curating manual and automated test cases, supporting continuous integration initiatives (CI), and orchestrating delivery of related tickets that might be under development by different teams… All of this before an epic that represents some incremental feature change is updated to ‘done.’ Here are several details:

- If every team opts into using versioned feature branches…

- And curates the set of collaborative tests that define each element of the feature prior to a merge; adding, deprecating, or removing relevant tests tagged with feature branch versions

- And child feature branches are rebased to versioned parent branches for every pull request

- … a certain amount of magic happens!

Notes:

• This is not an invitation to a culture war on gitflow strategies - I believe these generic patterns apply to any gitflow…

• Versioned builds can be converted to a release candidate and pushed to production with well understood tests that pass..

• The most important tests are not unit tests, but ones that can be run manually. These can be decomposed to a set of smaller test cases that can become the basis for automation (more later)

What happens if you have no tests? It’s a brand-new product or feature? Well that’s addressed by an ‘incremental’ approach..

• Each scrum team must add a test to the test management system only for the current code being written! No boiling the ocean. It won’t take long for a comprehensive set of tests to get developed

• When developers can clone a set of manual tests to run before submitting pull requests, they are doing more of their own QA with less thinking (most developers like to write walls of code, and not do exploratory testing)

Back to Automation:

• The set of tests that CHANGE THE LEAST can be converted to automation…

• the ‘sets of smaller test cases’ that automation consumes can be run to create versioned initial conditions for both manual and automated tests…

• These versioned initial conditions for automation can be ‘inherited’ by every team, and run as regression on those feature branch before merging to whatever equivalent of release candidate branch is in use…

These are small, incremental, and achievable tasks that allow each team to increase their velocity. The feedback loops where team members suddenly realize they have more time to do what they love is relatively speedy - usually from three to six months. Consider this time frame compared to putting out fires, and asking your team member to context switch to solve preventable problems.

Disclaimer

This book covers speculative and exploratory ideas about Agile and Product Management. With discussion and iteration, some of these ideas may influence project management and agile framework choices - but that is aspirational.

The Agile Alliance and PMI organizations count on members to improve and build on their respective bodies of knowledge. I believe the two groups aim to solve similar challenges. Cross-fertilization will benefit both fields, which is why this is an open science project.

Overview…

Over the years I’ve used phrases such as acyclic graph, and friction to describe properties of feature development, deployment, and support. In hindsight, those terms needed more context. I own that entirely, and a goal of this book is to provide context: ways of thinking about frameworks and process that help develop intuition and practical approaches to getting stuff done. Another goal is to provide mathematical tools using multiway and causal graphs that deliver reasonably precise metrics that all business groups can use for planning. Real-time metrics and well-understood adaptive process can provide a competitive advantage from every perspective. A personal hope is that a study of these graphs will reveal simple rules that underly project complexity.

Systems of the world

This book explores best practices in functional product development. This chapter presents an overview of ideas, vocabulary, and concepts; with a few examples… and is very much a work in progress.

Additional chapters will present case studies, and explore how these concepts can be used to create a system to guide dynamic and adaptive choices in both framework and process. In other words - provide clear paths for individuals and teams to deliver consistently good releases and outcomes, even as the business needs and work environments change over time.

Modeling the processes used by an entire company seems daunting. While there are any number of ways to get from ideation to product delivery, for the most part people are the most complex component. Each business unit within a company along with customer behavior can be viewed as a ‘Production System’. The set of all production systems within a company, along with their lifecycles comprise a ‘System of the world’ from the perspective, or frame of reference of the company as a whole.

Reference frames, Rules, and Meta-project management

Reference frames

The idea that a company itself has an evolving frame of reference is so important that the tagline of this site is ‘adaptive systems of the world.’ Each team has a perspective that is the aggregate of each team-member’s point of view, or frame of reference… Looking at the larger problems as a set of finite smaller perspectives provides the scope to allow meta-analysis of both how people behave, and the processes they really follow. Quantifying individual behavior can be a sensitive topic, but identifying what influences individual choices in the context of process and management technique is functionally useful.

Rules

The term Software Development Life Cycle describes the aggregate systems of the world… As we look at how different companies evolve their life cycles, there are common sets of constraints that conform the choice of frameworks and processes. Individuals and teams choose systems of the world with sets of rules that allow work to get done without having to think about the whole system all the time.

People have an intuitive understanding of which rules to follow. As these rules are identified and explored, conscious choices can be made about process in the context of reference frames. The rules create maps that teams can follow to efficiently choose the right process at the right time.

Meta-analysis (and riverbeds)

In a system of the world for a software company we know what goes in, and we know what comes out. If the system functions well, we know properties of what comes out, such as quality and cost of production. The meta is identifying the constraints that explain how process and behavioral rules change over time. The idea is to consider all possible ‘rules that act on rules’ (meta-rules), and identify the ones that appear to be most useful: If the rules that govern process form a riverbed that both channels water (work done), and is itself changed by the water’s flow; the meta-rules describe how a map of this riverbed changes over time. If businesses identify relevant meta-rules, they can control the speed of flow, and where to strengthen the riverbed and shores.

Maps, computation, pattern matching and substitution

How I learned to stop worrying and love computational irreducibility

Maps are computational shortcuts. They are ways to reduce the amount of work needed to get from point A to point B. A silly example is that without a map while on a journey, you might have to evaluate which direction to head, and how far to travel (angle and length) with every step. This represents a lot of work! Since you may have to detour or backtrack, you probably can’t predict when you will arrive, if at all - until you get there.

The hypothetical example I prefer is a future with self-driving vehicles that receive real-time updates about the road ahead: elevation and turns over the next hundred feet, road condition, speed and location of other vehicles, perhaps as a dynamically updated point cloud… this information reduces the amount of computation required to navigate safely down the road with a minimum of risk. In this context the roadmaps are a shortcut and provide intuition about the best path to follow.

What happens when the information from the road is reduced, or you take your vehicle ‘off the grid’? Well - your self-driving car will take on a greater computational load to create that point cloud from onboard sensors - and very likely slow down. What if road and environmental conditions become more complex? There will be an inflection point where your vehicle’s ability to predict what will happen next at at a specific speed falls below the number of seconds required to respond to typical road hazard. At some level of complexity, that speed limit will approach zero.

A popular example is finding the value of Pi… the only way to find the next digit in Pi is to compute the value. All those digits are ‘there’, but we simply cannot know them any faster than we can compute them…

All of the above are examples of computational irreducibility… and for the first two at least, maps allow us to get some ‘headroom’ in which to be clever about the problems at hand at a level of complexity where they can still be solved.

Pattern matching and substitution

Everyday maps represent topographical data, usually with agreed-upon patterns: icons, legends, or keys… When we look at a map, and see the legend for a river, we know there is a body of water. In our heads we substitute the pattern with the idea of a river. Functionally we need to plan for a crossing, or traveling up or down the river. We take yet another step in pattern matching and substitution by planning “bring a canoe” as a means of conveyance when we are at the river represented by the legend on the map…

Rules as graphs; graphs as maps…

How do these ideas apply to software or more general product development? The classic approach to build a thing is to define requirements, factor them into prioritized epics, with prioritized stories - with a develop <—> deliver continuous delivery strategy… OK - at both a process level, and a product detail level this implicitly uses pattern matching and substitution.

What are these rules: An intuitive look at software development complexity

Think of all the ways we start a software project, do some work, get a product out the door, prove an idea, create customer relationships, and iterate on all the above… An intuitive way to view the set of rules by which these actions evolve is a virtual space filled with graphs that represent all ways this stuff moves from ideation to done. This feels overwhelming because it is: “modeling the processes used by an entire company” is not just complex - at any given time we probably can’t predict exactly what will happen next faster than it actually happens. This is a wonderful example of computational irreducibility.

However, if we instead use Agile distributed collaboration, we can capture useful metrics from the point of view, or reference frame of each person, team, business unit, and of the company (implicitly in the views of board members, perhaps). If everyone takes on a lightweight set of tasks in their day that generate relevant metrics, we can identify a few rules that underly the paths actually taken through this space of all graphs! The maps that emerge from the choice of paths provides shortcuts and intuition that saves enough computation to make conscious choices about how to get stuff done before it happens.

It turns out that a coordinated distributed effort can change the paths from one possible graph to another that follows different rules (we change the riverbed!). Better maps of how to get stuff done emerge from better paths along more efficient graphs. Adapting our resources to keep production systems in balance (develop vs deploy, for example) can reduce computation. A feel for the meta-rules that underly a system of the world begins to satisfy the goal of ‘a run book for the unexpected.’ My claim is that there are a fairly small number of meta-rule patterns (graphs) that describe most product development.

Note: Illustrations will help

The processes and stages used in iterative design, implementation, delivery, and support can be captured by sets, or bundles of related graphs. Units of work being done move from one node to another. Consider a product design team defining a feature, and a dev team implementing and testing that feature as two nodes in a graph. To reflect reality, there are feedback loops between the teams, and so there are two ‘edges’ connecting the nodes: one leading forward and one leading back. That’s it - a simple multigraph… this can be extended to subsequent nodes for merging or aggregating features in code repository branches, iterative build and deploy strategies; and nodes representing comprehensive support and customer behavior.

The develop <—> deploy system can be represented by two graphs whose nodes have a lot of inter-dependancies. Business groups such as Marketing, Finance, IT, and Sales have their own graphs whose nodes are similarly constructed, and nodes from across all the graphs have interactions and dependancies that are relevant.

For completeness - here is some intuition related to thinking about these graphs: There is no going back in time. So the edges that lead back are really connecting an implementation node to a future design node; and onward in time to another implementation node. Nodes can branch (edges to more than one node) and merge. For a feature under development, the nodes all must eventually merge. If they do not - what is accomplished in the earlier nodes is not included in subsequent (in time) nodes that might represent ‘merge to release branch.’ More on this later…

For the dev graph, there are any number of nodes that might take place between design and implementation: the work has to be factored out, tickets created; incremental releases require versioning of features to manage complexity; dev often needs to load balance implementations across teams using concurrent branching strategies; and on top of all of this - effective testing requires essentially a real-time description of expected behavior for each version of a feature in the pipeline… Otherwise what’s the point? You can’t evaluate tests if you don’t know their expected results.

Computation, or ‘the daily grind…’

Another important idea is that each node represents a certain amount of work. For the purposes of measurement - let’s call the work done in a node ‘computation’. Well - we can count the nodes to get from one point in a process to another. If we know the amount of computation needed for each node - we are able to compute how much time a feature will really take to develop. With a few other details such as distance between nodes (how many branches from a shared ancestor), and angle (can it converge to be included in a release) - we might be able to re-order and combine nodes to attain a faster path through the related bundle of graphs.

Are they done yet?

Along with distance and angles - we can create causal graphs that represent all the dependancies in the system. Again - this sounds complex - but compared to the actual work that people have to do - it is the easier problem to solve! A causal graph can help determine the set of nodes that must be completed to achieve a defined outcome. It also helps in choosing which nodes can be pruned to ‘degrade gracefully’ - by which I mean the business consciously chooses the quality and completeness of all code that is deployed, representing the amount of computation available.

On the tools side, these processes are complex, but reasonable to model with sets of directed multiway graphs. For example, the optimal number of teams working on concurrent implementation, or depth of feature branches might be determined by analyzing causally dependent branches in their respective graphs, and within the bundle of dependent graphs.

No boiling the ocean…

An important design goal in constructing these graphs mirrors the approach we take for curating tests cases during development! Start with a clean slate - no graphs other than what can be put together from a few team discussions… and then add nodes to each graph as tickets (units of work) flow through the development graph; add nodes to the design process graph as these same tickets required iteration with designers. As a concrete example of ‘combining nodes’ to increase velocity - have the Dev <—> designer iterations collapsed to one node shared by both the Design and Dev graphs (see the example below).. Updating the graphs, and testing-out process changes becomes another iterative and adaptive daily light-weight task for the program and project managers.

Putting it together…

To sneak in an agile principle - combining nodes between graphs represents a ‘one touch approach.’ Specifically, if ‘reviewed by design’ is added as a definition of done for every relevant ticket in a sprint, developers are then empowered to go to the designer and solve problems during implementation. If a problem can’t be solved, the product manager can call the current ticket a ‘spike’, create new tickets in the backlog, update the road map - and there are no surprises…

If a problem can be solved - versioned test cases are updated right then and there, and are available to any team who touches that feature. Consider the time saved by developers who no longer have to context switch back to complete a feature from a previous sprint to address an insufficiently spec’d design…

The bespoke design goals of transparency often take the form of incrementally updated heads-up displays that can track features and fixes as they move through the ‘pipeline.’ This information can be customized for the reference frames of designers, developers, automators, CI/CD DevOps; as well as marketing, finance, and customer support. Under the hood these displays depend on units of work traversing related nodes in bundles of multiway graphs… but the intuitive story of what is really going on can be customized to satisfy individual frames of reference: for individuals, teams, and executive summaries.

Next steps

This chapter will be expanded and made more readable. Some of the current pictures will be substituted with illustrations. Additional chapters will present case studies and analysis in the context of these ideas. Ideally they will have wolfram notebooks that allow readers to interactively adjust rules to create visual graphs, and see the consequences of different develop <—> deploy paths taken through the graphs to create a system of the world…

Many thanks to Stephen Wolfram’s physics project, which continues to help me intuit the metaphors presented here.

More to come!