Procrustes is Alive: Agile in Practice

Procrustes, the son of Poseidon, enticed weary travelers into his lair with the promise of sustenance and a night of sleep in an extraordinary bed. The bed was purported to be a perfect fit for all who laid in it. Once situated in the bed, Procrustes delivered on his promise by either hammering and stretching or amputating the feet of the victim in order to match their stature to the bed. The end result was always death. Theseus finally ended Procrustes’ reign of terror by forcing him to lay in his own bed. The myth of Procrustes is a perfect analog for modern day software engineering.

Software engineering has long been characterized by capacious trends with increasingly pernicious results. Every year brings a new “silver bullet”¹ that promises to bring order to the ensuing chaos that former ersatz silver bullets have left in their wake. What’s confusing is that these trends provide considerable benefits when applied in the right context and executed with a measure of prudence. However, “hypesters”² completely disregard context and prudence and proselytize dogmatic implementations. It’s as if Procrustes himself has been reincarnated in modern day with a software process in place of his bed. Software professionals fall victim to the trap in droves. Where is Theseus when we need him most?

The Procrustes analogy is a perfect fit for trends across many facets of software engineering such as languages (e.g. Go, Haskell), architecture (e.g. micro-services, data streaming) and tools (e.g. Kubernetes, Spring Boot) to name a few. For the sake of brevity, the focus of this particular piece is agile process methodology. At it’s core, agile processes are amazing and the benefits are difficult to overstate. However, overly-prescriptive implementations that fail to account for the intricacies of specific contexts are absolutely toxic.

Agile at its Core

Once you distill out the never-ending rhetoric surrounding agile process methodology, it’s a painfully simple concept. In 2001 seventeen industry professionals with “strong divergent opinions” (Robert Martin, Clean Agile, location 281) met to discuss the dismal state of software process³. At the time, the software engineering industry was trending toward behemoth processes with the dream of reducing programmers to fungible cogs in a machine. The approach was not without precedent as this is how Frederick Taylor revolutionized industrial manufacturing. In the world of Taylor, innovation and efficiency are a function of process ingenuity. Much to the chagrin of managers everywhere, the same does not hold true for software.

Software engineering was in need of a pragmatic approach for de-emphasizing ineffective software processes and prioritizing the end product⁴. Stated concisely: outcome over methodology. The result was the Agile Manifesto⁵: a concise set of guiding principles that can be read in less than a minute.

Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

What is shown above is all there is; the founders had little to no expectations, and they never met again. It’s curious how an entire industry sprung up around so few words. All the resulting frameworks, certifications, etc. are embellishments (what Robert Martin refers to as “Agile plus something else” (Clean Agile, location 296)). There are many Agile prophets who spend their time interrupting the sacred Agile Manifesto for consumption by the plebeian masses. While this is a necessary service for novices; unfortunately, these interpretations are where Procrustes lies in wait crafting his insidious bed.

Agile in Practice

Although the concept of Agile is straightforward, the manifesto provides no guidance for practical application. Off-the-shelf Agile methodologies are the “double-edged sword” indented to light the way for novices. Systematized frameworks such as Scrum⁶ and SAFe⁷ are an excellent way to introduce Agile concepts to an organization. However, it’s easy to fall into Procrustes’ trap by interrupting guidance as religious dogma.

Frameworks: The Bright Side

As stated previously, the Agile Manifesto provides no guidance for practical application. It requires an expert⁸ level practitioner to effectively execute abstract concepts in a real-world context. Unfortunately, regardless of the industry, experts are surprisingly scarce. Some sources estimate that they comprise between one and five percent of the population (Pragmatic Thinking and Learning, p. 21). It’s reasonable to posit an even smaller concentration in the software field due to its relative youth and rapid rate of growth⁹. How then does one get expert results without experts? The simple answer is, one doesn’t. A codified process is the next best thing.

The Dreyfus Model¹⁰, an abstraction for understanding skill acquisition, is an excellent tool for envisioning this concept. It presents a taxonomy for classifying the skill level of a population. A salient detail is that while it takes time to reach the expert stage, time is not an indicator of expertise. Progressing through the taxonomy is a function of talent and deliberate practice. The taxonomy along with its approximate distributions are abbreviated below.

Abbreviated Dreyfus Model

1. Novices (~20%): Requires explicit instructions and supervision
2. Adv. Beginners (~40%): Ability to operate independently
3. Competent (~30%): Ability to troubleshoot problems
4. Proficient (~7%): Ability to course correct based on feedback
5. Expert (~3%): Ability to work from intuition.

Ideally, Agile frameworks are written by experts and implemented by novice-proficient practitioners in order to compel them along the Dreyfus continuum. They are an educational resource and starting point for those who need explicit instructions. The goal isn’t framework competency: it’s building software process expertise that is applicable in a real-world context.

The key concept here is starting point. Robert Martin states, “In the end, regardless of which [Agile] method you choose, you will wind up tuning and tweaking the process to fit your own needs” (Clean Agile, location 2237). Applied in this capacity, frameworks are indispensable as they have the potential to improve the state of practice in the software industry.

Frameworks: The Dark Side

Procrustes has guilefully commandeered Agile frameworks and fashioned them into his treacherous bed. That is to say, frameworks acquire religious devotees who proselytize despotic adherence. Armies of consultants demanding obscene fees promise to conjure Agile goodness by applying their secret formula regardless of the context. There are even certification courses that teach the wizardly ways. It’s ironic that Agile has become what it meant to usurp: a dogmatic process observed without regard to prudence.

Some original luminaries are expressing consternation with the current state of the industry. For instance, Martin Fowler coined the term Agile Industrial Complex¹¹ as a pejorative appellation for the giant money machine that is modern Agile. Robert Martin goes as far as to say, “The Agile certifications that exist are a complete joke and an utter absurdity” (Clean Agile, location 2353). Many modern processes identifying as Agile are contorted Agile wraiths grotesque enough to challenge the constitution of the most valiant software professional.

Admittedly, there may be a hint of hyperbole in this section but the premise is valid. Fred Brooks, as always, expresses the situation with reasoned tones: “Often the original proponent of a theory or technique understands its promise, its liabilities, and its proper domain more clearly than his later disciples. Less gifted, more fervent, their very fervor leads to rigidity, misapplication, oversimplification” (Design of Design, p. 31). Those two sentences perfectly summarize this entire article.

The pertinent takeaway is that what should be applied as a starting point is being promoted to an axiom. Therein lies the dilemma; frameworks aren’t inherently evil but fashioning a framework into a Procrustean bed has perilous results. This section illuminates the predicament; however, the question remains, why is dogmatic adherence bad?

Software Diversity

The aim of this section is to open the reader’s mind to the rich heterogeneity that is software. “The one-size-fits-all approach to software problems and solutions prevents us from seeing some important truths” (Software Creativity 2.0, location 5166). The truth is that context is the single most important process design consideration. It seems obvious to the point of banality: disparate contexts require different processes¹². Peter Denning goes as far as to equate the notion of generalized computing knowledge to “wishful thinking” (Computational Thinking, p. 8). The very essence of Procrustes’ trap is to hammer a context into a process for which it doesn’t quite fit: the proverbial “tail wagging the dog”.

Contemplate this riddle: what do the following software projects have in common?

• Personal web application created for educational purposes
• Enterprise grade distributed business system for an online retailer
• Embedded firmware for an IOT doorbell
• Embedded system responsible for guiding a nuclear missile
• Accounting system for a company with 25 employees
• Open source social media plug-in with a distributed team

The answer is obvious: not much. This is not to imply that select concepts do not generally apply. For instance, decomposition of modules is likely a good idea in every case; however, the method and extent of decomposition should vary wildly. The world of software development is incredibly broad and diverse. “Your ability to solve a problem for someone depends on your understanding of their context in which the problem exists” (Computational Thinking, p. xiii). There is no process certification program that could possibly prepare a person to operate effectively across the entire field.

To be fair, some frameworks are designed for projects of a specific size. While this is commendable, size is only a minute portion of the menagerie of factors that comprise context. Exigencies include culture, regulatory constraints, criticality, timeline, complexity, workforce skill, domain, political landscape, profitability, and the list goes on. Rest assured, none of these factors are considered by canned processes.

To add to the conundrum, many processes considerations are fluid which implies that the ideal process is likely to change over a product’s life cycle. Fred Brooks astutely observes that, “product processes are, properly, designed for follow-on products. For innovation, one must step outside of process” (The Design of Design, p. 236). The goals that exist during the initial phases of a consumer product (time to market, innovation) change as the product matures (reliability, profitability)¹³. Shifting goals necessitate shifting processes. This fact confounds the notion of a silver bullet.

Slay Procrustes

Just as Theseus slayed Procrustes by forcing him to lie in his own bed, dogma is neutralized by holding it to its own standard. A single example of failure, which one can hopefully envision, dispels the myth of infallibility. With fantastical notions out of the way, one can set about to the important work of applying generalized concepts.

Agile frameworks are a great starting point for obtaining the expertise required to translate principles into practice. However, as experts know, the only purpose of a processes is to facilitate the creation of a product; therefore, the needs of the business always trump rote adherence. An effective process is perfectly crafted to fit a context and flexible enough to respond to shifting demands.

For readers dedicated to reaching expert status on the Dreyfus taxonomy, the following reading, listed in order of importance, is recommended.

• The Design of Design: Essays from a Computer Scientist by Frederick P. Brooks
• Balancing Agility and Discipline: A Guide for the Perplexed by Barry Boehm and Richard Turner
• An Elegant Puzzle: Systems of Engineering Management by Will Larson
• The Mythical Man-Month: Essays on Software Engineering by Frederick P. Brooks

TL;DR

For time constrained readers, here is a summary of the main points:

• Agile was intended to be a pragmatic approach for de-emphasizing ineffective software processes and prioritizing the end product
• Dogmatic adherence to practice reduces Agile to what it was meant to usurp
• At its core, Agile is a concise set of guiding principles that can be read in less than a minute
• An expert is required to derive actionable practices from Agile principles
• Ideally, Agile frameworks are created by experts to provide novices a starting point for learning Agile concepts
• “Hypesters”, along with their victims, interrupt frameworks as dogmatic law
• Agile framework competency alone does not translate to competency in a real-world context
• An effective process is perfectly crafted to fit a context and flexible enough to respond to shifting demands.

Bibliography

Brooks Jr., Frederick P.. (1995) The Mythical Man-Month . Pearson Education. Kindle Edition.

Brooks, Frederick P.. (2010) The Design of Design: Essays from a Computer Scientist. Pearson Education. Kindle Edition.

Denning, Peter J.. Computational Thinking (MIT Press Essential Knowledge series) The MIT Press. Kindle Edition.

Glass, Robert L.. (2010) Facts and Fallacies of Software Engineering. Pearson Education. Kindle Edition.

Glass, Robert L.. (2006) Software Creativity 2.0 . developer.* Books. Kindle Edition.

Hunt, Andy. (2008) Pragmatic Thinking and Learning: Refactor Your Wetware (Pragmatic Programmers) Pragmatic Bookshelf. Kindle Edition.

martinfowler.com. (2020) The State of Agile Software in 2018. [online] Available at: https://martinfowler.com/articles/agile-aus-2018.html [Accessed 19 Jan. 2020].

Martin, Robert C. (2019) Clean Agile (Robert C. Martin Series) . Pearson Education. Kindle Edition.

Martin, R. (2014). Clean Coder Blog. [online] Blog.cleancoder.com. Available at: https://blog.cleancoder.com/uncle-bob/2014/06/20/MyLawn.html [Accessed 19 Jan. 2020].