Procrustes is Alive: Agile in Practice

Procrustes, the son of Poseidon, enticed weary travelers into his lair with the promise of sustenance and a night of sleep in an extraordinary bed. The bed was purported to be a perfect fit for all who laid in it. Once situated in the bed, Procrustes delivered on his promise by either hammering and stretching or amputating the feet of the victim in order to match their stature to the bed. The end result was always death. Theseus finally ended Procrustes’ reign of terror by forcing him to lay in his own bed. The myth of Procrustes is a perfect analog for modern day software engineering.

Software engineering has long been characterized by capacious trends with increasingly pernicious results. Every year brings a new “silver bullet”1 that promises to bring order to the ensuing chaos that former ersatz silver bullets have left in their wake. What’s confusing is that these trends provide considerable benefits when applied in the right context and executed with a measure of prudence. However, “hypesters”2 completely disregard context and prudence and proselytize dogmatic implementations. It’s as if Procrustes himself has been reincarnated in modern day with a software process in place of his bed. Software professionals fall victim to the trap in droves. Where is Theseus when we need him most?

The Procrustes analogy is a perfect fit for trends across many facets of software engineering such as languages (e.g. Go, Haskell), architecture (e.g. micro-services, data streaming) and tools (e.g. Kubernetes, Spring Boot) to name a few. For the sake of brevity, the focus of this particular piece is agile process methodology. At it’s core, agile processes are amazing and the benefits are difficult to overstate. However, overly-prescriptive implementations that fail to account for the intricacies of specific contexts are absolutely toxic.

Agile at its Core

Once you distill out the never-ending rhetoric surrounding agile process methodology, it’s a painfully simple concept. In 2001 seventeen industry professionals with “strong divergent opinions” (Robert Martin, Clean Agile, location 281) met to discuss the dismal state of software process3. At the time, the software engineering industry was trending toward behemoth processes with the dream of reducing programmers to fungible cogs in a machine. The approach was not without precedent as this is how Frederick Taylor revolutionized industrial manufacturing. In the world of Taylor, innovation and efficiency are a function of process ingenuity. Much to the chagrin of managers everywhere, the same does not hold true for software.

Software engineering was in need of a pragmatic approach for de-emphasizing ineffective software processes and prioritizing the end product4. Stated concisely: outcome over methodology. The result was the Agile Manifesto5: a concise set of guiding principles that can be read in less than a minute.

Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

What is shown above is all there is; the founders had little to no expectations, and they never met again. It’s curious how an entire industry sprung up around so few words. All the resulting frameworks, certifications, etc. are embellishments (what Robert Martin refers to as “Agile plus something else” (Clean Agile, location 296)). There are many Agile prophets who spend their time interrupting the sacred Agile Manifesto for consumption by the plebeian masses. While this is a necessary service for novices; unfortunately, these interpretations are where Procrustes lies in wait crafting his insidious bed.

Agile in Practice

Although the concept of Agile is straightforward, the manifesto provides no guidance for practical application. Off-the-shelf Agile methodologies are the “double-edged sword” indented to light the way for novices. Systematized frameworks such as Scrum6 and SAFe7 are an excellent way to introduce Agile concepts to an organization. However, it’s easy to fall into Procrustes’ trap by interrupting guidance as religious dogma.

Frameworks: The Bright Side

As stated previously, the Agile Manifesto provides no guidance for practical application. It requires an expert8 level practitioner to effectively execute abstract concepts in a real-world context. Unfortunately, regardless of the industry, experts are surprisingly scarce. Some sources estimate that they comprise between one and five percent of the population (Pragmatic Thinking and Learning, p. 21). It’s reasonable to posit an even smaller concentration in the software field due to its relative youth and rapid rate of growth9. How then does one get expert results without experts? The simple answer is, one doesn’t. A codified process is the next best thing.

The Dreyfus Model10, an abstraction for understanding skill acquisition, is an excellent tool for envisioning this concept. It presents a taxonomy for classifying the skill level of a population. A salient detail is that while it takes time to reach the expert stage, time is not an indicator of expertise. Progressing through the taxonomy is a function of talent and deliberate practice. The taxonomy along with its approximate distributions are abbreviated below.

Abbreviated Dreyfus Model

  1. Novices (~20%): Requires explicit instructions and supervision
  2. Adv. Beginners (~40%): Ability to operate independently
  3. Competent (~30%): Ability to troubleshoot problems
  4. Proficient (~7%): Ability to course correct based on feedback
  5. Expert (~3%): Ability to work from intuition.

Ideally, Agile frameworks are written by experts and implemented by novice-proficient practitioners in order to compel them along the Dreyfus continuum. They are an educational resource and starting point for those who need explicit instructions. The goal isn’t framework competency: it’s building software process expertise that is applicable in a real-world context.

The key concept here is starting point. Robert Martin states, “In the end, regardless of which [Agile] method you choose, you will wind up tuning and tweaking the process to fit your own needs” (Clean Agile, location 2237). Applied in this capacity, frameworks are indispensable as they have the potential to improve the state of practice in the software industry.

Frameworks: The Dark Side

Procrustes has guilefully commandeered Agile frameworks and fashioned them into his treacherous bed. That is to say, frameworks acquire religious devotees who proselytize despotic adherence. Armies of consultants demanding obscene fees promise to conjure Agile goodness by applying their secret formula regardless of the context. There are even certification courses that teach the wizardly ways. It’s ironic that Agile has become what it meant to usurp: a dogmatic process observed without regard to prudence.

Some original luminaries are expressing consternation with the current state of the industry. For instance, Martin Fowler coined the term Agile Industrial Complex11 as a pejorative appellation for the giant money machine that is modern Agile. Robert Martin goes as far as to say, “The Agile certifications that exist are a complete joke and an utter absurdity” (Clean Agile, location 2353). Many modern processes identifying as Agile are contorted Agile wraiths grotesque enough to challenge the constitution of the most valiant software professional.

Admittedly, there may be a hint of hyperbole in this section but the premise is valid. Fred Brooks, as always, expresses the situation with reasoned tones: “Often the original proponent of a theory or technique understands its promise, its liabilities, and its proper domain more clearly than his later disciples. Less gifted, more fervent, their very fervor leads to rigidity, misapplication, oversimplification” (Design of Design, p. 31). Those two sentences perfectly summarize this entire article.

The pertinent takeaway is that what should be applied as a starting point is being promoted to an axiom. Therein lies the dilemma; frameworks aren’t inherently evil but fashioning a framework into a Procrustean bed has perilous results. This section illuminates the predicament; however, the question remains, why is dogmatic adherence bad?

Software Diversity

The aim of this section is to open the reader’s mind to the rich heterogeneity that is software. “The one-size-fits-all approach to software problems and solutions prevents us from seeing some important truths” (Software Creativity 2.0, location 5166). The truth is that context is the single most important process design consideration. It seems obvious to the point of banality: disparate contexts require different processes12. Peter Denning goes as far as to equate the notion of generalized computing knowledge to “wishful thinking” (Computational Thinking, p. 8). The very essence of Procrustes’ trap is to hammer a context into a process for which it doesn’t quite fit: the proverbial “tail wagging the dog”.

Contemplate this riddle: what do the following software projects have in common?

  • Personal web application created for educational purposes
  • Enterprise grade distributed business system for an online retailer
  • Embedded firmware for an IOT doorbell
  • Embedded system responsible for guiding a nuclear missile
  • Accounting system for a company with 25 employees
  • Open source social media plug-in with a distributed team

The answer is obvious: not much. This is not to imply that select concepts do not generally apply. For instance, decomposition of modules is likely a good idea in every case; however, the method and extent of decomposition should vary wildly. The world of software development is incredibly broad and diverse. “Your ability to solve a problem for someone depends on your understanding of their context in which the problem exists” (Computational Thinking, p. xiii). There is no process certification program that could possibly prepare a person to operate effectively across the entire field.

To be fair, some frameworks are designed for projects of a specific size. While this is commendable, size is only a minute portion of the menagerie of factors that comprise context. Exigencies include culture, regulatory constraints, criticality, timeline, complexity, workforce skill, domain, political landscape, profitability, and the list goes on. Rest assured, none of these factors are considered by canned processes.

To add to the conundrum, many processes considerations are fluid which implies that the ideal process is likely to change over a product’s life cycle. Fred Brooks astutely observes that, “product processes are, properly, designed for follow-on products. For innovation, one must step outside of process” (The Design of Design, p. 236). The goals that exist during the initial phases of a consumer product (time to market, innovation) change as the product matures (reliability, profitability)13. Shifting goals necessitate shifting processes. This fact confounds the notion of a silver bullet.

Slay Procrustes

Just as Theseus slayed Procrustes by forcing him to lie in his own bed, dogma is neutralized by holding it to its own standard. A single example of failure, which one can hopefully envision, dispels the myth of infallibility. With fantastical notions out of the way, one can set about to the important work of applying generalized concepts.

Agile frameworks are a great starting point for obtaining the expertise required to translate principles into practice. However, as experts know, the only purpose of a processes is to facilitate the creation of a product; therefore, the needs of the business always trump rote adherence. An effective process is perfectly crafted to fit a context and flexible enough to respond to shifting demands.

For readers dedicated to reaching expert status on the Dreyfus taxonomy, the following reading, listed in order of importance, is recommended.

  • The Design of Design: Essays from a Computer Scientist by Frederick P. Brooks
  • Balancing Agility and Discipline: A Guide for the Perplexed by Barry Boehm and Richard Turner
  • An Elegant Puzzle: Systems of Engineering Management by Will Larson
  • The Mythical Man-Month: Essays on Software Engineering by Frederick P. Brooks

TL;DR

For time constrained readers, here is a summary of the main points:

  • Agile was intended to be a pragmatic approach for de-emphasizing ineffective software processes and prioritizing the end product
  • Dogmatic adherence to practice reduces Agile to what it was meant to usurp
  • At its core, Agile is a concise set of guiding principles that can be read in less than a minute
  • An expert is required to derive actionable practices from Agile principles
  • Ideally, Agile frameworks are created by experts to provide novices a starting point for learning Agile concepts
  • “Hypesters”, along with their victims, interrupt frameworks as dogmatic law
  • Agile framework competency alone does not translate to competency in a real-world context
  • An effective process is perfectly crafted to fit a context and flexible enough to respond to shifting demands.

Bibliography

Brooks Jr., Frederick P.. (1995) The Mythical Man-Month . Pearson Education. Kindle Edition.

Brooks, Frederick P.. (2010) The Design of Design: Essays from a Computer Scientist. Pearson Education. Kindle Edition.

Denning, Peter J.. Computational Thinking (MIT Press Essential Knowledge series) The MIT Press. Kindle Edition.

Glass, Robert L.. (2010) Facts and Fallacies of Software Engineering. Pearson Education. Kindle Edition.

Glass, Robert L.. (2006) Software Creativity 2.0 . developer.* Books. Kindle Edition.

Hunt, Andy. (2008) Pragmatic Thinking and Learning: Refactor Your Wetware (Pragmatic Programmers) Pragmatic Bookshelf. Kindle Edition.

martinfowler.com. (2020) The State of Agile Software in 2018. [online] Available at: https://martinfowler.com/articles/agile-aus-2018.html [Accessed 19 Jan. 2020].

Martin, Robert C. (2019) Clean Agile (Robert C. Martin Series) . Pearson Education. Kindle Edition.

Martin, R. (2014). Clean Coder Blog. [online] Blog.cleancoder.com. Available at: https://blog.cleancoder.com/uncle-bob/2014/06/20/MyLawn.html [Accessed 19 Jan. 2020].

  1. The term “Silver Bullet” as applied to software engineering originated with Fred Brook’s seminal 1984 paper, No Silver Bullet Essence and Accidents of Software Engineering. The paper is reprinted in his 1995 book The Mythical Man Month

  2. In his 2010 book Facts and Fallacies of Software Engineering, Robert Glass describes hypesters: “The hypesters, as it turns out, almost always are nonobjective folks who have something to gain—product sales, high-priced courses, or funding for research projects.” (pp. 20-21). 

  3. Read about the inception of agile process methodology directly from its creators: https://agilemanifesto.org/history.html 

  4. The prioritization of process and product are a major theme in Robert Glass’ 2006 book Software Creativity 2.0

  5. The Agile Manifesto https://agilemanifesto.org/ 

  6. Scrum: https://www.scrum.org/ 

  7. SAFe: https://www.scaledagile.com/ 

  8. For the purposes of this article, an industry expert is a person who has reached Stage 5 in the Dreyfus Model

  9. Robert Martin clearly articulates the correlation between the software industry’s growth rate and lack of talent in the following expert from his blog.

    I estimate that the world, today, has twenty-two million programmers. One of every 300 people is a programmer. In the US it’s closer to 1%. But in 1974 the number of programmers worldwide was vastly smaller, and probably numbered less than 100,000. That implies that in 40 years our ranks have increased by a factor of 220. That’s a growth rate of 14.5% per year, or a doubling rate of five years.

    If the ranks of programmers has doubled every five years, then it stands to reason that most programmers were hired within the last five years, and so about half the programmers would be under 28. Half of those over 28 would be less than 33. Half of those over 33 would be less than 38, and so on. Less than 0.5% of programmers would be 60 or over. So most of us old programmers are still around writing code. It’s just that there never were very many of us.

    What does this imply for our industry?

    Maybe it’s not as bad as Lord of the Flies, but the fact that juniors exponentially outnumbers seniors is concerning. As long as that growth curve continues there will not be enough teachers, role models, and leaders. It means that most software teams will remain relatively unguided, unsupervised, and inexperienced. It means that most software organizations will have to endlessly relearn the lessons they learned the five years before. It means that the industry as a whole will remain dominated by novices, and exist in a state of perpetual immaturity.

    Martin, R. (2014). Clean Coder Blog. [online] Blog.cleancoder.com. Available at: https://blog.cleancoder.com/uncle-bob/2014/06/20/MyLawn.html [Accessed 19 Jan. 2020]. 

  10. What has become known as the Dreyfus Model was incepted by Stuart E. Dreyfus and Hubert L. Dreyfus in their 1980 paper entitled A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Chapter 2 of Andy Hunt’s book Pragmatic Thinking and Learning: Refactor Your Wetware provides an excellent introduction to the topic. 

  11. The term “Agile Industrial Complex” originated from a presentation Martin Fowler delivered at Agile Australia in 2018. Read more about it here: https://martinfowler.com/articles/agile-aus-2018.html 

  12. Robert Glass supports the notion of domain-specific software processes in his 2010 book Facts and Fallacies of Software Engineering

    The controversy surrounding this particular fact is among people who don’t want to give up on the notion of fully generalized reuse-in-the-large. Some of those people are vendors selling reuse-in-the-large support products. Others are academics who understand very little about application domains and want to believe that domain-specific approaches aren’t necessary. There is a philosophical connection between these latter people and the one-size-fits-all tools and methodologists. They would like to believe that the construction of software is the same no matter what domain is being addressed. And they are wrong.

    (pp. 48-49)

  13. Peter Dinning’s book Computational Thinking describes how the customer’s expectations change as the product matures.

    Amazon created the Kindle reader service that enables users to purchase ebooks from the Amazon store and begin reading them instantly from any device with a Kindle app. Google and Apple maps use location information from smartphones to detect traffic congestion, overlay it on street maps, and propose alternate routes around congested areas. Blizzard Entertainment accumulated as many as 10 million subscribers to its World of Warcraft online game because of its rich complexity, easy entry, and detailed graphics. Uber allows users to hail rides whose drivers come to their exact location within minutes. In each case customers found they could do previously impossible things with the app than without, well beyond their expectations.

    The interesting thing about these examples is that many of them failed important ISO metrics such as portability, speed, efficiency, or reliability. Yet customers ignored those shortcomings and became avid and loyal subscribers to the software developer.

    (p. 145).