Of Waterfalls, Pigs, and Red Red Tape

Once upon a time we used to develop software via something known as the “waterfall model.” When trying to envision this, don’t think about something like Angel Falls, where the water just falls off a cliff.  Picture instead something along the lines of Detian Falls in Vietnam: a series of steps as the water drops, level by level, to its final destination.  See, back in those olden times (which, honestly, were mostly before I got into the industry, though there was some residual inertia even in the mid-80s, when I came along), back then, as I say, the specifications (or “specs”) were developed by one team, who then passed them along to the design team, who designed the whole system in a very abstract way, and then passed it along to the programming team, who did all the hard coding work, and then passed it along to the QA team, who verified that the functionality matched the original specs, and then they passed it on the customer and then it’s done.  The primarily analogy to a waterfall for this development model is that the water only flows one way: for the real-world waterfall, that’s due to gravity, and for the waterfall model, it’s because going backwards—“upstream,” if you will—is expensive.  You really want to get each phase just perfect, because if you find a mistake, you essentially have to start over ... and that costs the company money.  Sometimes, with this model, starting over was so expensive that they just didn’t.  The greatest stories of software development debacle were due to the sunk cost fallacy: too expensive to try get back up to the top of the waterfall, so we just gotta make due with whatever horror show we’ve ended up with.

So throughout the 80s and 90s software developers started saying there had to be a better way.  In 1986 the “spiral model” was proposed: it was built into the system that, instead of planning out the whole system at the beginning, you’d spec out just an initial prototype, then design that, code it, test, then go back to the spec stage and tack on more features.  Starting over was no longer a bug, but a feature.  Instead of losing a bunch of money because we had to start everything from scratch, we were only starting the next thing from scratch ... and, if we needed to tweak some stuff from the first iteration, well, we already had the mechanisms in place for specifying, desiging, coding, and testing.  Those phases were in our past, true: but they were also in our future.

Of course, the spiral model is a very abstract concept.  How do you actually implement such a thing?  That is, what are the actual processes that you put into place to make sure the company and its employees follow the model and achieve the goals of iterative design?  For that, we needed to move beyond models and into methodologies.  Enter Agile.

Agile software development practices, usually just referred to as “Agile,” were a way to concretize the spiral model abstraction.  Sometimes they would propose tweaks to the model, sure, but the main thing was, no going back to waterfall.  And, to distance themselves from those crusty old waterfall methodologies—many of which were by this point formalized as standards, such as the DOD’s 2167A—they all had cool new names: RAD (“Rapid Application Development”) and Scrum and Crystal Clear and Extreme Programming (if you didn’t just hear a Bill and Ted’s-style guitar lick, you’re doing it wrong).  This last one, usually abbreviated to “XP” (no relation to the Windows version) was not the first agile methodology to come along ... but it was the first one I was ever exposed to, and they say you never forget your first.

Kent Beck, author of “Extreme Programming Explained,” presented to me a perspective that literally changed my (software development) life.  He pointed out that, in order for the waterfall model to work, you have to be able to predict the future.  The whole thing is predicated on predicting what problems will happen, anticipating them, and building them into the plan.  If you fail to predict something, then everything falls apart.  Except ... humans really suck at predicting the future.  When we say “predict,” what we really mean is “guess.” And we usually guess wrong.  As Kent so succinctly put it:

The problem isn’t change, per se, because change is going to happen; the problem, rather, is the inability to cope with change when it comes.

Stop trying to keep change from happening: it’s a fool’s errand.  Rather, create a better methodology which says “yeah, things change: so what? we got that covered.”

Agile is all about being flexible.  Hell, the reason it’s called “agile” is because the old waterfall methodologies were ponderous and slow to course-correct.  It’s common for business people to talk about agility in terms of responding to changes in the market: the creators of the Agile Manifesto (one of whom was Beck himself) wanted to capitalize on that perception.  Our development practices can make your company more agile, and that makes you quicker to respond, and that helps you beat your competitors.

And yet ... it’s kind of strange that we need all these procedures and guideliness and principles and a whole friggin’ manifesto to perform something for which the entire purpose is to be flexible.  The thing I never liked about XP, despite all its merits (and the aforementioned life-changing-ness), was that it had all these notes about how, if you’re not following every single rule, then you’re not “doing” XP.  You’re just playing at it.  I always found that inherent dichotomy cognitively dissonant: so I have to do things exactly according to these rules so that I can break the rules? I have rigidly fit into the straitjacket so that I can have the flexibility to move freely? I have to precisely walk the straight line so that I have the freedom to jump in any direction?  Surely you see the contradiction.

And XP is certainly not alone in this strange philosophy.  I’m not sure we can claim any of the Agile methodologies to have “won,” but in my experience Scrum has made the most extensive inroads into corporate culture.  And it is chock full of prescriptive little strictures: mandatory stand-up meetings with strict time limits and precisely defined cycles called “sprints” and detailed reporting pathways between developers and business owners.  Maybe all this red tape is why business people have embraced it more than the other Agile practices.  But it presents a weird, oxymoronic message to the developers: we want to you to be free, we want you have flexibility, but you have to all these things, just so.  And sometimes the business owners can get very upset if you question this.  Because they’ve been trained, you see?  They’ve taken courses in “how to do Agile” and “how to run Scrum” and all that, and (of course) all those courses stressed that you have to do everything perfectly or else it will all fall apart, so as soon as the developer suggests that maybe we should change this one thing because it’s actually making our lives harder ... well, it won’t be pretty, let me tell you.

One of things I always liked about Scrum was that they made clear the difference between involvement vs commitment.  The traditional explanation for this is via the fable of the pig and the chicken.  Now, these days Agile folks will tell you not to use that story to explain things any more.  The first reason they cite is that people will take offense: calling someone a pig implies they’re greedy, or dirty; calling them a chicken implies that they’re cowardly.  These are, of course, human metaphors that we’ve placed on those particular animals, and also they have nothing to do with the actual story.  But people won’t hear the message, they point out, if they’re hung up on the words used to deliver it.  I would probably say that people will look to any excuse to get offended, especially if it gets them out of following rules, but I’m a bit more of a cynic.

The story points out that, in the context of preparing a breakfast of eggs and bacon, the chicken is involved, but the pig is committed.  This is a very simple concept to grasp, and the analogy illustrates it perfectly, but, yes, yes: let us not offend anyone.  I would be fine if this first reason were the only reason that modern Agile advocates had dropped the pig and chicken story: that would just mean that they had replaced it with a different analogy that perhaps involved more noble animals, or fruits, or something.  But, no: they’ve started to question the whole concept.  See, the original point of pigs and chickens was to point out to the business people that it wasn’t particularly fair (or, you know, sensible) for them to set deadlines for how long something would take.  They weren’t the ones doing it.  The developers have to actually accomplish the thing, and they know how long it should take (even if they’re bad at estimating that for other reasons, which are happily addressed by other Agile practices).  The business owners are involved, but the developers are committed.  This not only stresses to the business folks that they don’t get to say how long something takes, but it also stresses to the developers that, once they say how long it will take, they’ve made a commitment to getting it done in that timeframe.  These are all good things.

But not so, says the Updated Scrum Guide.  Those poor business people shouldn’t be made to feel like they can’t dictate timelines.  “In some cases these people were key project sponsors or critical system matter experts. These are individuals who, while possibly needing some education and guidance from a Scrum Master, can be critical to the success of a project.” If you’re not familiar with how to translate business bullshit back into English, this means “we want the business people to feel important, and they don’t like it when we try to put restrictions on them, and if I say it this way it’ll make you think that you developers are actually gaining something, rather than that we’re just caving in and letting the business people run roughshod over all we’ve built.” The thing I always liked about the Agile practices was that they were pretty balanced in terms of business vs development.  They said to the developers “we want you to feel respected and like your creativity is valued, and you should be in control of what you produce and the quality of your work.” But they also said to the business side “we want you to feel respected and like your market acumen is valued, and you should be in control of what gets produced and how viable it is as a product.” See? everybody is respected equally.  When you start breaking down that balance, bad things happen.

And maybe business people feel empowered to just set some processes up because it sounds good, or because it looks official, or maybe, like most other humans in workplaces, they just like telling other people what to do.  And maybe those processes actually slow things down instead of making them more efficient.  And maybe the developers feel like they can’t speak up any more—it’s no longer the case that they’re the ones who are committed because everyone’s committed now—or maybe they do speak up but they’re ignored because setting up workflow processes ... that’s above your paygrade, don’t you know.  And gradually, slowly, everything goes back to the bad old ways when we spent way more time talking about getting things done than actually doing things.

Doesn’t feel very agile, does it?