Pitfalls in Agile

[This is part 2 of a series of posts on Agile Development.  First you should decide if you care about that at all.  Then, if you decide you do, you should make sure you’ve read what I wrote last week about what agile means.]

So now we know what the attitudes towards agile should be.  But, as I’ve said, often they aren’t.  Sometimes it is that tech are the ones who don’t get it.  But I’ve found in my experience that it’s more often business.  Tech, after all, is used to not always getting what they want.  But business is the driver of a business—it’s right there in the name, after all.  What business needs, they get.  They’re not used to hearing “no.”

But here’s a crucial fact: agile did say to business “you’re not going to get everything you want,” true.  But that’s not the same as saying “no.” What it’s saying instead is, “you must make a choice.”

In 1975, Frederick Brooks was considered a revolutionary for pointing out that nine women can’t make a baby in one month.  When I said that in a meeting recently, I got nervous titters and a few frowns.  Yes, that’s right: even after forty years, people are worried that I might be talking about something vaguely related to sex instead of recognizing that I’m quoting The Mythical Man-Month, possibly the most seminal work on software management ever published.  How is it that there is a single person in software development who doesn’t know this work, or at least the general concepts of it?  MMM is whence cometh Brooks’s law, which states simply that “adding manpower to a late software project makes it later.” Surely if you’re in the business of software development—from either side—these are concepts you need to understand.

But the point that both Brooks and agile are attempting to make is that we don’t say “no.” We must instead reframe the question.  If you need to buy two items that each cost $100, and you have $100 in your budget, you don’t say “no” to one or the other—you simply choose which one you will buy now, and which you will buy later.  In fact, allocation of limited resources is something that business does extremely well.  This is because of the numbers thing again.  Juggling numbers and balancing expenditures is the true forté of business.  Agile is not attempting to deny business’s requests.  It is reframing the question in terms of prioritization of one of the scarcest resources business has: the time and effort of its tech department.

Because tech can’t possibly do everything at once.  It can often do many things at once (depending on the size of the team), but it shouldn’t come as a surprise to anyone that coming up with ideas is easy and implementing them in the real world is hard.  Which means that business will always have more for tech to do than tech can possibly complete.  Which, in turn, means that business has to choose which things are the most important.

This should not be a radical concept.

And, yet, in every corporate environment I’ve ever worked in (and I dare not even exempt the business I ran myself for 12 years), business tells tech that every project is “the most important one.” Perhaps not all the time—perhaps not even most of the time—but always there comes that time when somehow it seems rational for business to just expect tech to do it all at once, without exceeding the budget, without missing the deadline, and without any one project suffering or dragging down the others.

But agile exposes the fallacy in this.  Agile gives us the concept of the “Iron Triangle.” One side of the triangle is time.  Another is functionality.  And the third is quality.  And, what agile tells us, quite explicitly, is that, if business will not move the deadlines, and will not accept any redcution in functionality, there’s only one other place where reductions can be made.

In fact, the whole point of agile is to institute a negotiation over functionality: the concept of “if you want to add something to this iteration, you have to take something out” is exactly that.  The time is not negotiable in one sense, because the dates of the iterations (often called “sprints”) are fixed.  But agile addresses time as well, by forcing regular releases before the final deadline of the whole project.  In this way, business gets value earlier, which mitigates potential losses should the deadline have to be pushed back.  And the end result is that tech can put in the quality that they wanted all along.

Where can this all go wrong?  Several places, but one of the big ones is with the Product Owners.

Agile is designed to work with small teams.  If you have a big team, you divide it up into several small teams.  These are tech teams, but they’re led by a person called a Product Owner, or just “PO.” The job of the PO is to be the liaison between business and tech.  That is, the job goes something like this:  First the PO goes to the business and say “What do you want?” Business tells the PO all the things it wants (of which there are likely very many).  The PO nods sagely and goes back to tech.  “Here’s what the business wants,” they say.  Tech promptly objects: it’s too much, or it doesn’t consider this challenge or that hurdle, or it will result in this drop in quality or maintainability, or it conflicts with this other piece of the system over here.  In this conversation, the PO represents business: “If you don’t want to give them this,” they’ll say, “then you better tell me why.  And use small words, and hard numbers.” After some back and forth, tech gives in on a few points, but stands firm on others.  Once the PO has pushed all they can push, they go back to business, and now they’re representing tech.  “The good news,” they say, “is that you can have this and this.  The bad news is that you can’t have that and that, or you can’t have it right now, or you can’t have it in that form.  And here are the numbers that demonstrate why.” Business loves this, because they love having numbers to analyze.  That makes it all make sense to them, as opposed to vague technical discussions of efficiency and flexibility.  “Okay, we’ll grant you that one,” business may say, “but this one over here we just can’t give up.” And the PO responds: “Tell me why.  And use precise words, and outline the risks and challenges of not doing it.” After more back and forth, the PO goes back to tech and represents business again.  “They agreed to give up on this thing.  But they have to have some solution to the other thing.  Here’s exactly why we can’t just not do it.  If you say you can’t give them exactly what they want: fine.  But you have to give them something.  You’re the technical geniuses: figure it out.” Tech loves this, because now they’ve been given a challenge, a hard problem to solve, something which will require them to think outside the box and push themselves to the limit to come up with an elegant solution.

So, as you can see, the PO has to represent both sides, at different times.  So obviously the first mistake you can make when trying to implement agile is to choose the wrong people to be the POs.  Most often what happens is that the Product Managers are chosen.  This is disastrous.  They have an inherent conflict of interest: when tech says “no,” they take it personally and throw up roadblocks.  When it’s time to go back to business, they are business.  They only people they’re likely to be reporting to are their bosses, and are they really going to push hard to represent the interestes of tech when it means disagreeing with their boss?  You can’t blame them for not doing that—it’s political suicide.

And agile is supposed to be all about communicating.  When a problem comes up, you’re supposed to report it immediately, so the business is aware that there’s a potential problem.  And the person you should report it to is, of course, your PO.  What happens when the PO is also the Product Manager?  You get immediate pushback.  “Unacceptable!” you’re told.  Suddenly you’re in the exact situation that agile is supposed to avoid: you’re under pressure to agree to continue to meet a deadline that is no longer feasible.  And tech people are, for the most part, an agreeable bunch who really don’t like confrontation: usually they’ll tell you what you want to hear, eventually, just to stop the uncomfortable situation.  So now either they have to work extra hours to make an unreasonable deadline (which leads to employee burnout and your best and brightest tech resources polishing up their résumés on Monster), or they starting cutting corners (the Iron Triangle strikes again), or the deadline just plain gets missed, which was probably going to happen anyway, except this time business didn’t get to find out ahead of time (surprise!).  All that communication that agile was supposed to be fostering gets destroyed.

The other thing that suffers when Product Managers become POs is team cohesion.  Agile is all about small teams, and people working together, and helping each other out, shoring up each others’ weaknesses.  This is why you always measure the velocity of the team and never that of its individual members.  When people work together on a team for long periods of time, they develop efficiencies of coordination: they know when someone is going to have trouble with this, or exactly who to go to when there’s an issue with that.  They can do things efficiently with a minimum of noise-to-signal in their interpersonal communications, because they know each other so well.  But when Product Managers are the POs, or even just assigned to teams, they have a tendency to want to shuffle the team members around instead of shuffling the work around.  If Product Manager A has more work to do than her team can handle, she goes to Product Manager B, who happens to have less work right now.  She wants to borrow some of his team members.  Perhaps Product Manager B doesn’t want to give them up; what happens if my projects heat up while you’ve got some of my people?  Even if you somehow manage to avoid the disruptive turf wars, and the transfer of resources is smooth, you’ve destroyed your team dynamic.  The transfer only appears smooth to the business side: on the tech side, the employee has a whole new team to learn about, new politics undreamt of, new social waters to navigate.  And, on top of everything else, you give your chronic underperformers a way to hide: in a situation where teams are for the most part permanent, you can count on your good employees to root out people who aren’t pulling their weight; when teams are constantly in flux, it’s more likely that they’ll just try to pass them around to another team, to (again) avoid the confrontation.

I’m not actually suggesting anything radical here; in Dean Leffingwell’s excellent Agile Software Requirements, for example, he makes it clear that Product Managers are not ideal POs, and I could cite many other sources.  But, in the end, this is a symptom, not a cause.  The cause is business not taking agile for what it is.  Too often it’s seen as some palliative: let’s just keep the developers happy, because we need them.  If this is what keeps them around, fine.  But we still want our deadlines met, no matter what.

This is contrary to the spirit of agile.  Agile, as we started out discussing, is about making hard choices.  It’s about saying what’s most important and how long things should take and how to deal with problems as they ariese.  When the business says, “this must be done on this date, and it must include all the features,” I’m somehow reminded of talking to small children.  Declaring something to be so doesn’t mean that it will happen.  You may even stamp your foot and hold your breath, if you like.  That’s not going to make the work go any faster.

In fact, unreasonable date pressure is probably the most common area where agile fails.  Of course, the hardcore agilists will complain that, if you have hard deadlines without functionality negotiations, you’re not doing agile at all.  And, perhaps, in a strict technical sense, you aren’t.  That doesn’t actually help address the problem though.  The problem is that tech people get very frustrated with these types of situations.  They’ll either buckle to the pressure, and make themselves miserable (back to polishing their résumés), or they’ll mentally give up, slow down, and make business miserable (and, I don’t know if business has noticed this or not, but it’s typically not the tech people who get fired if this scenario deteriorates).  Either way, business loses.  Their numbers suffer.  And this is not what business wants.

So the solution is for business to come at it with a fresh perspective.  For them to take the time to understand what agile means, and what it has to offer.  Agile is certainly not perfect (nothing is).  Agile certainly has lots of room for improvement (everything does).  But agile also takes a lot of different aspects into account, and actually produces a process that has the potential to make everyone happy in the long run.

There are companies out there who have seen that potential fulfilled.  Perhaps your company will join them.

[Having finished the second part of my two-part series, you may think you’re done.  However, I encourage you to stay tuned for part 3, where I talking about managing programmers.]