My Theory About Agile

[If you don’t follow software development, and in particular if you have no idea what “agile development” is, or how it contrasts with the “waterfall style,” this week’s blog post is not going to make much sense.  But that’s okay: I told you not to read this stupid blog anyway.]

Agile software development is really quite a cool thing, when done properly.  (If you disagree, please see my rant about hype.)  Unfortunately, most people who do Agile don’t really do it properly.  And, even more unfortunately, when Agile people try to help failing Agile shops, they often do it by pointing out where they deviate from a certain process (in whatever flavor of Agile is appropriate—XP, Scrum, whatever).  “You’re trying to do XP but you’re not pair programming,” they’ll say, or perhaps “you’re trying to do Scrum but you’re not standing up in your daily scrum meetings.” This is ironic (insanely ironic, even) because point #1 on the Agile Manifesto is: “Individuals and interactions over processes and tools.” So we’re supposed to downplay (although not ignore entirely) the processes, except when figuring out what we’re doing wrong, when suddenly it’s all about the process.  This seems wrong to me.  Most of the time when I’ve seen, or heard of, Agile going wrong, it isn’t the processes that are to blame.

It’s the attitudes.

I think that, in order to implement a good version of Agile—whether XP, Scrum, Crystal, or some melange—you need to start with an understanding of what Agile is supposed to give everyone.  So let me give you my theory on that.  Note that this theory is not backed up by scholarly research.  It is based solely on my own experience, and therefore is 100% anecdotal and useless from a strictly scientific perspective.  (As always, please refer to the masthead.)

Once upon a time, there was business, and there was tech.  These two camps didn’t always get along, primarily because of their radically different outlooks on the nature of reality and perception.  And one of the biggest conflicts they had (and, honestly, still do have) is when it comes to figuring out when something is going to get done.

See, tech people are horrible estimators.  We are, at heart, hopelessly optimistic.  I think this is mainly because, for the most part, we love what we do.  Which is unusual in the professional world, if you think about it.  Actors love what they do.  Musicians love what they do.  Professional athletes love what they do.  Really, if you think about it, programmers are some of the only people on earth who love what they get paid to do without ever getting rich or famous for it.  So, when you say, “we’re going to give you a really hard programming problem to solve,” we start salivating.  We’re looking forward to it.  When you say “now how long is that going to take?” ... we think about all the fun we’re going to have.  All the coolness, and all the interesting trickiness.  What we don’t think about is how long we’re going to spend going down blind alleys before we figure out the right way.  Or how long we’ll spend chasing down documentation on the new tools we’re going to have to learn.  Or how many bugs in IE are going to make us tear our hair out, or how long it’s going to take us to find our own bugs that QA thoughtfully pointed out to us after we were “done.” Or how much time we’re going to have to spend in meetings telling business why it’s taking so long.

So we underestimate.  It’s understandable, if not exactly forgivable.

Business has a different perspective.  They need to plan.  Us techies don’t plan.  We just do.  “It’ll be done when it’s done!” we say.  This makes perfect sense to us—after all, it couldn’t possibly be done before it was done, right?  But that makes businesspeople insane.  They need to market.  They need to develop release strategies and conduct market research and hold pricing meetings.  They need to write slick advertising materials about what the product will do, and there’s no way they’re going to wait to start doing that until after us techies are done making the damn thing.  So, to them, asking when it’ll be done is a perfectly rational question.

And thus the trouble begins.  I think that, probably, back in the dark ages of the software-induced shotgun wedding between business and tech, business would say “when?” and tech would answer “then” and business would say “fine” and then go away for however long that was.  Then they’d come back at the end and say “okay, where is it?”

This didn’t work.

After several iterations of not getting anything on the date they were “promised” it, business decided that they needed to check in with tech a little more frequently.  Maybe if they could at least know there were delays before it was too late to do anything about it, that might help.  Seems rational enough.  So let’s have a meeting every month just to see where we are.  And, you know, if meetings every month are good, meetings every week are better.  That’s just math.  Actually, meeting every day would be even better still.  Pretty soon, tech found that they might be spending as much as 20-25% of their entire workweek explaining why they weren’t getting any work done.

This didn’t work either.

Besides estimation problems, the other serious problem was specifications.  Business would say to tech “this is what we want the software to do.” And tech would say “okay.” And tech wouldn’t make it do that.  Exactly that; as I mentioned before, techies tend to be literaliststs.  Occupational hazard.  Whereas businesspeople tend to live more in the real world, where common sense is supposed to be applied to anything you say.  Silly businesspeople.  So there were communications problems.  And also, sometimes businesspeople don’t know what they want until they see it in action, wherein they promptly realize that they wanted something different all along.  Also, software takes time to develop, and the market doesn’t just sit around doing nothing while new software is being written.  Sometimes the requirements change in the middle of the project, and that’s not really anyone’s fault at all.

But somehow the solution to this problem became to make the specs bigger.  Longer, and more exact, and more like a contract.  There are legal documents—hell, there are bills in Congress—which make more sense and are easier to read than software specs produced by a serious waterfall methodology, like say ISO 9000.  And then, if something changed, everyone scrambled through their copy of the specs looking for the reason why it wasn’t their fault.

This was not particularly helpful.

See, there’s not only a fundamental communications gap between business and tech, there’s a fundamental difference in goals.  Tech wants everything to be beautiful.  Not perfect, because it can never be perfect, but at least pretty.  Most people don’t realize it, but programmers are craftsmen.  If you’ve ever engaged in a creative hobby—say, woodworking, or perhaps gardening—you understand pride of craftmanship.  When you make a chair, it better not be just good for sitting on.  It better be beautiful to look at, and detailed, and supremely comfortable.  When you plant a vegetable garden, it better not just produce food.  It better produce gorgeous, succulent vegetables that make the children’s mouths water and the neighbors green with envy.  If it’s not doing all that, you’re not done yet.  So it is with programmers.  It can’t just work.  Any moron can give you code that just works.  It needs to be code that other programmers will come along and read, and whistle to themselves, and be happy that such an amazing craftsman was able to provide them this as a starting point, because she just made their lives a thousand times easier.

Whereas business is all about numbers.  Mainly that bottom line number, the number that shows you how much profit you made at the end of the day.  But there are all sorts of numbers: customer satisfaction numbers, and product defect numbers, and market penetration numbers.  Some you try to maximize, and some you try to minimize, but it’s always about moving the needle.  Maybe you pull off some business maneuver that makes a good story over cocktails, but that’s just a bonus.  At the end of the day, you can tell whether you did well or poorly just by looking at the numbers, and numbers don’t lie.  And everyone else can tell how well you’re doing too.  It’s all very satisfying, in a visceral way.  When you get that report back, and the numbers are bigger (or smaller), you get this excitement in the pit of your belly, and you just want to pump your fist in the air and go “YES!”

And the thing is, it seems like these are completely incompatible viewpoints, but that’s not the real problem.  The problem is, they’re both wrong.  Oh, they’re both right, too (balance and paradox), but they’re both wrong.  And, when you put them together, they balance out nicely.  Tech without restraint becomes perfectionist, always tinkering with everything and never finishing anything.  Business without restraint becomes lost in the weeds of short-term gains and ignores the long-term goals that are being crushed.  Techies often forget that, if they don’t make deadlines, the bottom line suffers, and that’s what puts food in their mouths.  Businesspeople often forget that cutting corners today means defects—and therefore lost revenue—tomorrow.  But together, there’s a yin and a yang that usually produces pretty good results overall: not too slow, not too sloppy.

And this is what Agile attempts to do.  Agile says to tech, “You’re not going to get everything you want—sorry.  We’re going to give you an outlet for your concerns about doing things right, but you’re going to have to justify them.  And we’re going to put you in the driver’s seat on how long things are going to take, but you’re going to have to break it down into tiny little pieces and esimate every one.  And, yes, you’re going to have to give your status every day, but we promise it won’t take but 10 minutes.  If you want to spend longer on doing something the “right” way insead of doing it the expedient way, you’re going to give business numbers to show the difference.  This is how much it costs now, and this is how much it will cost later, and then business can make a rational decision on whether that trade-off makes sense.  And sometimes they’re going to agree with you, and you’ll be happy.  And sometimes they’re not, and you won’t be.  But it’s better than nothing.”

And then Agile says to business, “You’re not going to get everything you want—sorry.  We know you want 100% visibility into what tech is doing, but you can’t have that.  100% visibility equals 0% productivity, because it would be you and tech sitting in a room, with you going ‘What are you doing now?’ and them replying ‘Answering your question’ over and over again.  But we’re going to institute a structure so that nobody gets stuck for more than a day.  And we’re going to give you numbers.  How many units something will take, and how many of those units this team can deliver in a given time period.  That time period will be short—a month, or even two weeks—and you can specify exactly what will get done in each period (no more ‘tech spent 3 weeks doing this thing because they thought it would be useful’), but you can’t change your mind in the middle of the period.  If there are changes, just wait a few days until the new period comes along.  And the team is a team: they’re a cohesive unit that works together, and you can’t just replace team members or juggle them around like they’re factory components.  So you can’t bother them while they’re working, and you can’t try to measure them individually (which ends up creating destructive competition anyway), but you’ll get lots of numbers that let you know what’s going on and when things are going to get done.  And that’ll have to do.”

This is where business and tech need to start from if they’re going to try to make Agile work for them.  They both need to start from a place where they each realize they’re not going to get everything they want.  In my experience, most of the time when Agile goes wrong, it’s because tech is on board with this compromise, but business didn’t get the memo.  Next week we’ll look in more detail about just what can go wrong when that happens.