My Theory About Agile

[If you don’t follow software development, and in particular if you have no idea what “agile development” is, or how it contrasts with the “waterfall style,” this week’s blog post is not going to make much sense.  But that’s okay: I told you not to read this stupid blog anyway.]

Agile software development is really quite a cool thing, when done properly.  (If you disagree, please see my rant about hype.)  Unfortunately, most people who do Agile don’t really do it properly.  And, even more unfortunately, when Agile people try to help failing Agile shops, they often do it by pointing out where they deviate from a certain process (in whatever flavor of Agile is appropriate—XP, Scrum, whatever).  “You’re trying to do XP but you’re not pair programming,” they’ll say, or perhaps “you’re trying to do Scrum but you’re not standing up in your daily scrum meetings.” This is ironic (insanely ironic, even) because point #1 on the Agile Manifesto is: “Individuals and interactions over processes and tools.” So we’re supposed to downplay (although not ignore entirely) the processes, except when figuring out what we’re doing wrong, when suddenly it’s all about the process.  This seems wrong to me.  Most of the time when I’ve seen, or heard of, Agile going wrong, it isn’t the processes that are to blame.

It’s the attitudes.

I think that, in order to implement a good version of Agile—whether XP, Scrum, Crystal, or some melange—you need to start with an understanding of what Agile is supposed to give everyone.  So let me give you my theory on that.  Note that this theory is not backed up by scholarly research.  It is based solely on my own experience, and therefore is 100% anecdotal and useless from a strictly scientific perspective.  (As always, please refer to the masthead.)

Once upon a time, there was business, and there was tech.  These two camps didn’t always get along, primarily because of their radically different outlooks on the nature of reality and perception.  And one of the biggest conflicts they had (and, honestly, still do have) is when it comes to figuring out when something is going to get done.

See, tech people are horrible estimators.  We are, at heart, hopelessly optimistic.  I think this is mainly because, for the most part, we love what we do.  Which is unusual in the professional world, if you think about it.  Actors love what they do.  Musicians love what they do.  Professional athletes love what they do.  Really, if you think about it, programmers are some of the only people on earth who love what they get paid to do without ever getting rich or famous for it.  So, when you say, “we’re going to give you a really hard programming problem to solve,” we start salivating.  We’re looking forward to it.  When you say “now how long is that going to take?” ... we think about all the fun we’re going to have.  All the coolness, and all the interesting trickiness.  What we don’t think about is how long we’re going to spend going down blind alleys before we figure out the right way.  Or how long we’ll spend chasing down documentation on the new tools we’re going to have to learn.  Or how many bugs in IE are going to make us tear our hair out, or how long it’s going to take us to find our own bugs that QA thoughtfully pointed out to us after we were “done.” Or how much time we’re going to have to spend in meetings telling business why it’s taking so long.

So we underestimate.  It’s understandable, if not exactly forgivable.

Business has a different perspective.  They need to plan.  Us techies don’t plan.  We just do.  “It’ll be done when it’s done!” we say.  This makes perfect sense to us—after all, it couldn’t possibly be done before it was done, right?  But that makes businesspeople insane.  They need to market.  They need to develop release strategies and conduct market research and hold pricing meetings.  They need to write slick advertising materials about what the product will do, and there’s no way they’re going to wait to start doing that until after us techies are done making the damn thing.  So, to them, asking when it’ll be done is a perfectly rational question.

And thus the trouble begins.  I think that, probably, back in the dark ages of the software-induced shotgun wedding between business and tech, business would say “when?” and tech would answer “then” and business would say “fine” and then go away for however long that was.  Then they’d come back at the end and say “okay, where is it?”

This didn’t work.

After several iterations of not getting anything on the date they were “promised” it, business decided that they needed to check in with tech a little more frequently.  Maybe if they could at least know there were delays before it was too late to do anything about it, that might help.  Seems rational enough.  So let’s have a meeting every month just to see where we are.  And, you know, if meetings every month are good, meetings every week are better.  That’s just math.  Actually, meeting every day would be even better still.  Pretty soon, tech found that they might be spending as much as 20-25% of their entire workweek explaining why they weren’t getting any work done.

This didn’t work either.

Besides estimation problems, the other serious problem was specifications.  Business would say to tech “this is what we want the software to do.” And tech would say “okay.” And tech wouldn’t make it do that.  Exactly that; as I mentioned before, techies tend to be literaliststs.  Occupational hazard.  Whereas businesspeople tend to live more in the real world, where common sense is supposed to be applied to anything you say.  Silly businesspeople.  So there were communications problems.  And also, sometimes businesspeople don’t know what they want until they see it in action, wherein they promptly realize that they wanted something different all along.  Also, software takes time to develop, and the market doesn’t just sit around doing nothing while new software is being written.  Sometimes the requirements change in the middle of the project, and that’s not really anyone’s fault at all.

But somehow the solution to this problem became to make the specs bigger.  Longer, and more exact, and more like a contract.  There are legal documents—hell, there are bills in Congress—which make more sense and are easier to read than software specs produced by a serious waterfall methodology, like say ISO 9000.  And then, if something changed, everyone scrambled through their copy of the specs looking for the reason why it wasn’t their fault.

This was not particularly helpful.

See, there’s not only a fundamental communications gap between business and tech, there’s a fundamental difference in goals.  Tech wants everything to be beautiful.  Not perfect, because it can never be perfect, but at least pretty.  Most people don’t realize it, but programmers are craftsmen.  If you’ve ever engaged in a creative hobby—say, woodworking, or perhaps gardening—you understand pride of craftmanship.  When you make a chair, it better not be just good for sitting on.  It better be beautiful to look at, and detailed, and supremely comfortable.  When you plant a vegetable garden, it better not just produce food.  It better produce gorgeous, succulent vegetables that make the children’s mouths water and the neighbors green with envy.  If it’s not doing all that, you’re not done yet.  So it is with programmers.  It can’t just work.  Any moron can give you code that just works.  It needs to be code that other programmers will come along and read, and whistle to themselves, and be happy that such an amazing craftsman was able to provide them this as a starting point, because she just made their lives a thousand times easier.

Whereas business is all about numbers.  Mainly that bottom line number, the number that shows you how much profit you made at the end of the day.  But there are all sorts of numbers: customer satisfaction numbers, and product defect numbers, and market penetration numbers.  Some you try to maximize, and some you try to minimize, but it’s always about moving the needle.  Maybe you pull off some business maneuver that makes a good story over cocktails, but that’s just a bonus.  At the end of the day, you can tell whether you did well or poorly just by looking at the numbers, and numbers don’t lie.  And everyone else can tell how well you’re doing too.  It’s all very satisfying, in a visceral way.  When you get that report back, and the numbers are bigger (or smaller), you get this excitement in the pit of your belly, and you just want to pump your fist in the air and go “YES!”

And the thing is, it seems like these are completely incompatible viewpoints, but that’s not the real problem.  The problem is, they’re both wrong.  Oh, they’re both right, too (balance and paradox), but they’re both wrong.  And, when you put them together, they balance out nicely.  Tech without restraint becomes perfectionist, always tinkering with everything and never finishing anything.  Business without restraint becomes lost in the weeds of short-term gains and ignores the long-term goals that are being crushed.  Techies often forget that, if they don’t make deadlines, the bottom line suffers, and that’s what puts food in their mouths.  Businesspeople often forget that cutting corners today means defects—and therefore lost revenue—tomorrow.  But together, there’s a yin and a yang that usually produces pretty good results overall: not too slow, not too sloppy.

And this is what Agile attempts to do.  Agile says to tech, “You’re not going to get everything you want—sorry.  We’re going to give you an outlet for your concerns about doing things right, but you’re going to have to justify them.  And we’re going to put you in the driver’s seat on how long things are going to take, but you’re going to have to break it down into tiny little pieces and esimate every one.  And, yes, you’re going to have to give your status every day, but we promise it won’t take but 10 minutes.  If you want to spend longer on doing something the “right” way insead of doing it the expedient way, you’re going to give business numbers to show the difference.  This is how much it costs now, and this is how much it will cost later, and then business can make a rational decision on whether that trade-off makes sense.  And sometimes they’re going to agree with you, and you’ll be happy.  And sometimes they’re not, and you won’t be.  But it’s better than nothing.”

And then Agile says to business, “You’re not going to get everything you want—sorry.  We know you want 100% visibility into what tech is doing, but you can’t have that.  100% visibility equals 0% productivity, because it would be you and tech sitting in a room, with you going ‘What are you doing now?’ and them replying ‘Answering your question’ over and over again.  But we’re going to institute a structure so that nobody gets stuck for more than a day.  And we’re going to give you numbers.  How many units something will take, and how many of those units this team can deliver in a given time period.  That time period will be short—a month, or even two weeks—and you can specify exactly what will get done in each period (no more ‘tech spent 3 weeks doing this thing because they thought it would be useful’), but you can’t change your mind in the middle of the period.  If there are changes, just wait a few days until the new period comes along.  And the team is a team: they’re a cohesive unit that works together, and you can’t just replace team members or juggle them around like they’re factory components.  So you can’t bother them while they’re working, and you can’t try to measure them individually (which ends up creating destructive competition anyway), but you’ll get lots of numbers that let you know what’s going on and when things are going to get done.  And that’ll have to do.”

This is where business and tech need to start from if they’re going to try to make Agile work for them.  They both need to start from a place where they each realize they’re not going to get everything they want.  In my experience, most of the time when Agile goes wrong, it’s because tech is on board with this compromise, but business didn’t get the memo.  Next week we’ll look in more detail about just what can go wrong when that happens.

Guides: Willie Vadnais

[This is one post in a series about people who have had a great impact on my life.  You may wish to read the introduction to the series.]

I started my company back in 1992.  It wasn’t much more than just me for a while, but through the 90s I grew it, adding more and more employees.  Mostly I hired fellow coder geeks, but I also spent a fair amount of time trying to find someone to run the business side of things so I didn’t have to think about it, someone to handle the acquisition of new clients, the management of old ones, deal with the financial crap, etc.  Because all I really cared about was coding.

I had different amounts of success with this, but never did my company achieve any sort of greatness.  Sometime around 2004 business had slowed to such a trickle that I had to start looking for a full-time job for myself just to pay the bills.

Meanwhile, in 1995, in the exact same town I lived in, three friends started an ISP that would eventually turn into ThinkGeek.  And they actually did manage to achieve a sort of greatness—the chances are that you’ve heard of their company, but you’ve never heard of mine (trust me on that).  They did that because they had a core group that was actually good at acquiring customers.  They could have used some coding help, perhaps (though the original ThinkGeek code monkey was a demon of a workhorse and a great friend of mine to this day, he was only one man, and I’m sure he wished he had some help many times).

I’ve talked before about twists of fate that we think of as “coincidences,” and in fact the story of how I went to work at ThinkGeek is in that very post, so I shan’t repeat it here.  What interests me now, as I look back on this story, is how much our lives are shaped by the strange twists of fate that don’t happen.  For instance, the ThinkGeek founders and I were both working with Linux software in the same DC suburb.  The tech community there wasn’t all that huge.  How did we never meet?  Why is it that they, who had brilliant ideas and needed programmers, and me, who had brilliant programmers but needed ideas, never managed to connect?

“The ThinkGeek founders” consist of the three original partners, plus their original programmer.  I got to know all of these people pretty well over the course of the three years I worked there.  They’re all amazing, and amazingly talented, people.  In the music that ThinkGeek created, they were the core band: the rest of us were backup singers and studio musicians.  And, in the band that is ThinkGeek, the lead singer is Willie Vadnais.

Just as I was the one who founded my company, the one who, even though he was not its CEO, its President, or the chairman of its board of directors, was still the heart of it, the man behind its vision, the oracle of its Delphi ... so Willie was to ThinkGeek.  Not always in charge, but always the center.  The man with the plan.  If you wanted to know what ThinkGeek was, at its core, you talked to Willie.

There’s only one person I ever worked for who was more fun to work for than myself, and it was Willie.  I’m not saying he’s the best boss I ever had—hell, technically he was never my boss at all—but in terms of sheer joy in coming to work every day, nobody has ever beaten Willie.  I looked forward to going to work every day when I worked at ThinkGeek even moreso than when I worked for myself.  And he was a big part of that.

Willie is in some ways a walking oxymoron.  He’s a true salesman, and also a consummate geek.  Now, in my post on reality and perception, I discussed why these are generally opposite sorts of people.  They have diametrically opposed outlooks on life.  So it’s pretty rare to find both outlooks in the same person.  But Willie is that guy.  He can come up with the brilliant ideas, sell them to the people who need convincing, and still talk technical details with the people who need to implement them.  Honestly, he’s ruined salesmen for me forever: I’ll probably always compare any new ones I meet to him, and they won’t come out looking very good.

The folks who founded ThinkGeek sold it to a larger corporation many moons ago, before I ever came along.  But they all stayed on to help run the company, to help keep things going smoothly, to keep the company true to its roots.  Jon still wrote all the back-end code, and Jen still wrote all the front-end code; Scott still dealt with all the geeky, techy hardware that ThinkGeek sold.  And as for Willie, he still kept coming up with the ideas, finding the unique products, rethinking how the website should work, tossing out the witty slogans that ThinkGeek would put on the tee-shirts.  But, one by one, the founders left ThinkGeek, moving on in frustration or pushed out by corporate overlords who didn’t understand that the company couldn’t be the same without them.  Finally, only Willie was left, and, this past Monday, ThinkGeek said goodbye to him as well.

Now, I don’t have any special insight into the corporation that currently owns ThinkGeek.  I don’t have any insider information.  But I’ve seen this pattern in corporate dealings before.  It’s amazing how efficiently a corporate machine can dismantle a golden goose to produce goosesteaks.  ThinkGeek will go on, I’m sure.  There are still many good people who work there, and they will do their best to make sure the company lives up to its reputation.  But its heart is gone.  ThinkGeek without Willie is ... strange to me, a foreign beast that I’m unsure what to do with.  I’m not sure I can in good conscience continue to shop there.

I consider Willie a friend of mine.  He was wonderful with my children, he had me over to his house on many an occasion, and I’ve gotten roaring drunk with him several times.  He’s wise in some ways, and full of childlike wonder in others.  He knows what people want, what they like, what will please them and tickle their funnybones.  He’s one of the few people in the world who, if they came up to me today and said “I have an idea for a new business,” I would say “I’m in” without even needing to hear the rest of the pitch.  He’s inspired me sometimes, and frustrated me sometimes, and made me laugh a whole lot of times.  He challenged my beliefs about salemen.  My life is richer for having known him, and I hope I get a chance to work with him again someday.

Every now and again, I get a late-night IM from Willie, although it hasn’t happened in quite a while.  Usually, he’s trying to convince me of some insane premise that’s he’s come up with, often after imbibing a few intoxicants.  This might be a business premise, but just as often it’s something else entirely.  I am typically skeptical; he is typically persuasive.  Often I end up rolling my eyes at his ideas, virtually speaking.  Still, I sort of miss it.  Maybe now that he has more time on his hands, I’ll get another midnight message.  Maybe he’ll have another hare-brained scheme percolating in his brain.  If so, I’ll be honored that he chose to share it with me.

Working Man's Lament

There is a bit of “grass is always greener” going on in today’s post, I’ll warn you.  Of course, for someone who believes in balance and paradox, this is perhaps not surprising.  In any given situation where you’re trying to decide “which one is better?”, the answer is almost always: both.  Everything in life has both advantages and disadvantages, so any given binary choice is going to have you weighing pros and cons.  If you’re lucky, it’s easy to see which side outweighs the other.  Generally, you’re not lucky.

For instance, I went to college in two different spurts, with a gap of about 3 years in between.  What I usually tell people by way of explanation of this is that I dropped out of college, because college sucked, and went to work in the real world.  Then I went back to college because the real world sucked even more.  But of course the truth is that both sucked.  And both were awesome.  Just differently, and in different ways, and at different times.  The first time I went to college, I wasn’t prepared to appreciate how cool college can be.  Having to hold down a real job certainly made me appreciate that a lot more.  On the other hand, the first time I had to hold down a real job, I definitely wasn’t prepared to appreciate the freedom in it, the satisfaction that comes with responsibility.  After 3 more years of college, I was much better prepared.

Now I’m going to back to full-time work after a six week sabbatical.  I’ll be honest: I’m having a bit of trouble adjusting.  I’ve been working full-time for a lot of years now—decades, in fact—and I never thought that I’d be interested in retirement.  I always figured I’d be that guy that worked so long he was retired forcibly.  Now, though ... now I think I see the attraction.

My working life has had its interruptions too.  I spent 3 years in between college and college (as I mention above) working on jobs from furnace cleaer up to C programmer.  Then, while I was in school (the second time), I went back to part-time work, but eventually fell in with some guys who made me a business partner.  Which worked for a while, until it didn’t.

After I graduated, I felt ready to take the step of starting my own business.  I wasn’t too keen on lots of aspects of it (I had seen first-hand the effects of the stress on my partners), but I was also ready to be in charge of my own business fate for a change.  I ran my own company for 14 years, with anywhere from zero to 15 employees (or subcontractors), most of whom were (naturally enough) other programmers, but also project managers, bookkeepers, graphics artists, sys admins, salesmen, and office handymen.  I met The Mother while running my own business.  I hired nearly every one of my friends at one point or other (including The Mother, who was our bookkeeper and office manager for many moons), or at least put them on the Board of Directors.  Our biggest sales year was three-quarters of a million dollars, which doesn’t seem like much to many of the business folks I know, but, for a company that never accepted a dime of venture capital, it was plenty impressive to us.

Then I went back to working for other folks.  My current job is the second I’ve had since those days.  And, right now, I won’t lie: I’m starting to miss the freedom I had when I ran my own shop.  Constantly working for someone else can sometimes make you feel like you’re working hard to make other people rich.  Plus it’s twice as frustrating when they won’t even listen to you on how to get rich.

But of course being the boss was no picnic either.  Feeling that responsibility for everyone’s livelihood is pretty scary.  There’s the simultaneous pressure to do things yourself to make sure they get done right, and the pressure to teach other people how to do things so there’s a backup, and the pressure to learn how to delegate simply to keep your sanity intact.  There’s never anyone to go to to ask what to do next: everyone comes to you and expects you to have that answer.  You work harder than everyone else, you put in longer hours, and you often pay yourself less (at least I did).  Moreover, you expect more from yourself than you do from everyone else, and everyone else returns the favor by expecting more from you too.  It can be very stressful, which impacts not only your mental health but your physical health as well.

And, then again, there are certainly a lot of upsides.  I don’t mind failing.  What I can’t stand is being forced to fail.  When you work for someone else, particularly in the world of publicly-owned corporations, you end up doing a lot of things you know are not going to work.  You tell people they’re not going to work.  You send emails and hold meetings and write reports explaining that they’re not going to work.  Then your boss makes you do them anyway.^*  Then they don’t work.  And what satisfaction is it to say “I told you so” when the answer is “yep, you were right; now go clean up the mess.”  At least when you work for yourself, your mistakes are honest, your own, and you can do your damnedest to make sure you only ever make them once.  Not that you always succeed at that, of course.  But then, if you fail, you have only yourself to blame.  And I’ve always felt that that was cleaner, somehow.  I personally never got hung up blaming myself.  I give myself permission to make mistakes.  But when I have to blame someone else because there was nothing I could do about it, that feeling of helplessness makes me a little crazy.

So neither position—being the boss or being the worker bee—is perfect.  They both have their good days and bad days.  Perhaps one day I’ll go back to running my own business.  And perhaps, after that, I’ll go back to working for someone else again.  No reason I have to stick with only one or the other the rest of my life.  I suppose I’ll probably complain either way, some days.  But I also try to look at the bright side, whenever I can.  Right now, I’m working for a company owned by a company owned by another company.  I have five levels of bosses from three different companies, and that’s just counting the ones who are in the same building as I am.  There’s more floating around in other buildings, in other cities.  But, you know what?  They’re in charge.  All the responsibility and all the stress is on them.  And I’m happy enough with that.

For now.


* This is not to imply anything negative about my boss, who is an awesome guy and generally doesn’t do that sort of thing to me.  But sometimes these types of things just come down from above.

Sabbatical Report: New Year's

Welcome to Sabbatical Report #8; for explanations, you may want to read Sabbatical Report 1.

Moving on from Christmas to New Year’s.  This year we went to celebrate with our Sister Family.  They have a cool idea about how to deal with celebrating New Year’s in California with small children: you just celebrate East Coast New Year’s.  So, at 9pm, we had some champagne and sparkling cider and then went outside and made lots of noise and then, for the most part, went home and went to bed.  Except we watched Looper first, which was fairly awesome.  Joseph Gordon-Levitt is starting to become one of my favorite actors, and this is his second pairing with writer/director Rian Johnson, after the insanely good Brick.  But I digress.

I also returned to work, which was sort of anti-climactic after six weeks of sabbatical.  Other than that, all we did family-wise was a half day at Magic Mountain (where our middle child rode his first roller coaster with a loop in it) and a Heroscape game day, which we actually just returned from.  The two boys and I, plus the elder son of our Sister Family, enjoyed a fine day of killing each other via plastic soldier proxies, while the womenfolk stayed home and had a relaxing testosterone-free day.

My final numbers for sabbatical goals (see Sabbatical Report 3 if you don’t know what I’m talking about):  I’m declaring my todo tasks a failure, with only 15 out of 25 expunged.  However, I cleaned 455 (out of 500) emails out of my Inbox, and I think that’s plenty good enough, and, for project hours, I hit my goal exactly: 75 hours on personal projects, many of which I made excellent progress on.  So I’m pretty jazzed about that.

Well, this is the last of the sabbatical reports, so next week we’ll be back to normal posts.  Which of course you should continue not to read.  But apparently my warnings are going unheeded, so I suppose I’ll see you in seven days.