Work work work

This weekend I’ve been working on another big project for $work, and I’ve had no time to put together a proper post.  It’s been a fun project, though, so I’m not complaining.  I’ve gotten to remove quite a lot of code—nearly 5,000 lines—and replace it with only half as much.  I love doing stuff like that: simplifying, cutting away the dead wood, generally tidying up.  Less code means less for my fellow developers to have to wade through when they want to fix things or change things, and it reduces the overall cognitive burden of understanding our codebase.  Things like that make me happy.  I’m a man of simple tastes.

Anyways, I’m sorry that I didn’t have anything here for you to read.  But I’m sure you can find other things to entertain yourself in this wild wondrous place we call the Internet.  Good luck.

Series Listing: The Other Blog

I have two blogs: this one, and a technical, Perl-based blog.  When I refer to “my Other Blog”, it just means whichever one you happen not to be on at the moment.  I occasionally do series over on the Other Blog as well, so here’s a listing of those.

YAPC Reports

The first one of these was actually posted on this blog, as it was written before The Other Blog was created.  The remainder are over there though.

These are not terribly technical; mostly what I talk about are traveling, the social interaction, dragging family members along for the ride, etc.

• On Conferences and Travel (YAPC 2011: Asheville)
• YAPC Reflections 2013 (YAPC 2013: Austin)
• Remnants of a Deeper YAPC (YAPC 2014: Orlando)
• Breaking the Fourth YAPC (YAPC 2015: Salt Lake City)
• The Fifth Element (of YAPC) (YAPC 2016: Orlando again)

Perl and Me

This is probably the series that I’m proudest of (and the one that more people have actually read than any other, I would guess).  It’s moderately technical, in spots, but it also contains a lot of philosophical ponderings and more abstract points.

• The Road So Far
• The Power of OOP
• A Møøse Once Bit My Sister
• A Worthy Program, Exceedingly Well Read
• Speaking with the speech of coders
• Perl is Engineering and Art
• The Most Powerful Weapon Which You Can Use to Change the World
• Endless Forms Most Beautiful and Wonderful
• That’s Why I Failed Recess
• What We Talk About When We Talk About DWIM
• Please Mr. Perl, Will You DWIM?
• The End of the Beginning
• Here’s to Future Days

Not part of the series proper, but perhaps a fun read nonetheless: this post (not on the Other Blog) discusses the origin of all the titles I chose for this series.

• What Morris Wrought

A Date with CPAN

This describes my attempts to design a new Perl module to make dealing with dates easier for most Perl scripts.  It won’t work for every possible use of dates, but it should suffice for the most common cases.  It’s more technical than “Perl and Me,” but I think it has value for just about anyone who is a programmer, even if Perl is not their primary language.  It has a lot about general design that will apply in any language, and a lot about dates that is also language-agnostic.

• State of the Union
• Target First, Aim Afterwards
• Paving While Driving
• Construction Time Again
• Everything’s Relative
• Time Won’t Give Me Time
• In the Zone
• (commercial break) And Now, a Word from Our Sponsor
• Curse You, Daylight Savings Time!
• Composition Defeats Inheritance Yet Again
• Cleanliness Is Next to Timeliness
• Sweet Release

Another working weekend

I started on a blog post this week, but I never got the chance to finish it.  I’ve been working on a particularly thorny problem for $work.  Well, it’s particularly annoying to test in any event.  I have to build two different machines simultaneously, but they have to coordinate with each other at various points as well.  An interesting problem, and most of it is solved, but it’s still giving me more trouble than I’d expected.  So I’ll have to finish the post later this week and throw it up for you next week.

In the meantime, I’ve got to get back and iron out the last lingering issues in this script of mine.  It’ll annoy me no end if I can’t figure it out before it’s time for bed.

Thanksgiving fallout

Well, Thanksgiving was lovely.  We had plenty of deviled eggs and cornbread stuffing with cranberry sausage and we sat around and told each other what we were thankful for.  The 2-year-old happily proclaimed she was thankful for “DINNER!!”  The rest of the holiday weekend has been predictably lazy.

While I haven’t done the writing necessary for a full blog post, I have managed to eke out a few bits and bobs.  I replied to a comment on one of my old Perl blogs, and, if the Time Gods are willing, I’ll yet have a chance to post on my Heroscape forum about the marthon game my kids and I just finished up yesterday.

If you are visiting my page from the US, I hope you had a lovely Thanksgiving.  If you’re from a different country, I hope you had a lovely ... Thursday.  And Friday.  And whatnot.

What Morris Wrought

So, this week I’m going to talk about the titles I came up with for my 13-part blog series on my relationship to Perl that I did on my Other Blog.  When you do a long series like that, you have a number of challenges: presenting the topic concisely, laying the groundwork for the following week, the simple grind of cranking out the next 1500 words.  But there’s also the issue of coming up with titles.  Naming things is hard.  In my technogeek life, it’s probably the thing that we fight most about.  In fact, there’s a famous quote we’re wont to trot out at times:

There are only two hard things in Computer Science: cache invalidation and naming things. —Phil Karlton

Sometimes you see people online wondering why this saying is famous: naming things is easy, they say.  These are invariably young programmers who have never had to deal with users who can’t understand why a feature doesn’t (or can’t) work because they’re confused about what it is because it’s so poorly named.  Or the pain of having to use a word in one sense when talking to sales (because they use the industry standard definition) and a different sense when talking to fellow techies (because they use the literal meaning) and an altogether different sense when talking to management, because they use a completely arbitrary defintion that they got from the guy before the guy before the guy before you, who was invariably a young programmer who didn’t understand that naming things is hard.

So, yeah: coming up with good names for things is hard.  Coming up with consistent, good names for things is harder.  Coming up with consistent, good names for things 13 weeks in a row is very difficult indeed, and so hopefully I can be forgiven for doing only a mediocre job of it.

The first two or three came to me fairly naturally, and they established the pattern: quotes, either direct or paraphrased, that referenced different cultural things.  These might be songs, poems, television shows, movies, quotes by famous people, or whatever.  Several of them were as easy as the first few; some of them were so hard that I almost spent longer searching for a good title than I did writing the post in the first damn place.  Some of them are so obsure I don’t expect anyone else to know what the hell I’m on about; some were obscure enough that I didn’t know them myself until I Googled them for the purpose of the series.

Here’s the 13 titles I came up with, along with the hints I gave out last week.  Honestly, some of the hints are fairly obscure as well, but I didn’t want to make it too easy.

1. The Road So Far: a Winchester recap
2. The Power of OOP: Johnny Colla would have done a mean sax solo
3. A Møøse Once Bit My Sister: I apologize for the obscure references; those responsible have been sacked
4. A Worthy Program, Exceedingly Well Read: also, profited in strange concealments ...
5. Speaking with the Speech of Coders: a present from Vietnam
6. Perl is Engineering and Art: what’s to learn? it’s a snake ..
7. The Most Powerful Weapon Which You Can Use to Change the World: according to Tata, not Perl at all ...
8. Endless Forms Most Beautiful and Most Wonderful: there was grandeur in his view of life from the Beagle
9. That’s Why I Failed Recess: it was funnier when Rudy said it to Fat Albert
10. What We Talk About When We Talk About DWIM: involving two couples and a bottle of gin
11. Please Mr. Perl, Will You DWIM?: a plea to m’colleague Hugh
12. The End of the Beginning: once described as “sounding more like the Primitives than the Primitives”
13. Here’s to Future Days: why are they called “twins” if there’s three of them?

Now let’s look at which each one references, as well as discussing its relevance to the particular post it ended up tagging.

The Road So Far

This is what they put on the title card when they do a longer recap on the TV show Supernatural.  The card looks like this, or maybe like this.  The protagonists of the series are the Winchester brothers, thus this is “a Winchester recap.”

This was a fairly natural choice for the first post in the series, which told a highly abbreviated version of my programming life, from age 14 or so, up to the present.  It’s a cool reference if you get it, but it still works well if you don’t.

I think a lot of people think of Supernatural as a teeny-bopper series, probably because it’s on the CW along with other teeny-bopper series like Gossip Girl, or The Vampire Diaries.  Of course, I was watching Supernatural when it was on the WB ... which was the home of Charmed and Dawson’s Creek, so I suppose I’m not digging myself out of that hole very well.  I dunno; I suppose it is a teeny-bopper series in many ways, and it’s probably gone on far beyond when they should have called it quits, but I still enjoy it.  Call it a guilty pleasure.  Besides, every now and again Felicia Day shows up, and that just makes it all worthwhile.

The Power of OOP

My second post in the series was about object-oriented programming, or “OOP” for short, and what makes it so useful.  So it seemed natural to harken back to Huey Lewis & the News’ classic 80’s song, “The Power of Love”.  The hint refers to the great sax player of the News, Johnny Colla (who was also a co-writer of “The Power of Love,” as it happens).

I’m not actually a huge fan of “The Power of Love,” nor its companion piece “Back in Time,” both off the Back to the Future soundtrack.  As far as I’m concerned Lewis & the News peaked with Sports, and it’s all downhill from there.  By the time Huey was declaring that it was “Hip to be Square,” I was embarrassed to admit that I’d ever seen them live.  (But I did, with Stevie Ray Vaughan and Double Trouble opening, and it was a great show, I gotta tell ya.)

A Møøse Once Bit My Sister

No self-respecting programmer should have missed this one, which is of course is a reference to the ultra-classic Monty Python and the Holy Grail.  As you probably know, all the credits of the film are at the begining, and the Pythons couldn’t let it get too boring, so they peppered it with lots of moose references (for whatever reason).  The title is a direct quote from the credits, and the hint is a paraphrased version of a later credits quote.

For a post extolling the virtues of Moose, but also lamenting a few of its warts, there was no way I could pass up this title.

A Worthy Program, Exceedingly Well Read

This is one of the ones I spent a lot of time trying to find a good reference for.  The post was about legibility: the idea that a good program should be able to be read like a good story.  After several fruitless Googles, the phrase “well-read” popped into my head.  I wondered what the origin of that phrase was.  Of course, if you’re a native English speaker and you spend any time at all poking at the origins of common phrases, you know what the answer is 80-90% of the time: Shakespeare did it.

As it is here.  I paraphrased the relevant bit for the title, and I used the surrounding context for the hint.  Here’s the full text, from Henry IV, Part 1:

In faith, he is a worthy gentleman,
Exceedingly well read, and profited
In strange concealments, valiant as a lion
And as wondrous affable and as bountiful
As mines of India.

This is Mortimer speaking about Glyndwr, whoever that is.  I never read Henry IV, personally.  Still a good quote though.

Speaking with the Speech of Coders

Every once in a while we Americans wake up out of our egocentricity and remember that not all our blog post readers share our Western heritage.  By this point in my blog series, I felt it was time to pick a reference from the other side of the world.  I spent some digging through the Tao Te Ching, which is normally my go-to source for pithy quotes from the Orient.  I poked around The Art of War and Hagakure, both of which I also like, but they weren’t very helpful for this post, which was about linguistics.  I think I even explored the Analects briefly, but I lean much more towards Taoism than Confucianism, as you might imagine of one so obsessed by balance and paradox.

Then suddenly, after long and futile searching, it hit me: I already had a great source which would be perfect for this.  “The Red Cockatoo” is a short poem by Chinese poet Po Chu-i (also romanized as Bai Juyi), who lived in the Tang Dynasty and is very popular in both China and Japan (at least according to his Wikipedia page).  There are several different translations, but I prefer the one by Arthur Waley, the great British sinologist who gave us excellent translations of both the Tao Te Ching and the Analects.  Here it is in its entirety:

Sent as a present from Annam
A red cockatoo.
Coloured like the peach-tree blossom,
Speaking with the speech of men.
And they did to it what is always done
To the learned and eloquent.
They took a cage with stout bars
And shut it up inside.

Beautiful, and piquant.  The hint refers to the fact that “Annam” is an ancient Chinese name for Vietnam (or part of what is modern Vietnam).

Perl is Engineering and Art

This one was obvious to anyone who read this particular post, which spent a good deal of time analyzing a sidebar from the O’Reilly book Learning Python entitled “Python is Engineering, Not Art.”  I almost didn’t use this title, actually, as it’s so much more obvious than all the rest.  But then I decided that this title was just too good to pass up.  The hint is obvious as well, or at least is so in hindsight.

Fun side note: the animal on the cover of Learning Python is a rat.  Write your own joke here.

The Most Powerful Weapon Which You Can Use to Change the World

Another tough one to title.  This post covered several different subtopics that didn’t really fit anywhere else, so there wasn’t a great choice for a title anyway.  One of the topics I covered was my school experience with programming, so I started looking for quotes on education and ran across this one by Nelson Mandela:

Education is the most powerful weapon which you can use to change the world.

There’s a bit of contention on whether he actually said this or not (and whether he used the word “which” in it if he did), but overall it seemed solid enough.

The hint refers to one of Mandela’s nicknames: “Tata” means “father” in Xhosa.  His other nickname is “Madiba,” but some have argued that it’s inappropriate for non-South-Africans to use that one.

Endless Forms Most Beautiful and Most Wonderful

This one was a little easier.  The post was about evolution, so it made sense to peruse the words of Charles Darwin, who was not only a very influential scientist, but also an eloquent writer.  The full quote is:

There is grandeur in this view of life, with its several powers, having been originally breathed into a few forms or into one; and that, whilst this planet has gone cycling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being, evolved.

This is from the conclusion of Darwin’s seminal On the Origin of Species, and is in fact the only time Darwin ever uses the word “evolve,” in the first edition.  (And, in the second, he added the phrase “by the Creator” to make it clear what he was talking about.)

The hint, of course, is a bit of the quote above, combined with a reference to the famous ship that Darwin sailed on, HMS Beagle.

That’s Why I Failed Recess

The ninth post in my series was about Getting Shit Done, and, when I was trying to think of a title for it, I kept remembering a joke from my childhood.  As the hint suggests, I’m pretty sure the first time I heard it was on Fat Albert.  It might have been Rudy who said it, or then again it might have been Russell—he was always a smartass.  Then again, we’re talking about 40-odd years ago, so I might be misremembering altogether and it was never in Fat Albert at all.

Anyways, here’s how I remember the joke:

A: I don’t play.  That’s why I had to quit school in the third grade.
B: Whaddaya mean?
A: ‘Cause the teacher said “recess,” and I said “no, I don’t play.”

There are countless variations of this joke, including the more concise version I used for my title, used in the common venacular, multiple rap songs, blog posts by other people, Facebook user names, tweets, and Internet memes.  In fact, this is a meme from before we knew what memes were.

Plus it’s really funny.

What We Talk About When We Talk About DWIM

Along about Part 10 I wrote a post that was so damn long I had to break it into two pieces.  Originally the title of this post and the following one were going to be switched, so that the title of this one could be a callback to the mention of “m’colleague” which I had dropped into the text.  (Instead, I ended up using that for the hint for Part 11.)  But eventually I made the switch to the titles that we have now because it just made better sense: this post was a fairly long digression in the form of a story from my college days, and this title fit that perfectly.

The title, of course, is a paraphrase of the title of a famous short story by Raymond Carver, “What We Talk About When We Talk About Love,” as well as the book which contains it.  In the story, two couples talk about everything but love over a bottle of gin (thus the hint), but really love is all they’re talking about.  You see the parallel in my post.

Really, though, I’m not a huge Carver fan.  The best thing about “What We Talk About When We Talk About Love” is probably the title.  “Cathedral” is better.

Please Mr. Perl, Will You DWIM?

If you are a connoisseur of Britsh comedy, the television series at the very top of your must-see list is of course Monty Python’s Flying Circus.  After that, it should be The Young Ones and Blackadder, although we might quibble over which one should come first.  Next on your list, before Fawlty Towers, before Red Dwarf, and, yes, even before AbFab, should be A Bit of Fry & Laurie.  If you think of Hugh Laurie simply as House, or (even worse) as the insipid father of Stuart Little, you really don’t know Hugh Laurie (in fact, you may not even realize he’s British).  Likewise, if all you know of Stephen Fry is his voice—he’s the Cheshire Cat in the Tim Burton version of Alice in Wonderland, the narrator of Little Big Planet, and a prolific audiobook narrator, including the UK version of the Harry Potter books and The Hitchhiker’s Guide to the Galaxy—you’re missing out.

A Bit of Fry & Laurie is at once similar to Monty Python and also removed from it.  There’s still a certain amount of the surrealism (perhaps a bit less), but very little of the physical comedy such as the Ministry of Silly Walks or the Gumbys.  Most of it was like taking the best verbal humour of the Pythons (such as the Argument Clinic, or my all-time favorite, the penguin on top of your television set) and cranking it up to 11.  Stephen Fry would often do the heavy lifting in such sketches—playing the Groucho, or the Abbot, role—but Hugh Laurie had many talents other than just being an outstanding straight man.  One of which is an amazing range of musical ability: he plays guitar, drums, harmonica, sax, and, of course, piano.  At the end of every show, Fry would turn to Laurie (who he often referred to as “m’colleague”) and say: “Please Mr. Music, will you play?”  To which Laurie would respond by playing the piano in a loungy sort of way, usually while Fry mixed ridiculously named cocktails such as the Swinging Ballsack.  Occasionally he would elaborate the phrase to enhanced levels of flowery silliness; my favorite of these was:

I say, as I like to on these occasions, those six refreshing words that unlock the door to sophisticated evening happiness. I say: Please Mr. Music, will you play?

If you’ve not yet had the pleasure, I highly recommend it.

The End of the Beginning

Here at Part 12 I finally decided to start wrapping things up.  However, I knew it would take me (at least) two posts to conclude satisfactorily, so I needed a title to reflect that.  “The End of the Beginning” is (appropriately) the final track on the sophmore album of the Darling Buds, Crawdaddy.  Although Crawdaddy came out in 1990, it definitely has that late 80’s sound, including a remarkable similarity to the Primitives, particularly their first two albums Lovely (‘88) and Pure (‘89).  Although technically speaking the Primitives were English while the Darling Buds were Welsh.  But to us stupid Americans that subtle distinction is lost.

Although it was a Brit who made the comparison I reference in the hint: specifically, Dave Kendall, creator of MTV’s 120 Minutes.  He made the clever observation in his review of Crawdaddy, and I couldn’t help but agree, even though I probably like the Darling Buds a bit more than the Primitives.  But it’s a close thing.

The first track on Crawdaddy, “It Makes No Difference,” has one of the coolest hooks of the 80’s.  Too bad you’ve never heard it.

On the other hand, if you want to hear this track, YouTube is your friend.

Here’s to Future Days

And finally we reached the end, and I decided to touch on my thoughts about Perl’s future.  The title for this one took absolutely no thinking or searching at all.  While there can be no doubt that Into the Gap is the pinnacle of the Thompson Twins’ career, Here’s to Future Days is also a great album, the last of the good TT records before they transmogrified into Babble (whose debut was better than the last three efforts from the Twins put together ... not that that’s saying much).

Here’s to Future Days was also (probably not coincidentally) their last album as a threesome: it may not have seemed like Joe Leeway was adding much other than standing around looking cool (much as Andrew Ridgely did for Wham!), but apparently that was an illusion, because they sure sucked without him.  Definitely most people think of the Thompson Twins as a trio, and wonder what’s up with calling themselves “twins.”

But of course the truth is the name has nothing to do with the number of band members.  The first (little known) TT album was recorded with four members, and the second featured a whopping seven, before they trimmed it down to the famous three, who would go on to produce the Twins’ three great albums: Quick Step & Side Kick (known simply as Side Kicks in the US), Into the Gap, and Here’s to Future Days.  Nope, the name was simply a reference to Thomson and Thompson, the detectives from The Adventures of Tintin who only look like twins.

“Future Days” is the track on this album that contains the lyrics “Here’s to future days / Here’s to future ways,” which is what I hear in my head whenever I read this title.  If you’d like to have it stuck in your head as well, YouTube can arrange that for you.

In Conclusion

Perhaps unsurprisingly, I spent quite a bit of time mentally wrestling with a title for this post itself.  Should it be some sort of self-referential thing, being that it would be the title of a post about titling posts?  Should it somehow proclaim to the world that it was a meta-title?  Should it be a quote about naming things, or about clever wordplay?

In the end, I decided to make it a shout out to one of my favorite book-gifts as a child.  I got my fair share of fiction, certainly, but my family also recognized that an aspiring writer must have a love of language, so I got a fair number of dictionaries, thesauri, etc.

I was eleven years old on Christmas in 1977, the year that my grandfather presented me with the Morris Dictionary of Word and Phrase Origins, which had been published for the first time that very year (although much of it was derived from the earlier version, which was similarly titled but without the “Morris”).  It’s a “dictionary” only in the sense that the entries in it are alphabetized.  Lovingly crafted by husband and wife William and Mary Morris, it’s not so much a reference work (although it can be used as such) as it is a mishmash of fascinating tales of how English expressions came to be; I was fond of just opening it to a random page and reading whatever I found there.  I was rarely disappointed.

The Morris’ youngest son Evan carries on the family tradition on the web, writing as the Word Detective.  On his “about” page, he quotes fellow etymologist John Ciardi:

The more words I traced back through time for our readers, the more I appreciated Ciardi’s observation that each word, no matter how humble, was “a miniature fossilized poem written by the human race.”

And that’s what this exercise in naming was like: a verbal archaeology expedition, a paleontologist finding words trapped in amber.  My love for this sort of thing is certainly directly traceable back to the Morris dictionary, and the many hours I spent perusing how words and meanings become bent and reshaped to suit new ends across the generations.  Yeah, I was a weird kid.

So, this week’s installment, while longer than I’d anticipated (and probably longer than you’d hoped), at least may provide some insight into how these titles get here and where they come from, and why I tend to obsess over them more than is probably healthy.  Next week I probably won’t be so garrulous, most likely because I’ll be busy catching up on all the things I didn’t do this weekend because I spent too much time on this blog post.  But it’s been fun.  For me, anyway.  For you ... well, didn’t anyone tell you not to read this blog?

Technical Debt Strategies

For a new role I’m starting at work, I’ve been thinking about the concept that we in the software development business refer to as “techincal debt.”

In software development, there is always a tension between two opposing forces: the desire to do it fast, and the desire to do it right.  I could probably write an entire blog post on just that topic, but for now I’ll settle for the short version.  If you do it right, then, later, when you want to extend it, or modify it, or use it as a jumping off point for branching out in a whole new direction (and you will always want to do these things eventually, if your software lives long enough), you can do so easily, with a solid foundation as a base.  The downside is that it will take longer.  If you do it fast, you get results faster, which means you can serve a customer’s needs before they change, fill a window of opportunity before it closes, or perhaps even beat your competitors to the market with your offering.  But when you have to modify it later (which you will), it will end up taking even more time to clean things up than if you’d just done it right in the first place.

You can see why we often call this “technical debt.”  You’re saving time now, but you’ll have to “pay it back” later, and the amount of extra time it takes is like the interest.  Primarily, we software people invented this analogy because it makes good sense to business people.  When you’re running a business, sometimes you need a piece of equipment.  You have two choices: you can borrow some money and buy it now, or you can save up and purchase it outright, later.  Buying it now allows you use it now, thus saving you costs somewhere else or giving you new capabilities you can capitalize on.  But you’re going to have to pay for it eventually, and you’ll have to pay the interest too.  Buying it later saves money in the long run, but denies you the advantages of the equipment for however long it takes to save up enough money.

This analogy really works because, despite what some people tend to believe, neither choice is always right or always wrong.  Sometimes it makes sense to borrow the money and buy it now; sometimes it makes sense to wait.  Likewise with software, sometimes it makes sense to just get the damn thing programmed as quickly as possible, and sometimes it makes sense to take the time and do it right the first time.  And business people have to make that call, and expressing the choice in financial terms makes it easy for them to understand the trade-offs.

For instance, when a company is first starting up, “do it fast” is nearly always the right answer.  (David Golden has a great post which explores this in greater detail.)  Basically, there’s no point in worrying about how hard it will be to change your software tomorrow when you’re not even sure there’s going to be a tomorrow.  Once a company leaves the startup phase, though, it’s time to think about paying down some of that technical debt.  If it doesn’t, the lack of agility means it may not be able to respond to customer demands quickly enough to thrive in the marketplace.

Now, technical debt is in many ways unique to every individual company, and there’s no one technique or approach that will work for everyone.  But everyone faces the same basic problem: I need my development team to keep making new features while they’re also paying off this debt; how can I have them accomplish even more than they were doing before without burning out, or creating new problems as fast as they fix the existing ones?  Hiring more people helps, of course, but it isn’t the complete solution.  But there are some general strategies that will work, and can be used by nearly every company that finds itself in the position of having a fair amount of technical debt to pay off.  These five areas nearly always need some attention, and they’re excellent things to think about when looking for ways to make your programmers more efficient, which not only gives them time to work on refactoring messy areas of your codebase, but also gives them more time and confidence to avoid creating technical debt in the first place.

Unit Tests  Few companies have no unit tests, but few have enough either.  I’ve heard people worry about creating too many unit tests, but honestly this is like your average American worrying about eating too many vegetables: I suppose it’s technically possible, but it’s unlikely to be a problem you’ll ever face.  It is possible for your unit tests to be too messy, though.  Remember: your unit tests are supposed to give you confidence.  The confidence you need to get in there and clean up some of that technical debt without being afraid you’re going to break something.  But your tests can’t give you confidence unless you’re confident in your tests.

Now, many people will tell you that you should put as much time and effort into your tests as you do your code (this is often—but not always—what people mean when they say “tests are code”).  I don’t necessarily agree with this.  I think that all the effort you put into your unit tests should go towards making your unit tests effortless.  I think your code should be elegant and concise: you don’t want it too simple, because it’s a short hop from “simple” to “simplistic.”  Imagine the Dr. Suess version of War and Peace and how hard that would be to get through—that’s what I’m talking about.  You need to use appropriate levels of abstraction, and using too few can be as bad as using too many.  You have to strike a balance.

On the other hand, your tests should be simple.  Very simple.  The simpler they are, the more likely your developers are to write them, and that’s crucial.  Whether you’re using TDD or not (but especially if you’re not), devs don’t particularly like writing tests, and anything that gives them an excuse not to is undesireable.  You want your tests to read like baby code, because that way not only will people not have to expend any brainpower to write new tests, but reading them (and therefore maintaining them) becomes trivial.  This doesn’t work as well for your code because of the crucial difference between code and tests:  Code often needs to be refactored, modified, and extended.  Tests rarely do—you just add new ones.  The only times you modify tests is when they start failing because of a feature change (as opposed to because you created a bug), and in many cases you’re just going to eliminate that test.  And the only times you should be refactoring them is when you’re working on making them even simpler than they already are.

So, in addition to trying to extend your test coverage to the point where refactoring becomes feasible, you should probably work on creating underlying structures to make writing new tests simple, and to make reading existing tests even easier.

Code Reviews  There are still plenty of companies out there not doing this one at all, so, if you fall in that category, you know where to start.  But even if you’re already doing them, you could probably be doing them more efficiently.  Code reviews are tough to implement well, and it has as much to do with the process you implement as with the software you choose to utilize.  Of course, it has a lot to do with the software you choose to utilize.  Code reviews, like unit tests, need to be simple to do, or people will find excuses not to do them.  Good software can really help with that.

Hopefully, like unit testing, code reviews are something that you don’t really need to be convinced that you should be doing.  But more people are resistant to code reviews than any other practice mentioned here.  They absolutely do require an investment of time and effort, and they absolutely do mean that you will deliver some features more slowly than you could without them.  Of course, if that extra time and effort catches a crucial customer-facing bug, you won’t be complaining.

But, honestly, catching bugs is not the primary benefit of code reviews ... that’s just the icing on the cake.  Code reviews just produce better software.  They’re the time and place where one developer on your team says to another: “hey, you know you could have used our new library here, right?”  (To which the response is inevitably: “we have a new libary?”)  They’re fantastic for cross-training too: don’t think code has to be reviewed by someone who already knows that code.  Have it reviewed by someone who’s never seen the code before and they’ll learn something.  Have your junior people reviewed by your senior people, and have your senior people reviewed by your junior people: they’ll all learn something.  It makes your codebase more consistent and builds a team dynamic.  It diffuses business knowledge and discourages silo effects.  If it happens to catch a bug now and then ... that’s just gravy.

Once you have code reviews in place, though, you’ll probably still want to tweak the process to fit your particular organization.  If you require reviews to be complete before code can be deployed, then code reviews are a blocking process, which comes with its own hassles.  Then again, if you allow deployment with reviews still outstanding, then you risk making the review process completely ineffectual.  So there’s always work to be done here.

Code Deployment  I suppose there are still software folks out there for whom “deploy” means “create a new shrink-wrap package,” but more and more it means “push to the website.”  At one company I worked at, deployment was a two-hour affair, which had to be babysat the entire time in case some step failed, and, if a critical bug was discovered post-launch, rolling back was even more difficult.  From talking with developers at other companies, I don’t believe this was an exception.

If your organization is big enough to have a person whose entire job is dedicated to deployment (typically called a “release manager”), then you don’t have to worry about this as much.  If that two hours is being spent at the end of a long day of preparation by one of your top developers, you’ve got some room for improvement.  But even if you have a seprate release manager, think how much you have to gain as your deployment gets shorter and simpler and more automated.  You can release more often, first of all, and that’s always a good thing.  And there are simple strategies for making rolling back a release almost trivial, and that’s absolutely a good thing.

Now, I’ve never been a fan of “continuous deployment” (by which I mean deployment happening automatically, triggered by checking in code or merging it).  But I do believe that you should strive for being able to deploy at any time, even mere minutes after you just finished deploying.  Part of achieving this goal is reaching continuous integration, which I am a fan of.  Having those unit tests you worked on so hard being run automatically, all the time, is a fantastic thing, which can free your developers from the tedious chore of doing it manually.  Plus, continuous integration, unlike code reviews, really does catch bugs on a regular basis.

Configuration Management  Similar to deployment, and for the same reasons.  If building a new production server is a heinous chore, you’re not going to want to do it often, and that’s bad.  Horizontal scaling is the simplest way to respond to spikes in traffic, and you should be able to bring a new server online practically at the drop of a hat.  It should be completely automated, involve minimal work from your sysadmins, and zero work from your developers.  There are all kinds of tools out there to help with this—Puppet, Chef, etc—and you should be using one.  Using virts goes a long way towards achieving this goal too.

For that matter, it should be trivial to bring up a new development server, or a new QA server.  Once that ceases to be a big deal, you’ll find all sorts of new avenues opening up.  Your developers and/or your QA engineers will start thinking of all sorts of new testing (particularly performance or scalability testing) that they’d never even considered before.  Figuring out how to handle database environments for arbitrary new servers can be a challenge for some shops, but it’s a challenge worth solving.

For Perl in particular, you also need to be able to reproduce environments.  You can’t spin up a new production web server alongside your existing ones if it’s going to be different than the rest.  If your server build process involves cloning a master virt, you don’t have to worry about this.  If you’re rebuilding a server by reinstalling Perl modules, you must expect that CPAN will have changed since you last built production servers, because CPAN is always changing.  For this reason, a tool like Pinto (or Carton, or CPAN::Mini, or something) is essential.

Version Control  In this day and age, I doubt seriously I need to convince anyone that they should start using version control ... although one does hear horror stories.  More likely though, you should be thinking about what VCS you’re using.  If you’re using CVS, you should probably be using Subversion, and, if you’re using Subversion, you should probably be using Git.  But along with thinking about which software to use and how to switch to a better one (preferably without losing all that great history you’ve built up), you also need to be thinking about how you’re using your VCS.

Are you using branches?  Should you be?  If you’re not, is it only because of the particular VCS you’re using?  If you’re using CVS or Subversion and telling me branching is a terrible idea, I can understand your point of view, but that doesn’t mean you shouldn’t be using branches—it just means you should probably be using Git.  Now, granted, branches aren’t always the right answer.  But they’re often the right answer, and they allow experimentation by developers in a way that fosters innovation and efficiency.  But it takes work.  You have to come up with a way to use branches effectively, and not just your core developers should agree.  If you have separate front-end developers, for instance, they’re going to be impacted.  If you have a separate QA department, they’re definitely going to be impacted—in fact, they can often benefit the most from a good branching strategy.  Your sysadmins, or your ops team (or both), may also be affected.  If you’ve tried branches before and they didn’t work, maybe it wasn’t branching in general that’s to blame, and maybe it wasn’t even the particular VCS ... maybe it was the strategy you chose.  Feature branches, ticket branches, developer branches—again, there’s no one right answer.  Different strategies will work for different organizations.


So these are some of the things you need to be considering as you look to reduce your technical debt.  None of these things address the debt directly, of course.  But, in my experience, they all play a part in streamlining a development team by increasing efficiency and cohesiveness.  And that’s what you really need in order to start paying down that debt.

The Care and Feeding of Code Monkeys

Welcome to part 3 of my 2-part series on how to make Agile Development actually work.  This one has to do with managing programmers, which is both completely outside the purview of Agile and yet inextricably tied to it.  (It also ran a little long, since it’s a topic that I’m very passionate about.  Hopefully you’ll stick with it.)

First, a word on nomenclature:  At my current job, I am a “software engineer.”  At my last job, I was a “code monkey.”  What I really am is a programmer.  You can dress that up however you like, but, if you primarily write code for a living, you’re a programmer, and the following thoughts apply to you.  Be careful applying it too broadly, though: folks in other tech pursuits that hardly sling code at all, like system administrators and DBAs, may not fit the mold I outline here, and folks who write a lot of code but also do a lot of other things (e.g. QA engineers who specialize in automating tests, release managers who have to help code up things like continuous integration systems) may sort-of-kind-of fall into these patterns ... or not.

And, though I say above “if you primarily write code for a living, you’re a programmer, and the following thoughts apply to you,” what I really mean is, “if you primarily manage coders for a living, you’re managing programmers, and the following thoughts apply to you.”  For the advice I’m going to give you is pretty useless if you’re the one being managed.  We typically have so little say in those sorts of things.  Which is sad, really, but a topic for another time.

Next, a note on my qualifications.  I ran my own software development business for 12 years, and I followed one simple philosophy: keep your employees happy.  I never worried about keeping my customers happy, because I found that, if you keep your employees happy—and not just content, but deliriously, ecstatically happy; happy to the point where, at the end of the day, they’d rather stay at work because it’s more fun than going home—they’ll produce such amazing software that the customer satisfaction thing just takes care of itself.  I had more than one employee tell me that it was the best job they’d ever had.  Very few of my employees ever left for any other reason than I’d run out of work for them, and nearly all the ones that left because of that ended up coming back later.  I had an employee once sit at home jobless for a year and a half because he was just waiting for me to call with more work (and also because I’d paid him well enough that he could afford to do that).  When I tell you that I know how to manage programmers, I ain’t fucking around.

And finally (at least as regards this introduction), a cautionary note.  I generalize.  I think it’s good to generalize, even about people, as long as you remember that, at the end of the day, we’re all different, and we all need to be treated just a little bit differently.  You must be careful not to fall into the trap of stereotyping.  But there’s still a lot to be gained by understanding the whole through understanding the majority (because, even though we’re all different, we’re all the same), so absorb what I’m saying here, but remember that you’re always going to encounter exceptions.

So how does one go about managing programmers?

Well, there are two things that you must remember.  The first I’ve already discussed in some detail, and it would be pointless to belabor it again.  I’m talking about the issue of perception vs reality.  If you’re too busy to click that link, here’s the executive precis:  Programmers spend all day talking to computers, which are insanely literal; they tend to lose their concepts of perception.  You, however, are most likely a business executive, for whom things like your perception in the marketplace, your reputation, and your perceived brand value are paramount.  This leads to a fundamental disconnect.

The second one I touched on in my exploration of what agile means: programmers are craftsmen.  Very few people get this.  Most folks think programmers are all logical and mathematical, like Vulcans.  And it’s true that programmers have to have a logical component to their personality.  But some of the worst programmers I’ve ever met were mathemeticians, or mechanical engineers.  Not to disparage those professions—they’re very fine careers.  But the point is, those fields attract people who like things to fit together neatly and sensibly.  Programming isn’t neat.  It’s heinously messy.  Stuff goes wrong that no one can really explain, there are bizarre interactions between complex systems that were never foreseen, and there’s all these terribly illogical users involved in the process.  People who like things tied up in a bow get really crazy when they have to develop software.

But, even more than that, coding requires creativity.  There’s actually one backwater of mathematics that I remember from my school days which was a bit like programming: proofs.  I first learned them in geometry, which I took in the ninth grade.  A proof is a way to chain together different rules: this rule leads you to that rule and thence to the next, and so on.  To chain these rules together requires a very logical mind.  But you also need to have a sense of where you’re going, and that’s where the creativity comes in.  You need to be able to make a creative leap to the end first, and then be able to logically work your way there from the beginning.  If you start a proof without a clear idea of where you’re going, you most often end up right back where you started.

And so it is with programming.  That creative leap is absolutely crucial to being a top-notch programmer.  Without it you’re just a monkey pulling wrenches.  If more people understood this, the world (or at least the business world) would be a better place.

Because I think that the business world already understands that you have to manage creative people differently.  Advertising agencies, for instance, routinely hire artists, and writers, and people whose sole job is just to come up with ideas.  They’ve figured out that you can’t treat these people in the regimented fashion that you do for other workers—giving them dress codes, and set hours, and making up weird little rules about what they can and can’t put on their cubicle walls.  (Not that I’m saying that those are good ways to treat anyone, really.  Just that trying to treat creative people in such a fashion is even more doomed to failure.)  The Internet culture that came out of California in the ‘90s (and in particular out of Silicon Valley) has gone a long way towards fixing this for programmers.  There are many businesspeople who now understand that programming can be fun and they need to encourage their employees to have fun while doing it.  (That attitude is still more prevalent on the West Coast than the East, in my experience, but we’re getting there.)

But what I’m not sure of is whether the business folks realize why they need to promote the culture of fun.  See, there are many jobs where you don’t really need to think very hard to get the job done.  You just do them.  They’re mostly easy, and a bit repetitive, and, once you’ve done them two or three times, you can practically do them in your sleep.  Again, this is not to denigrate any of those jobs, or any of the people who enjoy them.  I can certainly appreciate a job where I can let my mind do its own thing all day while my body churns out work.  That’s a cool gig.  But that’s not my job.

My job doesn’t just require me to think; it requires me to innovate.  Every day, in a thousand little ways, I make tiny little creative leaps (and sometimes, if I’m lucky, a few giant creative leaps).  In order to do that, your mind has to be relaxed, and clear.  Focussed is good, sometimes, but other times stepping away and focussing on something else entirely (like a rousing game of ping-pong) is exactly what the doctor ordered.  The one thing you absolutely cannot do is just plug away at it until it works.  That takes forever, and ususally produces a mess.  You need to be able to distract yourself, or maybe even just close your eyes and lean your head back and ponder.

Note that I say you need to be able to distract yourself.  Having other people distract you doesn’t help.  In fact, it’s horrible.  Just ask Samuel Taylor Coleridge.  Programming is brief periods of seeming inactivity (waiting for inspiration to strike) followed by intense bouts of insane productivity (where you put the inspiration down on paper, or in this case on screen).  This is why programmers always want to work from home.  Other people can distract you at home occasionally (particularly if you have a family and if you don’t sit them down and go over the “rules”), but in general the phone doesn’t ring, and people don’t wander by your desk for a quick chat, and there are no meetings.  I know that you, as a businessperson, value meetings.  You think that if you spend all day in meetings, you’ve actually accomplished something.  We programmers are different.  Our job is to write code.  You can’t write code in a meeting.

And the business world (even the California business world) is reluctant to let programmers work from home.  It’s changing, a bit, but still there are too many classically trained managers out there who believe that, if they can’t see you working, you must not be working.  This is 100% a trust issue.  I’m a programmer.  I’ve been a programmer for 25 years—that’s over half my life (actually, I’ve been a professional programmer for over half my life; I’ve been a programmer for about two-thirds of it).  I want to program.  You want me to write code, and that’s what I want to do.  If you let me go home, you know what I’m going to do when I get there?  Program.  Programming has got be one of the few jobs in the world like this.  When you’re a programmer, you relax after a hard day of writing code by writing more code.

So you don’t have to come up with strategies for making me do what you want me to do.  What you need is strategies to make sure nobody stops me from doing it.  Reduce my interruptions.  Reduce my meetings (agile helps with this).  Reduce my emails, if you can, and reduce the people who just pop over to get my opinion on something.  Stop telling everyone that they should not send me an email because it’s better communication if they come talk to me face-to-face.  It may be better for you, but it’s death to me, and to my productivity (and isn’t that what you’re striving to maximize?).  I want fewer emails too, but given the choice between a communication that I can attend to when I was going to take a break anyway and one that blows my concentration and makes me drop the threads I was juggling in my feverishly inspired brain, guess which one I’m going to take?

Now, it’s true that you also have to find a way to focus me.  Programmers, like magpies, are notoriously easy to distract with shiny things, like new computers, new software, new web sites, and discussions about geeky hobbies like roleplaying and MMORPGs and Renaissance Faires.  And, along those lines, we can get off track, chasing down something that seemed like a good idea at the time, but in retrospect turned out to be a giant temporal black hole.  Getting back to how agile addresses these issues, that’s what “daily scrums” are for.  It’s the chance for the programmer to say “well, yesterday I spent all day looking for a decent Javascript beautifier that would unravel this minimized code I’ve been staring at for the past week,” and the chance for someone else on the team to say “dude, I got one back at my desk; I’ll email it to you.”  There you are: problem solved, distraction eliminated, yak shaved by proxy.  Or perhaps they’ll describe the problem they’re stuck on and someone else will say hey, I’ve solved that before, I can help.  Whatever.  The point is to keep the coder on track and working towards the solution.

And, as I also mentioned in the agile theory post, we programmers are tinkerers, and it’s the responsibility of the business to help us from getting lost in that and never finishing anything.  You have to be careful not ot take it too far of course, because some amount of tinkering is necessary to keep things running smoothly.  It’s a careful balance, but you’ll get the hang of it.  Give us some rope, and be prepared (but not anxious) to tug us back if we range too far.

But, other than that, there’s actually very little you have to do to get us to produce the software you want.  In fact, the more you do, the more likely you are to make it worse.  Programmers react to micromanagement by slowing down: you want to focus on trivial things? fine, I’ll focus on trivial things.  Programmers react to attempts at measurement by getting picky and measuring everything in sight.  Programmers react to bonuses based on lines of code by writing unnecessary code, they react to bounties for finding bugs by adding more bugs, and they react to having to meet detailed performance goals by adhering to the exact letter of the goals and completely ignoring the spirit.  Businesspeople tend to find this more than annoying: they find it downright insubordinate.  But think of it this way: first, a programmer is prone to taking things very literally (talking to computers all day, remember?), and secondly you actually pay them to find the most efficient solutions to thorny problems.  Don’t get ticked off if they end up turning that ability against you when you try to control their workday.

In fact, getting the most out of programmers generally amounts to not pissing them off.  Not treating them with respect, or not trusting them, pisses them off.  Asking their opinion about something and then ignoring it really pisses them off.  Asking them to produce shoddy software because you’re in a hurry makes them livid.  Expecting them to do a lot of things at once and not understanding the simple fact that shit takes time (and also that nine women can’t make a baby in one month) makes them rethink their choice of employer.  Agile will help you with a lot of this.  Some of it, though, you just have work out on your own.

And it ain’t hard.  We programmers are simple creatures, really.  You’re just not used to us because we’re different from anything you’ve ever had to manage before.  And here I’m going to be presumptuous and move from giving you advice on programmers to giving you general advice on management (but, remember: I’ve actually managed people, even people other than programmers).  Robert Heller, a well-known writer on the topic of management, once said:

The first myth of management is that it exists. The second myth of management is that success equals skill.

Now, like any good quote, that’s going to mean different things to different people.  But here’s what it means to me:

The first sentence merely means that most people treat management as an active thing, whereas they should be treating it as a passive one.  Voltaire (supposedly) said: “The art of medicine consists in amusing the patient while nature cures the disease,” and the same applies to management.  Much of the time, the best way to succeed is simply to get out of the way.

The second sentence means that you must be careful not to let yourself get caught in a feedback loop.  If you undertake a style of management which is actually very horrible, and your employees succeed despite you, you may believe your ideas are validated.  This is important to understand, I believe, because it answers of the question of “why haven’t people figured all this out by now?”  If you’re a business executive reading this post and shaking your head and saying “man, this guy is crazy!” then I urge you to go find a programmer and give it to them to read.  They will no doubt nod and mutter “oh, yeah, that’s so true” under their breath the whole time.  The fact is, nothing I’m saying here will come as any surprise to any programmer working in America today (and probably not to the ones working in other places either).  And yet business has managed to go 50 years without figuring it out.  How?  Are businesspeople stupid?  No, of course they’re not.  But management, like medicine, can seem like it’s working even when it’s doing more harm than good.  Remember leeches?  People thought they worked too.  Those people weren’t stupid either.  But, as it turned out, leeches were a pretty bad idea.

So rethink your strategies with your employees, particularly if they’re programmers.  Don’t stand behind them and push.  The definition of the word “lead” implies that you have to be in front.  And not pulling either: just striding ahead, with confidence.  (And a few donuts wouldn’t hurt either.)  You don’t even have to look back that often.  If you are a business with hard problems to solve, we’ll be following you.  After all: solving hard problems is what we live for.

Pitfalls in Agile

[This is part 2 of a series of posts on Agile Development.  First you should decide if you care about that at all.  Then, if you decide you do, you should make sure you’ve read what I wrote last week about what agile means.]

So now we know what the attitudes towards agile should be.  But, as I’ve said, often they aren’t.  Sometimes it is that tech are the ones who don’t get it.  But I’ve found in my experience that it’s more often business.  Tech, after all, is used to not always getting what they want.  But business is the driver of a business—it’s right there in the name, after all.  What business needs, they get.  They’re not used to hearing “no.”

But here’s a crucial fact: agile did say to business “you’re not going to get everything you want,” true.  But that’s not the same as saying “no.” What it’s saying instead is, “you must make a choice.”

In 1975, Frederick Brooks was considered a revolutionary for pointing out that nine women can’t make a baby in one month.  When I said that in a meeting recently, I got nervous titters and a few frowns.  Yes, that’s right: even after forty years, people are worried that I might be talking about something vaguely related to sex instead of recognizing that I’m quoting The Mythical Man-Month, possibly the most seminal work on software management ever published.  How is it that there is a single person in software development who doesn’t know this work, or at least the general concepts of it?  MMM is whence cometh Brooks’s law, which states simply that “adding manpower to a late software project makes it later.” Surely if you’re in the business of software development—from either side—these are concepts you need to understand.

But the point that both Brooks and agile are attempting to make is that we don’t say “no.” We must instead reframe the question.  If you need to buy two items that each cost $100, and you have $100 in your budget, you don’t say “no” to one or the other—you simply choose which one you will buy now, and which you will buy later.  In fact, allocation of limited resources is something that business does extremely well.  This is because of the numbers thing again.  Juggling numbers and balancing expenditures is the true forté of business.  Agile is not attempting to deny business’s requests.  It is reframing the question in terms of prioritization of one of the scarcest resources business has: the time and effort of its tech department.

Because tech can’t possibly do everything at once.  It can often do many things at once (depending on the size of the team), but it shouldn’t come as a surprise to anyone that coming up with ideas is easy and implementing them in the real world is hard.  Which means that business will always have more for tech to do than tech can possibly complete.  Which, in turn, means that business has to choose which things are the most important.

This should not be a radical concept.

And, yet, in every corporate environment I’ve ever worked in (and I dare not even exempt the business I ran myself for 12 years), business tells tech that every project is “the most important one.” Perhaps not all the time—perhaps not even most of the time—but always there comes that time when somehow it seems rational for business to just expect tech to do it all at once, without exceeding the budget, without missing the deadline, and without any one project suffering or dragging down the others.

But agile exposes the fallacy in this.  Agile gives us the concept of the “Iron Triangle.” One side of the triangle is time.  Another is functionality.  And the third is quality.  And, what agile tells us, quite explicitly, is that, if business will not move the deadlines, and will not accept any redcution in functionality, there’s only one other place where reductions can be made.

In fact, the whole point of agile is to institute a negotiation over functionality: the concept of “if you want to add something to this iteration, you have to take something out” is exactly that.  The time is not negotiable in one sense, because the dates of the iterations (often called “sprints”) are fixed.  But agile addresses time as well, by forcing regular releases before the final deadline of the whole project.  In this way, business gets value earlier, which mitigates potential losses should the deadline have to be pushed back.  And the end result is that tech can put in the quality that they wanted all along.

Where can this all go wrong?  Several places, but one of the big ones is with the Product Owners.

Agile is designed to work with small teams.  If you have a big team, you divide it up into several small teams.  These are tech teams, but they’re led by a person called a Product Owner, or just “PO.” The job of the PO is to be the liaison between business and tech.  That is, the job goes something like this:  First the PO goes to the business and say “What do you want?” Business tells the PO all the things it wants (of which there are likely very many).  The PO nods sagely and goes back to tech.  “Here’s what the business wants,” they say.  Tech promptly objects: it’s too much, or it doesn’t consider this challenge or that hurdle, or it will result in this drop in quality or maintainability, or it conflicts with this other piece of the system over here.  In this conversation, the PO represents business: “If you don’t want to give them this,” they’ll say, “then you better tell me why.  And use small words, and hard numbers.” After some back and forth, tech gives in on a few points, but stands firm on others.  Once the PO has pushed all they can push, they go back to business, and now they’re representing tech.  “The good news,” they say, “is that you can have this and this.  The bad news is that you can’t have that and that, or you can’t have it right now, or you can’t have it in that form.  And here are the numbers that demonstrate why.” Business loves this, because they love having numbers to analyze.  That makes it all make sense to them, as opposed to vague technical discussions of efficiency and flexibility.  “Okay, we’ll grant you that one,” business may say, “but this one over here we just can’t give up.” And the PO responds: “Tell me why.  And use precise words, and outline the risks and challenges of not doing it.” After more back and forth, the PO goes back to tech and represents business again.  “They agreed to give up on this thing.  But they have to have some solution to the other thing.  Here’s exactly why we can’t just not do it.  If you say you can’t give them exactly what they want: fine.  But you have to give them something.  You’re the technical geniuses: figure it out.” Tech loves this, because now they’ve been given a challenge, a hard problem to solve, something which will require them to think outside the box and push themselves to the limit to come up with an elegant solution.

So, as you can see, the PO has to represent both sides, at different times.  So obviously the first mistake you can make when trying to implement agile is to choose the wrong people to be the POs.  Most often what happens is that the Product Managers are chosen.  This is disastrous.  They have an inherent conflict of interest: when tech says “no,” they take it personally and throw up roadblocks.  When it’s time to go back to business, they are business.  They only people they’re likely to be reporting to are their bosses, and are they really going to push hard to represent the interestes of tech when it means disagreeing with their boss?  You can’t blame them for not doing that—it’s political suicide.

And agile is supposed to be all about communicating.  When a problem comes up, you’re supposed to report it immediately, so the business is aware that there’s a potential problem.  And the person you should report it to is, of course, your PO.  What happens when the PO is also the Product Manager?  You get immediate pushback.  “Unacceptable!” you’re told.  Suddenly you’re in the exact situation that agile is supposed to avoid: you’re under pressure to agree to continue to meet a deadline that is no longer feasible.  And tech people are, for the most part, an agreeable bunch who really don’t like confrontation: usually they’ll tell you what you want to hear, eventually, just to stop the uncomfortable situation.  So now either they have to work extra hours to make an unreasonable deadline (which leads to employee burnout and your best and brightest tech resources polishing up their résumés on Monster), or they starting cutting corners (the Iron Triangle strikes again), or the deadline just plain gets missed, which was probably going to happen anyway, except this time business didn’t get to find out ahead of time (surprise!).  All that communication that agile was supposed to be fostering gets destroyed.

The other thing that suffers when Product Managers become POs is team cohesion.  Agile is all about small teams, and people working together, and helping each other out, shoring up each others’ weaknesses.  This is why you always measure the velocity of the team and never that of its individual members.  When people work together on a team for long periods of time, they develop efficiencies of coordination: they know when someone is going to have trouble with this, or exactly who to go to when there’s an issue with that.  They can do things efficiently with a minimum of noise-to-signal in their interpersonal communications, because they know each other so well.  But when Product Managers are the POs, or even just assigned to teams, they have a tendency to want to shuffle the team members around instead of shuffling the work around.  If Product Manager A has more work to do than her team can handle, she goes to Product Manager B, who happens to have less work right now.  She wants to borrow some of his team members.  Perhaps Product Manager B doesn’t want to give them up; what happens if my projects heat up while you’ve got some of my people?  Even if you somehow manage to avoid the disruptive turf wars, and the transfer of resources is smooth, you’ve destroyed your team dynamic.  The transfer only appears smooth to the business side: on the tech side, the employee has a whole new team to learn about, new politics undreamt of, new social waters to navigate.  And, on top of everything else, you give your chronic underperformers a way to hide: in a situation where teams are for the most part permanent, you can count on your good employees to root out people who aren’t pulling their weight; when teams are constantly in flux, it’s more likely that they’ll just try to pass them around to another team, to (again) avoid the confrontation.

I’m not actually suggesting anything radical here; in Dean Leffingwell’s excellent Agile Software Requirements, for example, he makes it clear that Product Managers are not ideal POs, and I could cite many other sources.  But, in the end, this is a symptom, not a cause.  The cause is business not taking agile for what it is.  Too often it’s seen as some palliative: let’s just keep the developers happy, because we need them.  If this is what keeps them around, fine.  But we still want our deadlines met, no matter what.

This is contrary to the spirit of agile.  Agile, as we started out discussing, is about making hard choices.  It’s about saying what’s most important and how long things should take and how to deal with problems as they ariese.  When the business says, “this must be done on this date, and it must include all the features,” I’m somehow reminded of talking to small children.  Declaring something to be so doesn’t mean that it will happen.  You may even stamp your foot and hold your breath, if you like.  That’s not going to make the work go any faster.

In fact, unreasonable date pressure is probably the most common area where agile fails.  Of course, the hardcore agilists will complain that, if you have hard deadlines without functionality negotiations, you’re not doing agile at all.  And, perhaps, in a strict technical sense, you aren’t.  That doesn’t actually help address the problem though.  The problem is that tech people get very frustrated with these types of situations.  They’ll either buckle to the pressure, and make themselves miserable (back to polishing their résumés), or they’ll mentally give up, slow down, and make business miserable (and, I don’t know if business has noticed this or not, but it’s typically not the tech people who get fired if this scenario deteriorates).  Either way, business loses.  Their numbers suffer.  And this is not what business wants.

So the solution is for business to come at it with a fresh perspective.  For them to take the time to understand what agile means, and what it has to offer.  Agile is certainly not perfect (nothing is).  Agile certainly has lots of room for improvement (everything does).  But agile also takes a lot of different aspects into account, and actually produces a process that has the potential to make everyone happy in the long run.

There are companies out there who have seen that potential fulfilled.  Perhaps your company will join them.

[Having finished the second part of my two-part series, you may think you’re done.  However, I encourage you to stay tuned for part 3, where I talking about managing programmers.]

My Theory About Agile

[If you don’t follow software development, and in particular if you have no idea what “agile development” is, or how it contrasts with the “waterfall style,” this week’s blog post is not going to make much sense.  But that’s okay: I told you not to read this stupid blog anyway.]

Agile software development is really quite a cool thing, when done properly.  (If you disagree, please see my rant about hype.)  Unfortunately, most people who do Agile don’t really do it properly.  And, even more unfortunately, when Agile people try to help failing Agile shops, they often do it by pointing out where they deviate from a certain process (in whatever flavor of Agile is appropriate—XP, Scrum, whatever).  “You’re trying to do XP but you’re not pair programming,” they’ll say, or perhaps “you’re trying to do Scrum but you’re not standing up in your daily scrum meetings.” This is ironic (insanely ironic, even) because point #1 on the Agile Manifesto is: “Individuals and interactions over processes and tools.” So we’re supposed to downplay (although not ignore entirely) the processes, except when figuring out what we’re doing wrong, when suddenly it’s all about the process.  This seems wrong to me.  Most of the time when I’ve seen, or heard of, Agile going wrong, it isn’t the processes that are to blame.

It’s the attitudes.

I think that, in order to implement a good version of Agile—whether XP, Scrum, Crystal, or some melange—you need to start with an understanding of what Agile is supposed to give everyone.  So let me give you my theory on that.  Note that this theory is not backed up by scholarly research.  It is based solely on my own experience, and therefore is 100% anecdotal and useless from a strictly scientific perspective.  (As always, please refer to the masthead.)

Once upon a time, there was business, and there was tech.  These two camps didn’t always get along, primarily because of their radically different outlooks on the nature of reality and perception.  And one of the biggest conflicts they had (and, honestly, still do have) is when it comes to figuring out when something is going to get done.

See, tech people are horrible estimators.  We are, at heart, hopelessly optimistic.  I think this is mainly because, for the most part, we love what we do.  Which is unusual in the professional world, if you think about it.  Actors love what they do.  Musicians love what they do.  Professional athletes love what they do.  Really, if you think about it, programmers are some of the only people on earth who love what they get paid to do without ever getting rich or famous for it.  So, when you say, “we’re going to give you a really hard programming problem to solve,” we start salivating.  We’re looking forward to it.  When you say “now how long is that going to take?” ... we think about all the fun we’re going to have.  All the coolness, and all the interesting trickiness.  What we don’t think about is how long we’re going to spend going down blind alleys before we figure out the right way.  Or how long we’ll spend chasing down documentation on the new tools we’re going to have to learn.  Or how many bugs in IE are going to make us tear our hair out, or how long it’s going to take us to find our own bugs that QA thoughtfully pointed out to us after we were “done.” Or how much time we’re going to have to spend in meetings telling business why it’s taking so long.

So we underestimate.  It’s understandable, if not exactly forgivable.

Business has a different perspective.  They need to plan.  Us techies don’t plan.  We just do.  “It’ll be done when it’s done!” we say.  This makes perfect sense to us—after all, it couldn’t possibly be done before it was done, right?  But that makes businesspeople insane.  They need to market.  They need to develop release strategies and conduct market research and hold pricing meetings.  They need to write slick advertising materials about what the product will do, and there’s no way they’re going to wait to start doing that until after us techies are done making the damn thing.  So, to them, asking when it’ll be done is a perfectly rational question.

And thus the trouble begins.  I think that, probably, back in the dark ages of the software-induced shotgun wedding between business and tech, business would say “when?” and tech would answer “then” and business would say “fine” and then go away for however long that was.  Then they’d come back at the end and say “okay, where is it?”

This didn’t work.

After several iterations of not getting anything on the date they were “promised” it, business decided that they needed to check in with tech a little more frequently.  Maybe if they could at least know there were delays before it was too late to do anything about it, that might help.  Seems rational enough.  So let’s have a meeting every month just to see where we are.  And, you know, if meetings every month are good, meetings every week are better.  That’s just math.  Actually, meeting every day would be even better still.  Pretty soon, tech found that they might be spending as much as 20-25% of their entire workweek explaining why they weren’t getting any work done.

This didn’t work either.

Besides estimation problems, the other serious problem was specifications.  Business would say to tech “this is what we want the software to do.” And tech would say “okay.” And tech wouldn’t make it do that.  Exactly that; as I mentioned before, techies tend to be literaliststs.  Occupational hazard.  Whereas businesspeople tend to live more in the real world, where common sense is supposed to be applied to anything you say.  Silly businesspeople.  So there were communications problems.  And also, sometimes businesspeople don’t know what they want until they see it in action, wherein they promptly realize that they wanted something different all along.  Also, software takes time to develop, and the market doesn’t just sit around doing nothing while new software is being written.  Sometimes the requirements change in the middle of the project, and that’s not really anyone’s fault at all.

But somehow the solution to this problem became to make the specs bigger.  Longer, and more exact, and more like a contract.  There are legal documents—hell, there are bills in Congress—which make more sense and are easier to read than software specs produced by a serious waterfall methodology, like say ISO 9000.  And then, if something changed, everyone scrambled through their copy of the specs looking for the reason why it wasn’t their fault.

This was not particularly helpful.

See, there’s not only a fundamental communications gap between business and tech, there’s a fundamental difference in goals.  Tech wants everything to be beautiful.  Not perfect, because it can never be perfect, but at least pretty.  Most people don’t realize it, but programmers are craftsmen.  If you’ve ever engaged in a creative hobby—say, woodworking, or perhaps gardening—you understand pride of craftmanship.  When you make a chair, it better not be just good for sitting on.  It better be beautiful to look at, and detailed, and supremely comfortable.  When you plant a vegetable garden, it better not just produce food.  It better produce gorgeous, succulent vegetables that make the children’s mouths water and the neighbors green with envy.  If it’s not doing all that, you’re not done yet.  So it is with programmers.  It can’t just work.  Any moron can give you code that just works.  It needs to be code that other programmers will come along and read, and whistle to themselves, and be happy that such an amazing craftsman was able to provide them this as a starting point, because she just made their lives a thousand times easier.

Whereas business is all about numbers.  Mainly that bottom line number, the number that shows you how much profit you made at the end of the day.  But there are all sorts of numbers: customer satisfaction numbers, and product defect numbers, and market penetration numbers.  Some you try to maximize, and some you try to minimize, but it’s always about moving the needle.  Maybe you pull off some business maneuver that makes a good story over cocktails, but that’s just a bonus.  At the end of the day, you can tell whether you did well or poorly just by looking at the numbers, and numbers don’t lie.  And everyone else can tell how well you’re doing too.  It’s all very satisfying, in a visceral way.  When you get that report back, and the numbers are bigger (or smaller), you get this excitement in the pit of your belly, and you just want to pump your fist in the air and go “YES!”

And the thing is, it seems like these are completely incompatible viewpoints, but that’s not the real problem.  The problem is, they’re both wrong.  Oh, they’re both right, too (balance and paradox), but they’re both wrong.  And, when you put them together, they balance out nicely.  Tech without restraint becomes perfectionist, always tinkering with everything and never finishing anything.  Business without restraint becomes lost in the weeds of short-term gains and ignores the long-term goals that are being crushed.  Techies often forget that, if they don’t make deadlines, the bottom line suffers, and that’s what puts food in their mouths.  Businesspeople often forget that cutting corners today means defects—and therefore lost revenue—tomorrow.  But together, there’s a yin and a yang that usually produces pretty good results overall: not too slow, not too sloppy.

And this is what Agile attempts to do.  Agile says to tech, “You’re not going to get everything you want—sorry.  We’re going to give you an outlet for your concerns about doing things right, but you’re going to have to justify them.  And we’re going to put you in the driver’s seat on how long things are going to take, but you’re going to have to break it down into tiny little pieces and esimate every one.  And, yes, you’re going to have to give your status every day, but we promise it won’t take but 10 minutes.  If you want to spend longer on doing something the “right” way insead of doing it the expedient way, you’re going to give business numbers to show the difference.  This is how much it costs now, and this is how much it will cost later, and then business can make a rational decision on whether that trade-off makes sense.  And sometimes they’re going to agree with you, and you’ll be happy.  And sometimes they’re not, and you won’t be.  But it’s better than nothing.”

And then Agile says to business, “You’re not going to get everything you want—sorry.  We know you want 100% visibility into what tech is doing, but you can’t have that.  100% visibility equals 0% productivity, because it would be you and tech sitting in a room, with you going ‘What are you doing now?’ and them replying ‘Answering your question’ over and over again.  But we’re going to institute a structure so that nobody gets stuck for more than a day.  And we’re going to give you numbers.  How many units something will take, and how many of those units this team can deliver in a given time period.  That time period will be short—a month, or even two weeks—and you can specify exactly what will get done in each period (no more ‘tech spent 3 weeks doing this thing because they thought it would be useful’), but you can’t change your mind in the middle of the period.  If there are changes, just wait a few days until the new period comes along.  And the team is a team: they’re a cohesive unit that works together, and you can’t just replace team members or juggle them around like they’re factory components.  So you can’t bother them while they’re working, and you can’t try to measure them individually (which ends up creating destructive competition anyway), but you’ll get lots of numbers that let you know what’s going on and when things are going to get done.  And that’ll have to do.”

This is where business and tech need to start from if they’re going to try to make Agile work for them.  They both need to start from a place where they each realize they’re not going to get everything they want.  In my experience, most of the time when Agile goes wrong, it’s because tech is on board with this compromise, but business didn’t get the memo.  Next week we’ll look in more detail about just what can go wrong when that happens.