Eating your own dog food versus usability? Maybe

greenspun.com : LUSENET : Joel on Software : One Thread

I'm the author of the Giga Information Group piece Joel described as a critique of his "eating your your own dog food" article. Since my piece isn't available for most readers of this column to read (yes, we do indeed charge for our ideas. It's kind of like, oh, for example... Joel charging for his forthcoming book?) I thought it was a little unfair to pull a few words out of context and criticize them. (By contrast, Giga's subscribers could read Joel's article in full, to which I provided a hyperlink, and make their own judgment). So I'd like to take the opportunity to set out my side of the debate on a few points.

- I didn't say not to eat your own dog food. As the title of the piece itself states, my position is precisely "Eat Your Own Dog Food, but Not as a Substitute for Usability". In other words, Joel agrees with me, I agree with him, we all agree with each other, group hug, etc. However, I think that if Joel goes back and re-reads his original piece, it doesn't come across that he advocates doing both.

- To say that this title is a logical fallacy is itself a logical fallacy. Joel's original article advocated EYODF, but didn't mention usability testing. I'm very happy to accept that Joel supports usability, especially based on reports from other people who know him, but that isn't at all clear from his article. Think of my title as a *clarification* of article's position, and perhaps it will be clearer why it isn't a logical fallacy.

- Joel quotes part of my concluding paragraph but without the surrounding context. Here it is more fully (who says you never get anything for free?):

"Unfortunately, “eating your own dog food” is one of the most pernicious barriers to doing true usability, not least because it is genuinely useful in many other ways. Eating your own dog food should be the first step on the way to usability, not a substitute for it."

In the rest of the piece I elaborate on the reasons why EYODF is so often a barrier to true usability: that some companies mistake it for usability so don't do the real thing, that it can induce complacency that all problems have been found, and so on. These issues were not pulled out of thin air: they come from observing the many, many vendors that I interview almost daily in my job.

- I'll stand by my characterization of Microsoft as a great example of this trend. Yes, it does a lot of usability testing; unfortunately the results of that testing don't always go into the products. (Clippy, anyone?). It also does a lot of "innovation" in the UI without usability testing. And, most significantly for the purposes of this article, it substitutes EYODF for usability in many cases. I know this because Microsoft product managers have said to me, on occassion, that a UI feature I had criticized was not a problem because it had been used successfully internally. Microsoft is, I maintain, a perfect example of a company that sometimes (by no means always) substitutes EYODF for true usability - and it shows.

Joel, good luck with the book. We certainly need more programmers plugged into usability. But lets not encourage, even by omission, the belief that programmers can be their own judge of good usability.

-- Anonymous, May 26, 2001

Answers

Eating your own dog food versus usability (Redux)

I just read some chapters of Joel's book online and felt moved to add a postscript to my own note. Joel advocates "hallway usability testing" on 5 or 6 people as sufficient testing to find 95% of the usability bugs, and cites various authorities in support, e.g. Nielsen. I have to disagree, and I think Nielsen would too. The problem with this approach is that you have no good reason to believe that the people you selected resemble your target users, whether in their skills, their needs, their experience, or how much they like you and are reluctant to criticize your work. The "5 or 6" rule from Nielsen works only if those 5 or 6 are good representatives of real users.

Sure, its much better than not usability testing at all. So is eating your own dog food. But its not nearly as good as testing with real users, and it certainly won't get you to the 95% result that Nielsen promises.

Incidentally, I also noticed in passing that in these chapters Joel cites Microsoft for lots of usability problems, even though he earlier described my choice of Microsoft as an example for such things as "unlikely". The Microsoft problems that Joel describes are directly caused by preferring internal opinions (whether individual programmer choices, design team debates, hallway usability testing, or eating your own dog food) over simple observation of actual users.

-- Anonymous, May 26, 2001


Moderation questions? read the FAQ