The latest .Net Rocks podcast was from a panel at TechEd 2008 (that I sadly missed while there). Richard Campbell was the moderator and the topic was Software Quality. It was a good, if somewhat one-sided discussion and I recommend giving it a listen if you are so inclined.

I particularly liked Billy Hollis' perspective on quality because it was a perspective that was grounded in both reality and sound business principles. He kept bringing up quality in terms of trade-offs and the others kept trying to waltz around those comments with a more absolutist, almost dogmatic, vision of software quality. I think it was a real shame that Billy Hollis was so out numbered because his points are important if we are to gain the stature we desire as a profession.

The final comments completely got on my nerves. Which is fortunate because it helped crystallize my discomfort with the others.

Billy Hollis started off with this simple statement.

My key insight that I try to communicate is it’s all a matter of trade-offs. And getting to what the user needs is first and quality is defined within that context.

To which Neil Roodyn responded.

I’ll disagree. It’s about quality. You’ve got to get your quality as high as you possibly can. Always. The higher the quality of the software, the higher the quality the software will remain through the development life cycle. So right from iteration zero keep that quality as high as you possibly can.

And Jeffrey Palermo added.

The higher the quality the better. You can’t have too high of quality, you would just say you have too much behavior for the situation.

The thing is that Hollis is right and the other two are not only wrong, but dangerously wrong. Other members of the panel (David Platt and Neelesh Kamkolkar) didn’t support Hollis either, though they weren’t as overt as Roodyn and Palermo.

What Are You Trading?

The thing is that software quality, besides being pretty much impossible to measure, isn’t a first-order good in the first place—i.e. it isn’t valuable in and of itself and delivers what value it adds indirectly. You can make the reality of this simple business fact plain by pointing out how the deciding factor between two competing products is often not quality. Betamax anyone? Rolls-Royce? Sun Microsystems?

People choose software for a lot of different reasons and those reasons are very rarely explicitly the “quality” of the software. Price, availability, familiarity, and usability all play roles in the purchase (or use/approval if internal) of software and these factors need to be kept in mind. It could be that it is better for your company to have software that is harder to maintain but completed two months earlier. I do this all the time because I have lots of projects that are single-use and with a restricted domain.

Software development involves time, money, commitment, and concentration. Spending those valuable resources should be done carefully and be based on actual value to the company. Quality has a cost. Quality doesn’t always have a pay-off. Business people have a term for a cost that doesn’t return a pay-off. That term is usually “good bye”.

Dangerous?

So Palermo and Roodyn are wrong, but why are they dangerous? This comes back to my observations during the dot com bust. I observed the glee in putting IT back in its place. You see, during that whole tech bubble IT got arrogant. Technical folks were wagging the business dog, dictating strategy, policies, spending, and sometimes even remapping the boardroom to their liking. Which would have been fine if the business rules really had changed as some thought. Unfortunately, they hadn’t. This arrogance set the entire profession back because business leaders learned not only of our arrogance, but of our ignorance as well.

We don’t bother learning even basic business principles before we make important, even critical, business decisions. This discussion of quality is an example of just such thinking. Roodyn and Palermo are talking about dictating business resources without any concern for accommodating business realities. Worse, they don’t seem interested in even learning about or exploring the business realities they, of necessity, work under.

As much as we might like the universe to bend to our personal theories and values, the fact of the matter remains that there are a lot of reasons to compromise aesthetical purity when the rubber meets the road. Those reasons don’t simply go away just because they make us uncomfortable. Indeed, how we react to the inevitable trade-offs required will largely determine how much business leaders decide to trust us in the future.

Here’s a helpful exercise. Imagine you are a business leader evaluating which company to hire to solve some software need. As part of your vetting process, you ask two finalists their stance on software quality. The first one answers with “Getting you the things you need is our first priority and quality will be defined in that context.” The second one answers with “Quality is number one. Always. Quality cannot be too high though it could be that you want too much behavior for the situation.” Which would you hire? If you answered the second, it is because of ideological blindness. Actual business leaders will not only tend to go with the first, but will have confidence and trust in doing so.

Are You Agile?

A lot of the value in IT comes from giving businesses flexibility. Processes that used to take indoctrinating a new generation of workers can now turn on a software release. While this is both a blessing and a curse for businesses, it remains a key to our value in IT. Businesses have learned to be wary of our use of resources, however, and with some cause. It is unfortunate that they cannot trust IT, but until we let the needs of the business drive our development I’d say that their trust hasn’t yet been earned. If we really want to be masters of our own fate, we must learn to live in the messy world of business. We can only do so by proving that we will use the flexibility inherent in our profession wisely.

You might say that I’ve had issues with Dependency Injection in the past. Well, with all the things I’ve learned not to do, I thought I’d go into a case where I’m considering dependency injection because it looks like it might be a good fit.

Project Background

Like many (most?) developers, I’ve been involved for many years in a particular vertical market. Like most, my involvement in that vertical market includes acquiring a substantial amount of very specific domain knowledge. I’m sometimes reluctant to admit it, but I spent a good decade and more of my career writing software designed for Multi-level Marketing companies. Specifically, I wrote programs that calculate the commission payments to their distributors. If you are familiar with that domain, my condolences. If you are considering entering that domain, talk to me (it’s not all bad).

Now, many accounting software packages deal with paying commissions, but compared to even the simplest MLM, their capabilities are pretty primitive. I won’t go into the full complexity of walking a tree with money on the line, but here’s what’s important for this post:

• The data you have to process is well contained, finite, and known in advance.
• Data concurrency tends not to be an issue (you read distributor information at the beginning and you write payout and promotion information as you go or at the end).
• Every company has their own, unique twist for qualifications, promotions, and payouts—indeed, commission plan differentiation is a key business differentiation.
• The only thing absolutely common to every commission plan is that you walk a tree of distributor nodes, processing orders at those nodes, and calculating payouts and/or promotions on them.

Most programmers working in that domain spend time designing their dream "bonus engine". One that can encapsulate the common elements so that their programming only has to deal with the payout and promotion logic. This turns out to be harder than you’d think to do right (if only due to the temptation to put more in the engine than really belongs there).

My Bonus Engine

Since I’m still getting occasional contract work programming commission payouts (and because I’ve seen a bonus engine or two and know what works and what doesn’t), I went and created an OSS bonus engine. Those of you dying to critique my code-fu to bring me up for well-deserved mockery, here’s your chance. So far I’m the only dev on the project, but that’s only because I don’t know anybody else masochistic enough to join me (hey, if you’re that nuts, let me know).

Right now, this engine is still pretty raw, but it works. Mostly. There are INode and IVolume interfaces for distributors and orders respectively. There’s a NodeBase object that implements INode with nifty events already implemented for easy inheritance and feedback and there’s a BonusManager object used to kick off a bonus run.

The tricky bit for this post is the data access. You see, when programming commission payouts, I seriously don’t care where the data comes from or where it is going. My clients all have their own tricky little data schemas and I’d just as soon not deal with the ins and outs of their systems. Abstracting that part out is beneficial all round.

In my initial design, I figured that something like that is a classic call for an Asp.Net-like provider model. This works well because the client can implement a provider (or have me create one for him) that we can drop in and we’re golden. That means that my eventual Distributor objects can be ignorant of all data implementation details and make a call to MLMBonus.DataProvider.BonusData.Promote() to notify of a promotion (for example).

Making the method references static seems like a good call because the distributor object has no other contextual information available. The engine is walking a tree and calling interface methods so the distributor and volume objects don’t really know much more about the outside world than their relationships to each other.

Because this is my first ever implementation of a "Provider Framework", I went digging for some sense of what kinds of things to look out for. While I initially went with an interface structure, I eventually decided to use Joel Ross' well written guideline for creating your own .Net providers.

There’s Always One

Unfortunately, I got some push-back when talking with my favorite client. You see, he’s chafing at the static nature of my provider model. He wants to make a grab for the holy grail of bonus commissions—executing mini-commission runs on-demand (most likely for partial distributor trees)—and the static provider just isn’t cutting it for him. mini-commission runs allow you to provide some interesting features to your distributors and that’s never a bad thing.

To support his functionality, he would need to be able to alter the data provider "on the fly" and/or be able to initiate multiple runs in the same application space without allowing them to step on each other. There’s just no way that static providers are going to work for that functionality. Naturally, I’ve been thinking about how best to work out meeting his need (mainly because I can see this feature will be something others will like but also because he’s my favorite client). Fortunately, my engine isn’t widely disseminated so it’s a good time for wholesale architectural alterations if we want to make them.

I’ve considered a number of ways to tackle the problem, playing with things like anonymous methods and context managers. They all turned ugly fast so it’s time that I face the fact that this functionality is pretty much tailor made for Dependency Injection.

Where To Inject

So where’s the best place to inject this little dependency? There are four options as far as I can tell.

• constructor injection
• property (or setter method) injection
• method parameter injection (since the number of methods affected will be very small).
• framework injection (Spring.Net or Castle Windsor for example)

Constructor and property injection are both object level references carried by the dependant object and have little to distinguish them in effect. Conceptually, I’ve heard that constructor injection is best for dependencies that are crucial to an object—something the object simply cannot do without. I think I agree with that guideline but I also like the convenience of being able to do grunt-work things in constructors, so if I were in charge of the objects, I’d include both a base constructor and one that includes the dependency as well as a setter method of some kind.

Method parameter injection is tempting, though. With method parameter injection, a client can set the dependency on BonusManager and rely on BonusManager to send the dependency to the Pay and Promote methods on INode. That way the reference to the data provider is short lived inside INode and IVolume objects. Thinking it over, though, I’m not sure why I’m so worried about carrying a bunch of references when garbage collection isn’t going to happen mid-run anyway. Yeah, there’s some overhead involved in tracking the reference on each INode object, but it’s negligible (are pointers in a 64 bit environment 64 bits long? They’d pretty much have to be right? I’ve never bothered to find that out—it’s never been significant before).

Framework injection simply isn’t going to be helpful in this case, I don’t think. For one, I’m not familiar enough with any DI Frameworks to make it worthwhile. For another, I don’t want to introduce a framework dependency on the project. And finally, I don’t want to commit future clients to a DI Framework that may be an issue in their environment.

Injecting

So my next task for the engine will be to muck around with the EngineInterfaces library. I’m thinking I’ll leave the provider object as-is (the BonusDataProvider abstract class) and simply add a property to INode to inject that sucker. In addition, I’ll update NodeBase to have an appropriate property and a new constructor that allows you to set the provider property on object creation.

Since I have the provider stuff already in place and no changes are needed to make this work, I’ll likely leave it alone. That way I won’t break anything currently working.

The only thing left is to decide if I want the IVolume dependency injected via a field or a method parameter on Pay. I’m pretty sure I don’t need the fine-grained control of allowing different orders for a single node to pay down a different path, but do I really want to commit to that? My arguments above against method parameter injection seem pretty solid for the IVolume object as well, but I haven’t completely convinced myself that they apply as strongly.

Aaron Lerch talks a bit about what it takes to be a technical manager. He enumerates five characteristics that he feels are essential for a technical manager (personally, I think that they are better classified as skills because they’re learnable and improvable). Here’s his five:

1. Communication
2. Technical Savvy
3. Organizational Skills
4. Priorities
5. Humility

It’s a good list for the good times, I think. As long as things run more or less smoothly, a manager with those traits will be in good shape. As you can probably tell by my qualifiers, I think that he’s missing something.

Tough Choices

Unfortunately, managers (technical or not) often find themselves in a position where a decision has to be made and where the choices are all distasteful. What do you do when it turns out that someone doesn’t live up to your expectations or, more importantly, their job description? Or they simply don’t get along with the other developers on the team? Or, heaven forbid, the company has decided that less overhead in the development department is needed?

Firing someone is, by far, the hardest thing I’ve ever done as a manager. Firing someone is both an admission of failure and one of the harshest things a person can legally do to another. So a manager who fires somebody is pretty much a big, mean failure.

It’s a real shame that sometimes, people simply need firing. If you’ve ever been on a team with someone who couldn’t (or wouldn’t) pull their own weight, you’ll understand what I mean. Not firing someone who needs it can have a larger impact on team productivity than just about anything else I’ve encountered. I know because I’ve been a team member when someone needed to be let go and the manager couldn’t bring himself to pull the trigger. The hit to both productivity and morale is huge and grows geometrically over time.

Firing someone is the extreme case of an important skill. Management is all about the hard decisions. Yes, communication is important. And making irrevocable technical decisions without team input is suicide. But consensus is the one sure path to mediocrity and, frankly, mediocrity doesn’t do that well in the marketplace. It’d be nice if everything would shake out into rational lines with obvious choices, but real life (and real development) doesn’t work that way.

Decisive Humility

To make things harder, Aaron’s point about humility is still a good one. Technical managers should do their very best to surround themselves with people who are smarter than they are. Being surrounded by smart folks can be hard on you if you wear your ego on your sleeve, though. Also, smart people are incredibly difficult to control.

Being able to make compromises and solicit (and listen to) advice from your team is important for both you and them. But eventually, you will find yourself in a situation where the alternatives are as known as they are going to be and it is time to make that tough call. Making that call (and sticking to it) is your job. Managers who waffle or leave it up to a vote (whether formal or not) or who weasel out of responsibility later compromise the integrity of their team.

Balancing humility with the power and ability to decide is tough. It is, nonetheless, necessary.

Top Skills

So how would I re-order Aaron’s top five?

1. Communication
2. Technical Savvy
3. Responsible Decision Making
4. Humility
5. Priorities
6. Organizational Skills

While this is a general ranking, situations will arise where any one can become paramount—meaning that none are optional. All of them should be as strong as you can make them and the better you do, the better manager you will be.

Man, I'm going to catch it for that title. Okay, let me get this out up front: this post is a hypothesis that I consider probable. That's why there's a question mark in the inflammatory title. Okay, asbestos underwear in place, let's continue.

The Hypothesis

Here's the logic flow for my hypothesis.

1. Unit testing (and its bosom buddy automated testing) provides substantial benefits when used regularly. These benefits are well-understood.
2. Ensuring or regularizing unit testing is the primary value proposition of Test-driven Development (TDD).
3. A majority of vocal unit testing advocates are also TDD proponents.
4. Developers searching for unit testing information are more likely to find references that describe unit testing as subservient to or at least included in TDD than they are to find references that are simply about unit testing.
5. Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn't have had a problem with simpler unit testing methods.

Simple enough, really. If 1, 2, 3, and 4 then 5.

I'm not going to bother trying to verify statements 1 and 2. You might be able to quibble with 2, but it'd take some real mental gymnastics to do so.

Testing Propositions 3 & 4

Propositions 3 and 4 are key to this hypothesis being correct so let's explore them. Here's a simple test: what are the top 10 links on Google for unit testing and are the addresses they link to substantially advocates for test-first development. This test assumes that the top results from a Google search will be representative of what unit testing advocates are saying. Since Google earns money by the metric ton to make sure this is the case, I'm going to let the results stand for both 3 and 4.

To help with the analysis, I'll include two rankings for each link. Both rankings are my subjective opinion; included so that you can check my figures for yourself if you wish by seeing how much you agree with my evaluation (or if you agree at all, I suppose). Both are a scale from 0 to 5.

The first scale, Helpful, will indicate whether a developer looking to find initial unit testing information is likely to read the whole thing. A five is something I think most developers will stop to read and digest. A 2 or 3 is either too general or too specific and thus might or might not be interesting to a developer looking for initial unit testing information. Lower scores indicate links that make me question Google's ranking system.

The second scale, Test-first Focus, will indicate how focused that link is on writing unit tests before coding. A five is something that puts unit testing squarely in the realm of test-first development and includes substantial explanations on how to do so or a base assumption that testing first will be practiced by those reading. A 3 means that it contains information on testing first, but in a more neutral tone—i.e. unit tests will be written when the developer determines it is important with testing first given as a (mere) option. Lower scores indicate that testing first is mentioned in passing if at all. And yes, I am going to lump TDD into XP and other test-first methodologies because they will tend to lead to step 5 as much as formal TDD will.

Okay, here's the top 10 Google hits for unit testing as of this moment:

Well, fancy that. Over half the entries are a 4 or 5 for their test-first focus. Of those that aren't directly test-focused, one is wikipedia and the others are unit testing frameworks (I think—that last one is weird).

I'm going to call propositions 3 and 4 proven.

So Then

Here's the question: given that 3 and 4 turn out to be true, do you think that "Faced with the substantial alterations in habit and development methodology inherent in TDD, many developers and development managers eschew unit testing altogether who wouldn't have had a problem with simpler unit testing methods"? Personally, I think this is likely the case.

Since the barrier to implementation is higher for TDD than it is for POUT, you have to acknowledge that more developers will be willing to spend the resources on POUT than will do so for TDD. The question is how many are we talking here? How can we find out? I find TDD a much more substantial resource hit than POUT. Thus, at least some people would be willing to spend resources on POUT that wouldn't spend them on TDD. You can double-check this by trying to imagine someone being willing to spend resources on TDD who would hesitate to spend them on POUT. Do you think there are any? I don't.

If this is a trend in the general population, then what happens when someone who would opt for POUT if they knew about it is only able to find information for TDD? It'd be only natural if they skipped unit testing altogether.

TDD folks gripe about projects with no unit testing. Non-TDD folks resist and resent being badgered about implementing a methodology they aren't convinced will justify its cost. I wonder if we aren't locked in an unnecessary struggle between poles with each side providing ammunition to the other.

I wonder what would happen if the TDD folk started with encouraging people to unit test regularly. After all, TDD folks would have to admit that having some unit tests is better than no unit tests. How about starting your testing discussions with POUT? Or how about defaulting to POUT when you run into resistance for TDD and push for more later if you can? How badly do you want regular unit testing? Badly enough to let go of presumptive TDD discourse? I've noticed that TDD advocates never discuss unit testing outside of a TDD framework. Since many of the most popular developers fit this category, I wonder if we aren't unwittingly encouraging a TDD or nothing effect in the wild.

I wonder what would happen if every TDD advocate went on the record that POUT is way better than no unit testing (with examples and/or encouragement and without going on to propound the vast superiority of TDD). Will you TDD advocates reading this accept the challenge from me to do so?

And here's an interesting question you can address: how much of the advantages of TDD do you think can be achieved with just good design and timely unit testing? 90%? 50%? 20%?

One of the things that irks me when discussing certain technical topics is a tendency towards boosterism that can hinder, or even halt, deeper evaluation and discussion. Some topics, tools, or practices arrive with a divine imprimatur of authority that is hard to resist. Since I believe in being personally responsible for the software development I undertake, I sometimes find myself exploring murky territory that seems like it should be better mapped. The latest of these pre-hallowed principles is Test-driven Development.

What comes across as a casual assumption of superiority gets my hackles up. I'll show what I mean using a recent post by Derik Whittaker titled Sound of One Man Testing. I want to be careful up front, though, because my purpose isn't to pick on Derik. Indeed, I really liked a previous post of his describing his love of being a software developer. I suspect that I'd like him if I knew him personally. I'm just using his post so that I have concrete illustrative examples for what I'm reacting to. His post is in good (and numerous) company.

TDD or A Poke in the Eye with a Sharp Stick

I mentioned this before, but it keeps happening. The comparison to TDD seems to always be a complete lack of testing. This is always a flawed comparison. It doesn't matter if you're addressing heathens or saints, it simply carries no useful information. It promotes sloppy thinking and lazy arguments because you set such a low hurdle to beat. It's worse than unconvincing because it irritates people who disagree with you and thus erects barriers to communication.

It is a textbook case of the Straw Man logical fallacy. Defeating a straw man isn't really much of an achievement and pretending that it is doesn't make it so. Skipping straight to 50 when counting out pushups is only impressive to someone who didn't notice. It doesn't actually do you any good and will certainly never result in those ripped abs your SO goes on about.

Going Mental

Derik says "Following TDD is more about your mind set and attitude [than] it is about the process in my opinion." David Tchepak recently made a similar point (admittedly tangential to his post) in his generous response to my earlier comments, "I feel TDD has the potential to change the way you think about coding."

I get their sentiment and I'm not trying to dismiss these statements out of hand. Attitude, mind set, and perspective matter, and things that keep you focused or have other purely mental benefits have a place in software development.

It's just that these benefits are highly personal and hard to make relevant. If you include them, it is helpful if you can make sure that they are an important part of your point and that you develop them in a way that stands a good chance of connecting with others. Failing to connect them to your larger point makes them stand out, weakening the whole.

Clique Here

Many posts about TDD are aimed at those who are already convinced. Derik's post is one such—he is clearly addressing those who already believe in TDD who want arguments to help them bring TDD into non-TDD environments.

Which would be fine in a discussion group or forum, but is more problematic in a blog post. Blogs are, by nature, public (I've seen private blogs but I really don't see their point). Blog posts are likely to be read by people who don't share your basic assumptions. It is wise to bear this in mind when discussing potentially controversial topics.

Addressing an in-crowd comes across very poorly for those who don't consider themselves part of your community. It's like those jerks in High School who talk about others in public as if they weren't there—which is essentially what you are doing.

Take this section of Derik's post, for example:

Benefits the project will gain

• Better code coverage
  Even if you only can cover 5% of the code in your tests, that is 5% less code to worry about
• Increased code quality
  Do I really need to explain this?
• Increased code reliability
  Do I really need to explain this?
• As foundation to grow on
  Because you have [taken the initiative] to create the test plumbing (i.e. configuration, db setup, logging setup, etc) you have reduced the barrier of entry for other developers. I have found that this barrier of entry is a common reason people don't want to start testing.  I know it is lame, but I have heard it.

Someone not already convinced of the superiority of TDD can take what looks like a casual disregard for their perspective poorly. My inner spite-demon pops up like a jack-in-the-box and says things like "Yes Derik, you do really need to explain this." or "Do you really mean to say that covered code == code you don't have to worry about?" or "Taking the initiative to provide infrastructure only you use sounds a lot like wasting time and resources to me."

While I got over myself (and reminded myself again that I liked his previous post), the reminder that I'm not in Derik's clique is off-putting. If you don't want to re-hash arguments that you feel have been established elsewhere (a perfectly reasonable impulse), give a link (or at least a reference). Bear in mind that there are reasonable people out there who don't share your convictions and if they show up you would probably like it if they would hear you out.

Remembering that Reasonable People Disagree

There is an unfortunate human tendency to believe that all smart people agree with you. The corollary is that anyone who disagrees with you is stupid. The thing is that even if you are right, communicating from that perspective doesn't work out very well. Reminding yourself from time to time that people disagree with you and some of them might be worth reaching out to is a good habit to cultivate. I know that I'd personally take it as a kindness.

Because Unit Testing is the plain-Jane progenitor of Test Driven Development, it's kind of unfair that it doesn't have an acronym of its own. After all, it's hard to get programmer types to pay attention if they don't have some obscure jargon to bandy about. UT is too awkward, besides being a state abbreviation in the U.S., so for this post (and, if it catches on, future posts as well) I'll borrow from the telco folks and call unit testing Plain Old Unit Testing.

The Best of all Possible Worlds

Part of my problem with TDD has been that it claims to provide complete testing. You see, even if this is true, it runs up against my gut feeling that worse is better when it comes to unit testing. Or, to throw in yet another development aphorism, it is my suspicion that unit testing lies squarely in the realm of the 80/20 rule—80% of the value of unit testing comes from 20% of your unit tests.

While it is functionally impossible to know in advance exactly which 20% of your unit tests will be important later on down the line, that doesn't mean that we're throwing darts at a board. Experienced developers can make pretty good guesses about what the problem areas are likely to be. And they're pretty good about developing unit tests that cover those problem areas.

Which is my way of saying (or restating, really) that number of tests is no more important to me than number of lines of code as a measurement of value in programming.

Crossing the Beams

I think some of the confusion with TDD discussions is that TDD is an intensified version of POUT. Both POUT and TDD use unit tests. Both do so as part of a process. Both are concerned with future maintenance and functionality. The primary substantive difference is that TDD writes the unit tests before developing anything else.

Unfortunately, the popularity of TDD followed the widespread use of POUT very closely. This hurts because we never had a chance to get comfortable with POUT on its own before TDD appeared on the scene. Indeed, many people went from no POUT (or just beginning to learn) straight into TDD.

What that means in practical terms is that we have a tough time separating the value of POUT from the value of TDD. Few people have done both thoroughly enough to judge them based on this relatively singular point of differentiation. The inability to distinguish or track the value of each separately lies at the heart of much of the cross-talk surrounding TDD and TDD evangelism. Indeed, many of the testimonials I've seen (including many comments on a prior post) seem to assume that without TDD, no testing is done at all.

Those of us who are happy with POUT (and looking at the extra effort needed for TDD with some distaste) are left wondering what TDD offers us that we don't already have. We don't want some hogwash about forcing us to do our job. We see the value of testing and have learned to take time out for POUT (Oh. There's a slogan that you can paste up on your very own office wall "Time out for POUT!" Remember you read it here first.). Frankly, it seems to me that we're getting the goodies the TDD folk go on about just fine and without having to retrain how we develop.

A Call for Differentiation

So here's my request for those of you who are using and enjoying TDD who want to invite us all to partake of its superior benefits: please couch your arguments in the assumption that I am already POUTing, that my POUTing already delivers substantial benefit, and that adopting TDD is a non-trivial training and practice burden. Thank you.

The discussion spawned from my TDD post has been interesting to me. I've enjoyed the comments by Phil on his post and by him and others on my own. I'm particularly flattered that one of the authors of the original study dropped by. I disagree with some of his points, but this isn't where I want to address them. Instead, I want to examine the discussion itself because this is definitely something that we as a software development community could be better at.

The Report

First, I want to emphasize something that I could have made clearer in my original post. The original study was extremely well designed and I have no issues with how it was conducted or how it was structured. I described the report in my prior post but neglected to give my opinion of the setup. Let me correct that now: this is an exceptionally well-designed, well-executed, and well-written study. And I'll point out that my criticisms were not with the data collection or design. If you go back over what I said, you'll see that my concerns are with presentation and interpretation.

On Commenting

As I said, I enjoyed most of the comments posted both on my post and on Phil's. That said, here's some advice for those expressing opinions online.

First off, if you are going to criticize something, it'd probably be best if you actually read it. Or at the very least, read what is said about it very carefully. I can tell that many of those supporting or criticizing the report hadn't bothered to get to know the study at all because of little misrepresentations they made (like assuming that two teams from a group of 24 must be split evenly—they weren't). Here's the thing, if you make little errors of fact, it becomes hard to take the rest of your points seriously.

Second, if you are going to criticize the way a study is put together, it'd help if you know what you are talking about. Take sample size, for example. While 24 seems like a small sample size to end up with, there are very detailed and specific methods to adjust for sample size in statistical mathematics. This fact is at the heart of the authors' use of the term "statistically significant". In essence (and strongly backed by decades of research) if a bias is strong enough, sample sizes don't have to be very big to be representative enough to be useful. A little cross-discipline humility goes a long way here.

Now, I have no problem with someone who might say "that seems like an awful small sample size" because that allows you to express your doubt but also leaves room for further enlightenment and avoids making an unsupportable assertion of failure. The thing is that this study wasn't put together by amateurs. This is an academic paper presented to the IEEE. Like all good academic papers, it includes an entire section titled "Threats to Validity". Reading it, you'll notice that sample size is not listed in this section. Asking (yourself or in general) why a sample size of 24 isn't considered a threat to validity is a good idea and potentially interesting in a geeky kind of way.

Third, the force of your conviction doesn't ameliorate the burden of making your point clearly and rationally if you want to convince someone else. Too often, advocates will substitute emotion for reason when discussing their Great New Thing™. That's a perfectly human thing to do. It is also perfectly human to reject overwrought arguments out of hand. Seriously, you do your cause no good if you cannot be bothered to make your case in clear and rational terms.

Finally, if you don't know people you can simultaneously respect and disagree with, the problem isn't that people who disagree with you are all stupid jerks. The limits of a system isn't in the brilliance of those arguing for it, it's the brilliance of those arguing against it. Untested theories and unquestioned understanding are as useful as untested code. Personally, for any given topic, I want the best arguments that I can find. It's not enough to examine both sides of some new thing, I want the very best arguments that both sides have to make.

Which is why I appreciate the efforts of people like the authors of this study. And why I appreciate guys like Phil who take the time to discuss their theories and to explain and expound them when challenged. None of this stuff is a no-brainer. If it were, people wouldn't pay us to do it.

This should be pretty short, but I could not find this information anywhere so here's something that will save you hours of frustration if you ever run into the same situation.

WCF Myopia

Windows Communication Foundation is nice and all, but it suffers from a really large myopia: it tends to assume that you control both the service and the client. This is a stupid assumption, but try finding information for a situation where a service throws a FaultException and you want to get at the details of the exception and you'll see what I mean.

The Setup

Dynamics Great Plains Web Services allow you to manipulate common business objects. I've recently been trying to use WCF in communicating with those services and it has been an adventure. A recent post detailed my solution for customized warehousing. What I left out last time is moving to WCF's error handling seriously messed up my error condition reporting.

GP Web Services sends a fault for both warnings and errors with a relatively informative status message and a detail section that is just a GUID that identifies the fault in the server log. The theory is that if you want more details, you can query the service again to pull further information in.

The Problem

All the solutions I could find for getting fault details out of WCF involve creating contracts on the service that the WCF client can read to create strongly typed objects in the generic version of FaultException. Well, I can't control the web service to do this. I could probably have hacked or wrapped the wsdl for the service but who wants that much low-level headache? And before you go there, the faults that the GP Web Service sends didn't trigger on what was billed as the default exception with generic details:

catch (FaultException<ExceptionDetail> ex)

{

}

Unfortunately, the regular FaultException doesn't expose a Detail property or methods that you can use to get at any details sent. This is a stupid oversight. I mean, yeah, anything can be in there, but whatever is there has to be XML serializable pretty much by definition. They couldn't throw us a bone and expose it as a string or XML fragment?

The Solution

So what do you do with WCF if you cannot change the service but need to get at a fault detail without a specific .Net-friendly contract?

The solution I came up with is actually extremely easy once I realized I had to go through a couple of classes to get there (don't ask how long this took to ferret out).

• Snag the message itself using MessageFault.
• Get an XmlDictionaryReader that can get at the contents of the details sent.
• Read the detail content.

Since I know that the detail content is always a GUID, this is really simple:

catch (FaultException soapEx)

{

    MessageFault mf = soapEx.CreateMessageFault();

    if (mf.HasDetail)

    {

        XmlDictionaryReader reader = mf.GetReaderAtDetailContents();

        Guid g = reader.ReadContentAsGuid();

    }

}

Amazingly, this works.

One of the most useful classes I took in college was an introductory statistics class that was intended to discourage Poli. Sci. majors from continuing in their course of study (I was one at the time). The interesting thing about the class is that it included study and analysis of research as a core part of the class. We learned how to put a study together, how to develop controls, and different formulas used to confirm whether we had statistically significant results or not.

So when Phil Haack announced that Research Supports the Effectiveness of TDD I was more than a little interested in seeing what the linked report actually contained. Phil quotes from the abstract.

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

Phil has obviously read the rest of the report and provides his favorite pieces that seem to do as his title suggests. One of the things I worry about when I see things supporting the latest and greatest software development practices, however, is a strong tendency towards confirmation bias—of looking for confirmation of current theories and overlooking counter-indicators.

So, being the curious type and since TDD is something I'm keeping an eye on to see if its something I might want to adopt myself some day, I went into the report.

The Report

The report itself is remarkably well thought-out. Though their sample size is small (a mere 24 students completed the exercise), it looks like they were very careful to make their study relevant to how software development actually works. They created tasks that built on and eventually altered previous functionality, communicated requirements as user stories (meaning the participants had to design the projects, including class layout and interfaces etc.) and the administrators had a large suite of black-box tests to confirm functional implementation (i.e. tested actual functionality delivered and ignored the structure, design, and tests).

It's important to note that both the TDD and non-TDD groups created and used unit tests. Indeed, the groups are described as "Test First" and "Test Last" throughout the paper.

The authors include their data and results and even mention counter indications (as good report authors should). This is a good thing. If you have a turn for that kind of thing, I recommend reading through it. It isn't long, particularly as these things go.

Lies, Damn Lies, and . . .

Unfortunately, the authors disappoint me when I compare the abstract to the data in the paper. Indeed, the report becomes an example of why I don't give much credence to abstracts anymore. Clever academics have long used abstracts to support the conclusions they desire and this seems to be one such. If you're careful (as these authors are) you can even do it without actually lying. Each of the following statements from the abstract is true:

• The test-first students on average wrote more tests.
• Students who wrote more tests tended to be more productive.
• The minimum quality increased linearly with the number of tests.

Note that only the first is TDD-"Test First"-specific. The other two stand alone, though most readers won't catch that. Here are some equally true statements based on the data (pages 9 and 10 of the report if you want to read along with me in your book):

• The control group (non-TDD or "Test Last") had higher quality in every dimension—they had higher floor, ceiling, mean, and median quality.
• The control group produced higher quality with consistently fewer tests.
• Quality was better correlated to number of tests for the TDD group (an interesting point of differentiation that I'm not sure the authors caught).
• The control group's productivity was highly predictable as a function of number of tests and had a stronger correlation than the TDD group.

So TDD's relationship to quality is problematic at best. Its relationship to productivity is more interesting. I hope there's a follow-up study because the productivity numbers simply don't add up very well to me. There is an undeniable correlation between productivity and the number of tests, but that correlation is actually stronger in the non-TDD group (which had a single outlier compared to roughly half of the TDD group being outside the 95% band).

My Interpretation

One of the things my statistics class pounded home is that correlation doesn't equal causation. That's particularly important to remember in reports like this one. Indeed, I think causation is a real problem with this study, at least insofar as "proving TDD effective" is concerned. It actually serves to undercut many common TDD claims for superiority.

Productivity is an example where causality is far from certain. It makes sense to me that more productive programmers write more tests if only because productive programmers feel like they have the time to do things the way they know they should. Even with "Test First" the emotional effect of "being productive" is going to have an impact on the number of tests you create before moving on to the next phase. (note: you have to be careful here because the study measures tests per functional unit so it's not that productive programmers are creating more tests over all, but rather that they're creating more tests per functional unit.) Frankly, I think it's natural to feel (and respond to) this effect even without a boss hanging over your shoulder waiting to evaluate your work.

In other words, you create more tests because you are a productive programmer rather than being a more productive programmer because you create more tests. At least, it makes sense to me that it would be so.

Truly problematic, however, are the quality results. That's simply a disaster if you propound "Test First" as a guarantor of quality. I mean, sure, the number of unit tests per functional unit suggests a minimum quality through increased testing, but that's only interesting in a situation where minimum quality is important (like, for example, at NASA or in code embedded in medical equipment). The lack of any other correlation here is pretty pronounced any way you care to slice the data. Having a big clump in that upper left quadrant is troubling enough but then having the "Test Last" group almost double your "Test First" group in the over 90% quality range is something that should be noticed and highlighted.

While correlation doesn't equal causation, the lack of correlation pretty much requires a lack of causation.

Since quality is obviously not related to number of tests (at least in this study), I think it is highly disingenuous to highlight the possible connection between number of tests and minimum quality. While it's the only positive thing you can say about quality and unit testing, it is hardly the only thing revealed. The lack of correlation between number of tests and quality is incredibly important and of broad general interest whether you are using formal TDD or not.

Thoughts and Wishes

I really wish that the authors had included a third group that did no unit testing. Since they broadened their scope post-facto to make statements about the efficacy of unit testing over all, it would be nice to have a baseline that didn't do any unit testing yet went through the same process. The study setup was ideal for including such a group as both the productivity and quality measurements were entirely independent of whatever testing was done (or not done).

That said, something occurred to me while reading this study that hasn't before: quality can be a property of unit tests as well as of production code. That's a huge duh, but consider what that could mean. For one, it means that I'd love to see a follow-up that considers the quality of the tests themselves. I would expect to see a relationship between the quality of the product and the quality of the tests if only because quality programmers will be competent in both arenas. This might be such a duh, though, that people assume that test quality matches production quality. Still, I wonder if that is actually the case.

More interesting still would be the correlation between "Testing First" and unit test quality. Ask yourself this, which unit tests are going to be better, the ones created before you've implemented the functionality or those created after? Or, considered another way, at which point do you have a better understanding of the problem domain, before implementing it or after? If you're like me, you understand the problem better after implementing it and can thus create more relevant tests after the fact than before. I know better where the important edge-cases are and how to expose them and that could mean that my tests are more robust.

And yes, I realize after the fact that this is an inversion of section 6.2 which is quoted at length by Phil. I wish the authors had spent as much time in section 6.1 explaining reasons for the quality results instead of what looks to me like an attempt to excuse them. That's the problem with confirmation bias, those results that run counter to expectations aren't as interesting and receive cursory exploration at best.

Anyway, without question, testing first leads to having more tests per functional unit. The question is if this is valuable. This study would seem to indicate that this is probably not the case, at least if quality is your intended gain. But then, I'm not that surprised that number of tests doesn't correspond to quality just as I'm not surprised that the number of lines of code doesn't correspond to productivity.

I thought I'd take a shot at introducing myself here as suggested by Jay and maybe dispel any pretense at being a thinker, heady or no. Frankly, it's probably long past time that I put together some kind of background post about myself if only to give those who disagree with me a way to discount my arguments out of hand.

I suspect this'll be long as I do go on sometimes. You can skip to a recent, similar post if you only want the heart of my current situation.

Street Cred

I've been playing with computers in one way or another since my fifth grade class acquired a then-new PET computer (I recognize the CBM Model 4032 from that link as the one we had). I think it had 8k RAM and we saved and loaded our programs using cassette tapes. I bought my own used Atari 800XL from hard-earned savings as a young teen and have owned a personal computer of some sort continually ever since (frankly, the Atari made it all the way to my Sophomore year of college when PC-envy finally became overwhelming).

It didn't occur to me to take computers seriously as a career until after I graduated college, though. That's right, I went through college working in the computer center and making money on the side setting up personal computers and local networks for small-companies and individuals without realizing that I could actually major in them. The one intro to programming class I took (which I remember was Pascal-based) was so easy (if only from familiarity and years of being a computer geek) that I treated it as a study hall.

After graduation, I discovered that my computer skills were worth more on the market than anything I'd actually studied so that's what I did. This was the mid 1990s so mildly talented lighting fixtures could find programming jobs. Kind of like the mid 2000s, really.

Jacking a Career

I started my actual, salaried career working for a couple of vertical market software vendors who dominated their respective niches by using Pick (which had it all over those stuffy relational databases for the ability to map real-world information). It was with Pick Basic that I discovered the true meaning of Spaghetti Code. With great power comes great responsibility and with great responsibility comes the opportunity to make very large mistakes (none of them mine, of course).

Which means that Visual Basic 6 was an upgrade in sophistication for me. Fortunately, I learn fast—my single greatest career asset. When Visual Studio and the .Net framework came around, I was all over that in turn.

I started my own company with a very talented developer and friend in early 2000 serving the Multi-level Marketing vertical market providing independent compensation programming. If you've ever been in one of those presentations where they describe how you can earn money from the sales of people you recruit and they mention "levels" or "generations" you've pretty much seen the the design documents I was given. Ever wonder how those were programmed? I did, too. I'll just say here that walking a tree with money on the line is an addictive form of adrenaline.

It's a small and competitive niche, though, so it was only really viable while the market was booming. Also, it turns out that I had a lot to learn about marketing and sales.

Developer In Da House

Since then, I've been an in-house developer and/or development manager for various small to medium-sized companies. I've learned enough about how businesses operate to be a good interface to upper management (I can speak all the relevant dialects including accounting, sales, management, and operations) and I work in the trenches enough to not have completely lost the respect of actual developers.

My current position is with a small reading-glasses wholesaler where I am the development department. We're relatively top-heavy in the IT department than you would think we need, but that's because the company's value-add in the marketplace is our ability to orchestrate complex vendor interactions (with up to 120 day lead-times) to provide a flexible product-line at a high-enough volume to satisfy our retailers.

I also contribute to a couple of open source and volunteer projects, the most notable being Subtext. I feel guilty about bringing that one up, though, because it's been months since I've done anything helpful there.

My Hood

Since my in-house projects are small and self-contained, I tend not to use many formal development processes. My users are all within fifty feet of me and I've trained them not to hesitate in letting me know when something goes wrong or when they think of a better way to do something. If you're doing things right in the first place, this isn't really as disruptive as you'd think. My longest project in the last two years has taken less than a month to go from the start of development to deployment. Some of that is because I am just that good, but a lot of it is that we're not doing anything that complex. It's a sweet-spot, I admit.

All of which makes any formal development process so much overkill.

Flashing it Hardcore

I've been around long enough to have seen silver bullets come and go. It has gotten to the point where the only dogma that I'll tolerate is anti-dogma. I am a Microsoft fanboi, but that doesn't mean I don't explore other technologies and ideas. I'm not into didacticism of any flavor.

I do have a core set of practices that I tend to use. They each have two criteria:

• I understand how they work.
• I've used them and verified that they work as expected.

There's nothing like real-life to expose flawed base assumptions or principles.

As you can see, this tends to make me a process skeptic. I love iterative development because I've used it and seen that it works. I like Agile as an idea and implementation of iterative development, but I've never actually seen it in action so I try to keep my discussions of it on a theoretical level.

I've been known to create and maintain the occasional unit test, but I've found that even that can be overkill on many of my projects. Test Driven Development is interesting and all but I can't really see myself using it. It'd be a pain to retrain my way of thinking and I'm not convinced that the payoff is there (and yeah, I've heard the testimonials). I'm not going to move to an experiment until I understand how the payoff is supposed to come. And with TDD, I just don't see it.

This attitude is what keeps getting me in trouble with the Dependency Injection crowd. I've worked out a specific, relatively narrow set of circumstances where it can be useful, but as I found out when I actually tried to implement it, those circumstances turn out to be extremely rare in an already well-architected solution.

Busting Out

Skepticism aside, software development fascinates me and I truly cannot leave new stuff alone. I echoed a sentiment by Jay Kimble recently that says essentially that claims of universal applicability are always wrong. But there's a flip side I want to acknowledge as well: every process, tool, or technology is useful somewhere.

What that means for me is that every popular tool or technique deserves a fair hearing and an attempt at understanding to see if there is something there that I can use. Rejecting something new merely because it is popular is no better than adopting it for the same reason. I admit that this is a weird blend of skepticism and optimism. That said it turns out that, while uncomfortable, it's really useful for growth as a developer and maintaining the ability to jump into new situations and still find yourself in familiar territory.

Software development is fun and I can't seem to be able to leave it alone. Fortunately, it's a useful addiction—not unlike, say, air. And if you've stuck with me this far, you're obviously interested in it yourself (or you're my wife who reads every post if only because she knows I'll eventually ask her what she thought of it). I hope you'll stick around and join the conversation if so moved, even if only to tell me where you think that I am wrong.