Whatever your stance on the merits or pitfalls of Test Driven Development, I think it’s worthwhile and educational to pay attention to a discussion that’s going on lately.

Testing is crucial. But is unit test focused TDD the right path?

I care about the conversation about TDD because I see serious flaws in the conventional understanding of TDD.

Much of the current view of TDD includes:

  • Units are tested in isolation of the rest of the system.
  • Unit tests are more important than any other form of testing.
  • A “unit” is a class or a function. Nothing larger is a “unit”.
  • If you test more than one class, that’s an integration test.
  • If you test from the API with all resources, that’s a system test. Let QA deal with that later. Isn’t that exactly where waterfall failed?
  • You can’t write any production code without a failing test.
  • You have to write only one test at a time, and it must fail.
  • Tests have to be fast.
  • Therefore, they cannot touch hardware, the file system, other services, or database.
  • Tests should be short.

All of this rubs me the wrong way.
I’ll get to my thoughts later, but my concern about this cemented view of TDD caused me to be very interested in the current talks.

On to the discussion

I came in after the 2nd video, while doing research on Agile and TDD.
I’m not sure if the order matters, but here’s a list of what I know about the discussions.

DHH presentation and posts

David Heinemeier Hansson gave the opening keynote at RailsConf 2014 on April 22.
Fast forward to like minute 30 if you want to jump to the part where he starts talking about TDD. But the rest of the talk is great too.

Some views from the talk:

  • Lots of developers that push TDD make you feel like your code is dirty if you are not using TDD.
  • Driving your design from unit tests is not a good idea.
  • The TDD notion of “tests must be fast” is shortsighted.
  • The faith in TDD can lead to completely forgetting about system testing.
  • The focus on the unit and the unit only doesn’t help with producing a great system.
  • 100% coverage is silly
  • Programmers want software to be a science, but it isn’t. It’s more like creative writing.
  • Good software isn’t like engineering.
  • It’s like writing. Clear concise writing is better than convoluted writing.
  • Clarity is good. So good that clarity should be the number one goal, not test coverage or test speed.
  • Being a good developer is as hard as becoming a good writer.
  • Just like writing, the way to become a good programmer is to write a lot of software, read a lot of software, aim for clarity.

Then he wrote a handful of posts:

These are all interesting reads. Highly recommended.
BTW, I’m not a Ruby or a Rails developer.
His comments apply to any language.

What do Martin Fowler and Kent Beck think of all this?

Kent Beck

Kent Beck has been the main proponent of TDD from the very beginning, as part of Extreme Programming. I’ve read quite a bit from him. So, yes, I’m quite interested in his viewpoint on all of this.

Even though a lot of the extreme views of TDD start with Kent’s writings, I don’t think the dogma is coming from him.

Quote attributed to Kent from DHH talk:

“I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence” - Kent Beck

Awesome!

Martin Fowler

Martin Fowler has written lots of great posts on software in general, and agile methods.
(A great overview of agile is written by Martin: The New Methodology)
I believe it was Martin’s initial contacting DHH over one of the posts that started this conversation.

The discussion, “Is TDD Dead?”

These three are talking about this with a series of public discussions.

Here are the links for the videos so far. (I’ll try to keep this list updated as they come out.)

Is TDD Dead? recorded Google hangouts:
Part 1, Part 2, Part 3, Part 4, Part 5 & 6

Martin also has a summary of the talks here.

My opinions

I love TDD.

But my version of Test Driven Development is probably quite different than yours.

Ever since I read about Test First Programming and TDD in the early days of Extreme Programming (XP), I grabbed on to it like a shipwreck survivor holding on to a flotation device.

When taken as just a darn good idea, and a goal, TDD is superb.

When taken as the only true professional way to do things, it’s just as annoying as waterfall.

TDD often has a misguided focus on units over systems

I think TDD is most useful when used at the system level.

The system is where your users interact with the software, so that’s where your tests are most important.

Focus on what customer need your new functionality is fulfilling.
The first tests should be if your feature actually satisfies the need, as a system.

Functional system tests test from the users perspective.
Isn’t that most important.

I want my code to be agile. To be adaptable. To be refactor friendly.
It’s the user level system tests that I really don’t want to break.

A redesign of part of a system may break many unit tests.
But it better not break any functional system tests.

So isn’t the system level the place where we really want most of our testing effort to go?

Functional system AND unit

Both functional system tests and unit tests are important.
I think functional is more important because that’s what the customer cares about.

Unit tests are great too.
Especially with tricky parts of the system.

I heard recently a great rule of thumb. If you feel like some class or method needs an explanation with a block comment, this bit of code needs a unit test.

Also, unit test the parts that are likely to fail.

The parts you don’t understand a lot.

The parts that keep getting questions about.

Slow tests are fine.

I’ve got 24 hours in a day and a continuous integration / continuous test server running for all 24 of those hours.
Even if all of my tests take 2 hours to run, my entire suite is going to be run 12 times per day.

Are you really checking in your code more frequently than that?

Isn’t that good enough?

Mocking just for isolation is bizarre.

There are reasonable times to mock stuff.

The best reasons I’ve come up with are:

  • Simulate infrequent events such as failures, error codes, etc.
  • Simulate resources that are not ready yet.

I don’t mock to isolate just for testing sake.
I just don’t think that way.

TDD is not dead

But let’s kill the dogma, the snobbery, the elitism.

Software IS quite like writing.

Don’t let your first draft be too precious.

Costly to fix small perfectionism

This is a subtle point.

Parts of your system might be over engineered.
Software engineers are very likely to violate YAGNI and make a class or a subsystem do more is needed for the current release.

Unit tests around classes and subsystems will test more functionality than is necessary. They may also be more rigorous than the real requirements of the system.

What if a unit test fails?
You should fix the code.

What if the failure is NOT reproducible at a system level?
Meaning I can’t write a system test that causes the failure?

Well, then the unit test is either more strict than the system requirements for the unit, or the unit test is testing a feature not used by the system at all.

Then, should you fix the code?

How much time and money are worth the effort to fix it?

Would you delay shipping?

When you have blind faith in TDD, I think you would spend a lot and delay.

I think that’s wrong.