Stephen Freeman Rotating Header Image

Test-Driven Development. A Cognitive Justification?

It’s been a busy week. Michael Feathers has an interesting post on the nature of Test-Driven Development, to which Keith has responded. I think Michael overstated my position on “most” people (it was probably a bar discussion) but over the years I’ve seen a lot of TDD code that doesn’t look right. Incidentally, Tim Mackinnon, who was there, tells the origin of Mocks story at the bottom of this page.

With that out of the way, I’d like to get to the real point of this posting

A Cognitive Justification for Test-Driven Development

Two influences coincided for me at XP2008 this week: Dave Snowden talking about social complexity, including current understanding of the how the mind works, and Naresh Jain pairing to understand different people’s approaches to Test-Driven Development.

Dave has spent a lot of time exploring how decision-making happens. In particular, it turns out that people don’t actually spend their time carefully working out the trade-offs and then picking the best option. Instead, we employ a “first-fit” approach: work through an ordered list of learned responses and pick the first one that looks good enough. All of this happens subconsciously, then our slower rational brain catches up and justifies the existing decision—we can’t even tell it’s happening. Being an expert means that we’ve built up more patterns to match so that we can respond more quickly and to more complicated situations than a novice, which is obviously a good thing in most situations. It can also be a bad thing because the nature of our perception means that experts literally cannot receive certain kinds of information that falls outside their training, not because they’re inadequate people but because that’s how the brain works.

Part of Dave’s practice is concerned with breaking through what he calls this “Expert Entrainment”. He has developed exercises to shuffle our list of response patterns and allow other ideas to break through the crust of skills we’ve worked so hard to acquire. One motivation for doing this is to stop experts jumping to a known solution when they haven’t really understood the situation.

Naresh, meanwhile, is on a mission to pair program with the world to understand how different people approach Test-Driven Development, with an example problem that he uses with everyone. My preference these days is to start with a very specific example of the use of the system and then, as I add more examples, extract structure by refactoring. As we talked this through, Naresh described another programmer who noticed that the problem was an instance of a more general type of system and coded that up directly, there was nothing in his solution that included the language of the example. The other programmer had used his expertise to recognise an underlying solution and short-circuit the discovery process—that’s why we claim higher rates for experience. This programmer was right about his solution, so why did the leap to a design bother me (apart from my own Expert Entrainment)?

Then it struck me, Test-Driven Development, at least as practised by the school that I follow, progresses by focussing on the immediate, on addressing narrow, concrete examples. Don’t worry about all those ideas buzzing around your head for how the larger structure should be, just make a note and park them. For now, just do something to address this little concrete example. Later on, when you’ve gathered some empirical evidence, you can see if you were right and move the code in that direction.

I think what this means is that Test-Driven Development works (or should do) by breaking our first-fit pattern matching. It stops us being expert and steam-rolling over the problem with, literally, the first thing that came into our minds. It forces us out of our comfort zone long enough to consider the real requirements we should be addressing. Even better, starting with a test forces us to think first about the need (what’s the test for that?), and then about a solution that our expert mind is so keen to provide.

Just in case you missed that (and it took me a while to see it), it makes a cognitive difference whether you write the tests first or the code.

The best supporting evidence is Arlo Belshee’s group that implemented Promiscuous Pairing. They found empirically that they were most productive when switching pairs every couple of hours, contrary to what anyone would expect; their view was that were taking advantage of constantly being in a state of “Beginner’s Mind”. Of course, to make TDD work in practice, we still need all that expertise underneath to draw on but to support, not to control.

Personally, I’m constantly surprised at the interesting solutions that come up from being very focussed on the immediate and concrete, with a background awareness of the larger picture. By letting go, I discover more possibilities. Very Zen.

17 Comments

  1. Great post! This is certainly true for me.

    This weekend I’ve been coding up an architectural spike of Twitter; I’ve got some notions about how to do things scalably that I wanted to put to the test. I had been thinking about it a bit during every (frequent) Twitter outage, and had some sketches in my notebook when I sat down to code.

    As I did small incremental bits that solved immediate problems, I was definitely applying a variety of familiar patterns — albeit small ones — semi-instinctively. But because I had put my bigger designs out of my mind, the small pieces added up to a different direction than I was expecting.

    It’s hard for me to know for sure, but I suspect the years of TDD practice have helped in two ways. One, they’ve trained me to attend much more closely to the code of the moment. Two, they have broken my learned responses into smaller pieces, more loosely joined. That is, I think TDD might have refactored the bits in my head similarly to the way it helped me refactor the various code bases.

  2. Michael Feathers says:

    That is interesting. One of the things that I noticed early on about TDD is that is forces me into a reflective mindset. In traditional design it’s easy treat design as a blank sheet. You draw some boxes and mentally proof your collaborations. In the process it’s easy to forget the context.

    When you do TDD, the context is right there staring you in the face over and over again. Each change that you make is in the context of code you or someone else have written, and programming becomes more akin to sculpting or sensitive gardening, an activity where you have to look at what you have and make a decision about what to do next.

    To me, that’s the core of it.. getting people to pay at least as much attention to what is there as they do to their ideas of what should be there.

  3. Colin Jack says:

    Wow, Michaels post resulted in two other equally interesting posts about different topics. Good stuff.

  4. […] Test-Driven Development. A Cognitive Justification? […]

  5. Ilja Preuß says:

    Thanks for your great post – the reasoning certainly rings true for me. I always felt that doing TDD significantly changed the way I tackled a problem, but always struggled putting it into words.

  6. […] You Wanted To Know About MVC and MVP But Were Afraid To Ask The Flawed Theory Behind Unit Testing Test-Driven Development. A Cognitive Justification? TDD, Mocks and […]

  7. Joseph Pelrine says:

    Great post, Steve. All too true. You might wnt to check out this article (http://www.nytimes.com/2007/12/30/business/30know.html) for some more arguments about why it’s a good idea to slow experts down a bit.

  8. @Joseph. Thanks. Nice link.

  9. David Harvey says:

    Great post, Steve, and got me thinking a good deal.

    Breaking the expert mindset can work both for pairing and TDD. I’ve found a real (and constructive) tension between an ingrained habit of approaching problems by thinking about structure and pattern (blame the musician!), and the context-driven imperative of test-first (“this is what the code wants to look like” – thanks to Paul Dyson for this!).

    I don’t think in either case this is _guaranteed_, though – we’ve all got experience of groups descending rapidly into group-think, and there’s no reason to believe that – if we write our tests first – the way the relationships and responsibilities in the code evolve will necessarily be that different from the last time we did it. In either case the onus is on the individual to allow themselves to be challenged out of their well-worn grooves – I’ll concede that promiscuous pairing helps (just so long as the nature of the pairing practice is understood by both, which is of course not guaranteed…) but fundamentally, individuals and groups don’t change without needing to, wanting to, and reflecting.

  10. Kris Kemper says:

    Cognitive Benefit of TDD: Confirmation Driven Development?…

    I recently read an blog post[1] describing a cognitive justification for test driven development. I think the topic is interesting. In doing TDD, there are a variety of reasons for why we write tests first, and it isn’t always to ensure correctne…

  11. […] Test-Driven Development also helps by smoothing the flow, removing the Variable Ratio effect, and, as I wrote before, by breaking us out of the programming Tar Pit long enough to refocus on the real […]

  12. Doug Stewart says:

    Excellent posts and comments. Arriving at solutions organically sometimes cuts down on waste but almost always achieves the better approach – always when assuming relentless refactoring. Short-circuiting expert behavior really enforces the principle that the design will become what it needs to and should be, from its own perspective.

  13. Brian Marick says:

    @mfeathers: “When you do TDD, the context is right there staring you in the face over and over again.”

    That makes me wonder about TDD and learning an existing code base. I’m writing a book on RubyCocoa, planned to introduce TDD fairly early, but have kept bumping it to later and later in the text. Some of the reasons have nothing to do with the topic of this post, but others may. Part of what I have to do with the book is explain how and why a large, complicated framework *makes sense* – how it hangs together as a whole – and the tests somehow seemed to get in the way. Maybe because they focused too much attention on the context, not enough on the concept?

    That may not be true. I used TDD from early on in my previous book, which was about teaching Ruby to novices, and it worked just fine. But it does make me think about “tests as documentation”, something that’s never quite worked out for me. How can we make what is mostly a historical trace of a development process into a teaching text without a whole lot of tinkering and rewriting?

    (If that’s even important: in a long-lived collocated project, nobody might ever need to approach the code base alone. They’d always be paired with someone who knew it and could speak what the tests didn’t say. Things are different with open source, though.)

  14. @Brian. I try to make the TDD effect works at multiple levels in the code, start with high-level tests to at least get me focussed on what I’m trying to achieve, use low-level tests to nudge me in the right design directions.

    I’ve jumped into existing TDD code bases before. I think I look at the code when I’m trying to understand the general structure, and look at the tests when I’m trying to make changes. So I guess, to think of it physically, I have the larger structure of the code behind me, and in that context I have a unit test in front of me which is the gatekeeper to the next thing I want to change. Er, or something like that.

  15. […] “通过单元测试可以改善代码质量”这一观点已经得到广泛认可。培训师、顾问兼咨询师Michael Feathers在最近的帖子中对其提出了质疑。他谈及单元测试、集成测试、TDD和净室软件开发(Clean Room Software Development),认为代码质量是反复思考的结果,仅靠解决bug无法获得。M3P的独立咨询师Steve Freeman进一步阐述了Michael的观点,从“认知方面对TDD进行了分析”,并解释了TDD的好处从何而来。 […]

  16. […] style constraints to re-examine old habits, as in this object-oriented calisthenics exercise. As I once wrote, Test-Driven Development itself can have this property of forcing a developer to stop and think […]

Leave a Reply

Your email address will not be published. Required fields are marked *