Choice of Words in Testing Frameworks (...and how many get it wrong, including RSpec)

One [word] to rule them all... and in the darkness bind them.

I want my testing framework to be able to express the ideas that I have in my mind in the best possible way. This includes giving the tests the best possible names. Unfortunately, lots of testing frameworks force the developer to start or end his test names with predefined words - such as "define", "it", "should", "given", "when", "then". This can be harmful, because they incline the user to write his test names always following the same structure, even in situations where that way of structuring tests is suboptimal.

Predefined words produce twisted sentences

Here are some trivial counterexamples of specification-style test names, to prove that requiring the test to start with a predefined word will sometimes lower the quality of the test names. Here is a specification of Fibonacci numbers which is taken straight from their Wikipedia article:

Fibonacci numbers:
- The first two Fibonacci numbers are 0 and 1
- Each remaining number is the sum of the previous two

RSpec requires the test fixtures to start with the word "define" and the tests with "it", in addition to which RSpec's documentation encourages the tests to start with "it should". Let's try to twist the above specification of Fibonacci numbers into that style. Here is my best attempt which still holds the same information:

Define Fibonacci numbers:
- it should have the first two numbers be 0 and 1
- it should have each remaining number be the sum of the previous two

Urgh. Totally unnatural way of saying the same information. Lots of unnecessary words need to be added to make the test names full sentences.

If we use example-style, then it's possible to write the test names, but valuable information is lost:

Define Fibonacci numbers:
- it should calculate the first two numbers
- it should calculate each remaining number

The problem appears to be that "it" is forced to be the subject of the sentence. We can get around that restriction by adding more "define" elements, but then the tests become awfully verbose without adding any new information:

Define Fibonacci numbers:
- Define the first number:
  - it should be 0
- Define the second number:
  - it should be 1
- Define each remaining number:
  - it should be the sum of the previous two

Here is another example, written in a slightly different style:

Stack:
- An empty stack
    - is empty
    - After a push, the stack is no longer empty
- When objects have been pushed onto a stack
    - the object pushed last is popped first
    - the object pushed first is popped last
    - After popping all objects, the stack is empty

And the same using RSpec's predefined words:

Define stack:
- Define an empty stack
    - it should be empty
    - it should, after a push, be no longer empty
- Define a stack onto which objects have been pushed
    - it should pop first the object pushed last
    - it should pop last the object pushed first
    - it should, after popping all objects, be empty

It was necessary to change the order of some of the sentences for them to make sense, and it was not natural to write the "it should, after..." tests - their sentence order should have been changed to make them more natural, but then the effect would have been before the cause, which is neither good. Also the subject of some sentences had to be changed from "the object pushed last" to "it" (i.e. stack) and the subject of the old sentence became the object of the new sentence.

The testing framework should obey the developer, not the other way around! The developer is the one knows best that how to make a sentence convey his intent. A testing framework, which forces the developer to use a predefined style of writing his sentences, is immoral!

Predefined words do not improve the test names

What about example-style test names? Predefined words are equally bad for them. Here are Uncle Bob's Bowling Game Kata's test names in RSpec format:

Define bowling game:
- it should score gutter game
- it should score all ones
- it should score one spare
- it should score one strike
- it should score perfect game

Adding "it should" does not improve the test names. They don't make the intent any clearer. It just adds lots of duplication and becomes background noise. The framework will not magically make a person who writes example-style tests to suddenly start writing specification-style tests.

What about implementation-style tests? I have seen lots of implementation-style tests written in an "it should have" pseudo-specification-style like this:

Define person:
- it should have name
- it should not allow null name
- it should have age
- it should have address
- it should save
- it should load
- it should calculate pay

Writing implementation-style tests is still perfectly possible. A framework alone can't make the developer better. He must first understand the philosophy behind the framework and how to write expressive tests, before his tests will get any better.

Many testing frameworks get it wrong

behaviour-driven.org says that "Getting the words right" was the starting point for the development of BDD, so it is absurdly ironic that lots of BDD frameworks get the words wrong.

First and foremost, RSpec gets its words wrong by forcing the tests to start with "describe" and "it", as described above. And because RSpec has become popular and was one of the first BDD frameworks, lots of other BDD frameworks copy RSpec and use the same predefined words. They just repeat mindlessly what others have done, without stopping to think why the things were done that way. They become angry monkeys and cargo-cults, which annoy me very much.

Many BDD acceptance testing frameworks force the use of words "given", "when", "then". For example Cucumber does this. Decomposing actions into those three parts gives you state transition tables. This is a very explicit way of defining actions, but also very verbose. Added verbosity does not always make things easier to read; on the contrary, it can make it harder to see what is really important. As said in a famous quote:

In anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away.
- Antoine de Saint-Exupéry

And another one:

Many attempts to communicate are nullified by saying too much.
- Robert Greenleaf

I have even found a framework which adds predefined words as suffixes to the test names. In the Specs framework for Scala, the top-level test names end with "should", which is even printed in all the reports (unlike RSpec's "it"). Thankfully there is a workaround to avoid that suffix.

Frameworks should not limit the developer

A good testing framework will allow the developer to choose himself in what way he writes his test names. Some examples of testing frameworks, which do not force the use of predefined words, are JUnit 4 and JDave. But those two still force a fixed level of test nesting - JUnit 4 has no nesting and JDave has one level of nesting. Of the frameworks that I know, the least restrictive one is GoSpec, which I wrote myself with that as a goal.

When designing GoSpec, my goals were to allow unlimited levels of nesting, and to not force the test names to start with any predefined words. In Scala and other similarly expressive languages it would be easy to cope without any predefined words. For example I like in Specs the "test name" in { ... } style, which can also be written using an unpronounceable symbol "test name" >> { ... }. Unfortunately Go's syntax is not as flexible, so I was satisfied with prefixing each test name with "Specify". I chose that word to be such that starting a sentence with it would be totally unnatural, so that the developers would be inclined to just ignore it. Also all the examples of using that framework are written so that they do not include that word in the test names.

In a future article I will write about my current ideals for a unit testing framework. One of the primary goals is allowing the developer to use any style that is best for the situation, but there are also other goals (for a sneak peek, see the project goals in GoSpec's README).