testing: using test factories to increase confidence

2020-11-03

~5 min read

974 words

How can you gain greater confidence that what you’re testing is what you intend? One solution is to automate away a lot of the noise with a test factory.

Test factories are a design pattern that split test cases from the assertions so that you write your assertion once and create multiple instances of the tests automatically.

To see how this might work, I’ll start with an example where I don’t use a factory to evaluate whether a password is valid.

My isPasswordAllowed function takes in a password and checks for a few minimum requirements: length and complexity (i.e. has to have at least one letter and one number).

auth.js

/**
 * @arg {string} Password
 * @returns {boolean}
 */
function isPasswordAllowed(password) {
  return password.length > 6 && /\d/.test(password) && /\D/.test(password)
}
export { isPasswordAllowed }

Now, in my tests I might have:

auth.test.js

test("isPasswordAllowed only allows some passwords", () => {
  expect(isPasswordAllowed("")).toBeFalsy()
  expect(isPasswordAllowed("aaaaaaaaaaaa")).toBeFalsy()
  expect(isPasswordAllowed("12345678")).toBeFalsy()
  expect(isPasswordAllowed("a123456")).toBeTruthy()
  expect(isPasswordAllowed("abcdef1")).toBeTruthy()
})

This works well enough, however, it is also a little difficult to see the gap between what should and what shouldn’t pass. The more cases we add, the harder this can be.

This is one reason why some folks still advocate that there should only be one assertion per test - however, if I’m spending the time setting up a test, I want to be able to test multiple cases at once. That seems particularly true when it’s a pure function like I have here.

A test factory pattern might work well for us here. So, let’s refactor the tests to make use of this pattern:

auth.test.js

describe("isPasswordAllowed", () => {
  const allowedPasswords = ["a123456", "abcdef1"]
  const disallowedPasswords = ["", "abcdefg", "1234567"]
  allowedPasswords.forEach((pwd) => {
    expect(isPasswordAllowed(pwd)).toBeTruthy()
  })
  disallowedPasswords.forEach((pwd) => {
    expect(isPasswordAllowed(pwd)).toBeFalsy()
  })
})

Even better, because I’m generating these tests on the fly, I can dynamically create the test titles:

auth.test.js

describe("isPasswordAllowed", () => {
  const allowedPasswords = ["a123456", "abcdef1"]
  const disallowedPasswords = ["", "abcdefg", "1234567"]

  allowedPasswords.forEach((pwd) =>
    test(`${pwd} should be allowed`, () =>
      expect(isPasswordAllowed(pwd)).toBeTruthy()),
  )
  disallowedPasswords.forEach((pwd) => {
    test(`${pwd} should **not** be allowed`, () =>
      expect(isPasswordAllowed(pwd)).toBeFalsy())
  })
})

Now, if one of the tests were to fail, say I tried to include the password a123 in the allowedPasswords collection, I’d see an output like the following (using Jest):

 FAIL   server  auth.test.js
  isPasswordAllowed
    ✕ a123 should be allowed (4ms)
    ✓ abcdef1 should be allowed (1ms)
    ✓  should **not** be allowed
    ✓ abcdefg should **not** be allowed
    ✓ 1234567 should **not** be allowed

  ● isPasswordAllowed › a123 should be allowed

    expect(received).toBeTruthy()

    Expected value to be truthy, instead received
      false

       6 |   allowedPasswords.forEach((pwd) =>
       7 |     test(`${pwd} should be allowed`, () =>
    >  8 |       expect(isPasswordAllowed(pwd)).toBeTruthy()),
       9 |   )
      10 |   disallowedPasswords.forEach((pwd) => {
      11 |     test(`${pwd} should **not** be allowed`, () =>

      at Object.test (src/utils/__tests__/auth.todo.js:8:38)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 skipped, 6 passed, 8 total
Snapshots:   0 total
Time:        0.504s, estimated 1s

Bonus: Jest-In-Case

Jest-in-case is a library from Atlassian to plug into Jest. It is designed to make cases like I had above easy to test.

Installing it into a project that already uses jest:

yarn add --dev jest-in-case

Then, in a test file, import it:

auth.test.js

import cases from "jest-in-case"

Now, to use it, the API is:

cases(title, tester, testCases)

Frankly, I don’t love the way they named the properties of their API. A tester argument is a confusingly named assertion method. Beyond this small quibble (which I’m sure I’ll change my mind about as soon as someone points out the folly of my ways), using jest-in-case appears straightforward.

Let’s refactor the tests from before to use them now!

auth.test.js

cases(
  "isPasswordAllowed cases",
  (opt) => expect(isPasswordAllowed(opt.password)).toBe(opt.outcome),
  [
    { name: "a123456", password: "a123456", outcome: true },
    { name: "abcdef1", password: "abcdef1", outcome: true },
    { name: "", password: "a123", outcome: false },
    { name: "1234567", password: "1234567", outcome: false },
    { name: "abcdefg", password: "abcdefg", outcome: false },
    { name: "a123", password: "a123", outcome: true }, // this should fail
  ],
)

When I run this test suite now, I see the following:

isPasswordAllowed cases
    ✓ a123456 (1ms)
    ✕ a123 (4ms)
    ✓ abcdef1 (1ms)
    ✓ case: 4
    ✓ 1234567
    ✓ abcdefg

  ● isPasswordAllowed cases › a123

    expect(received).toBe(expected) // Object.is equality

    Expected value to be:
      true
    Received:
      false

      4 | cases(
      5 |   'isPasswordAllowed cases',
    > 6 |   (opt) => expect(isPasswordAllowed(opt.password)).toBe(opt.outcome),
      7 |   [
      8 |     {name: 'a123456', password: 'a123456', outcome: true},
      9 |     {name: 'abcdef1', password: 'abcdef1', outcome: true},

      at Object.<anonymous>.opt (src/utils/__tests__/auth.todo.js:6:52)
      at Object.cb (../node_modules/jest-in-case/index.js:40:20)

Test Suites: 1 failed, 1 total

While the actual case isn’t called out in the stack trace, the name is right there for easy debugging:

isPasswordAllowed cases › a123

In the future, I might take advantage of this more by renaming my first parameter to something like isPasswordAllowed Case: - then reading it would be even closer to English!

Other benefits I see in using jest-in-case:

Each case is independent and isolated. I can see everything that’s involved in a case easily and immediately.
I don’t need multiple loops for different expected outcomes (since that’s now handled in my outcome property) — though this did require a slight refactor in what I’m testing (now I’m evaluating the outcome rather than whether it is truthy/falsy which was fine in this situation, but I can imagine it being more complicated in others).
The name property removes the need for much ceremony in actually writing the test. In fact, jest-in-case inserts the name into a test title automatically.

Wrap Up

Test factories, whether using jest-in-case or not, increase the confidence we get in our tests by increasing the visibility of which cases are being tested and how they differ by separating the cases from the assertions.

Frontend Masters: Javascript Testing Practices & Princples

Test Object Factory

Hi there and thanks for reading! My name's Stephen. I live in Chicago with my wife, Kate, and dog, Finn. Want more? See about and get in touch!