Writing a Valuable Test Suite: Strategies and Pitfalls

Automated tests set apart good software development from great. However, when testing practices go wrong it can be a drain on your team’s productivity.

There are plenty of introductory articles on how to write tests. You know the jargon, you can write a test, and you have a high-level idea of why tests are valuable.

We will give you the tools to take your testing strategy to the next level by pointing out what is valuable to test, how to test, and tests that degrade value.

Value-Driven Tests

Consider you have written the below pseudocode and wish to test it:

onPageLoad():
  view.loadingIndicator.isVisible = true
  today = getDayOfWeekAsync()
  view.loadingIndicator.isVisible = false
  If today == Friday:
     view.title = “It’s Friday!”
  Else:
    view.title = “Today is not Friday”

Business Logic

When testing software you should get straight to the “meat” of the code: the business logic. After all, the point of tests is to provide business value. If the application does not follow a business’ rules, that is a valuable thing to know.

If you have a limited time to write tests at least write them for the “happy path”:

Test 1:
   Given that onPageLoad() is called then getDayOfWeekAsync() is called
Test 2:
  Given that getDayOfWeekAsync() returns Friday Then view.title is “It’s Friday!”
Test 3:
  For each X in (Saturday, Sunday, Monday, Tuesday, Wednesday, Thursday)
    Given that getDayOfWeekAsync() returns X Then view.title is “Today is not Friday.”

Each test should verify one concept. A test can have multiple verifications if they share a concept (Clean Code, p. 131). These tests do not mention the loading indicator because it is a separate concern.

Is it valuable to verify that the loading indicator shows and then hides itself? This depends on implementation details. If it does not provide much business value or it is written in a foolproof, cross-cutting way you may not test it.

Exceptional Cases

The next piece to consider is exceptional cases. What if getDayOfWeekAsync() fails for some reason? onPageLoad() may be expected to propagate the error:

Given that getDayOfWeekAsync() fails with SomeError Then onPageLoad() fails with SomeError

Regardless of its responsibility in handling specific errors, onPageLoad() should recover gracefully:

Given that getDayOfWeekAsync() fails Then the loading indicator should hide

If you have been paying attention you would realize the above test fails with the current implementation. This process gives you the opportunity to discover and squash bugs before they leave your desk.

Your Code

Consider this test again:

Given that getDayOfWeekAsync() returns Friday Then view.title is “It’s Friday!”

Do not test that the text drawn on the screen matches the expected value. That is the job of your native UI framework. It is sufficient to verify the correct setup of the title attribute in the unit test.

If you are not confident of your integration with 3rd-party code, consider writing a learning test.

Given that view.title = “Hello World” Then the pixels arranged on the screen create “Hello World”

If both of these tests pass then you can be confident that the view properly draws “It’s Friday!” on the screen.

In other words: Given A → B and B → C, A → C.

This fact is powerful: if each unit test verifies its side of an interface boundary, you can test the entire system using primarily unit tests!

Testable Boundaries

External Boundaries

Consider you have written the below pseudocode and wish to test it:

calculateOrder(lineItems):
  For each X in lineItems:
      orderModel.lineItems.Append(new LineItem(X))
  db.save(orderModel.id, orderModel, otherDbProperties)

Problem: calculateOrder() calls save() on a type you do not own, but you want to verify line items are added and that the updates are saved.

Solution: House the third-party code using a slim Wrapper and verify the mocked wrapper is called.

By moving as much of the logic to your side of the court as possible you place yourself in charge of verifying functionality.

Internal Boundaries

Consider that order calculation increases in scope:

OrderCalculator:
  calculateOrder(inputData):
    orderModel = new OrderModel()
    lineItemCalculator.applyLineItems(inputData, orderModel)
    orderModel.deals = inputData.deals;
    orderModel.subtotal = orderModel.subtotal - sum of deal discounts
    return orderModel

LineItemCalculator:  
  calculateLineItems(inputData, model):
    For each X in inputData.items:
      model.lineItems.Append(new LineItem(X))
    model.subtotal = sum of lineItem subtotals
    model.name = “foo”

If a function under test does not have clear-cut boundaries with its dependencies then it will be difficult to test.

Problem: calculateLineItems() and calculateOrder() share responsibility in calculating orderModel.subtotal, so the value cannot be verified unless an integration test is written.

Problem: Maintainers of calculateLineItems() can tack on additional responsibilities, such as modifying orderModel.name.

Solution: Minimize scope of input and output parameters that enforce a single responsibility for each unit of work:

OrderCalculator:
  getOrder(inputData):
    orderModel = new OrderModel()
    orderModel.lineItems = lineItemCalculator.getLineItems(inputData.lineItems)
    orderModel.deals = inputData.deals;
    orderModel.subtotal = subtotalCalculator.getSubtotal(orderModel.lineItems, orderModel.deals)
    return orderModel

LineItemCalculator:  
  getLineItems(inputData, model):
    For each X in inputData.items:
      lineItems.Append(new LineItem(X))
    return lineItems

SubtotalCalculator:
  getSubtotal(lineItems, deals):
    return sum of lineItem subtotals - sum of deal discounts

A litmus test for boundaries is located in the return value of a function. The only proper place for void mutating functions is at the outermost boundary of the system.

Anti-Examples of Valuable Tests

When developers form a poor opinion of tests it is because the tests are preventing them from doing their job. It is more important to exclude bad tests than it is to include good tests.

All tests have a maintenance cost. When a test does not provide value it is not providing zero value, it is providing negative value.

Tests That Do Not Run

Examples:

• The test is commented out.
• The test is not running in CI/CD.

Why:

• These tests provide the illusion of code coverage when there is none.
• These tests should be integrated in the testing suite or deleted.

Tests That Obscure Acceptance Criteria

Examples:

• Tests that verify obvious behavior, such as a cross-cutting concern.
• Tests that verify behavior unrelated to the test’s name, such as an unmodified attribute.
• A collection of tests that contain duplicate assertions.

Why:

• These tests degrade readability so it is difficult to spot holes in code coverage.
• These tests should be slimmed down to only include valuable verifications.

Tests That Are Prone to Failure

Examples:

• Tests that are very long.
• Test classes with complex data setup.
• Tests coupled with or dependent on one another.
• Tests that inconsistently pass, also known as “flaky tests”.

Why:

• These tests are written to cover code but the work is not yet finished.
• These tests lead to the most frustration but they still hold some value.
• These tests should be refactored or rewritten to maintain existing code coverage.

Closing Thoughts

Unit tests inspire confidence by testing each boundary of the system. Assuming that a function’s dependencies work as expected in both a success and error case, any unit tests against the function can verify that the function also responds correctly.

When you invest time in writing tests, make sure the investment provides value for your team and is worth the cost. Your test code deserves the same loving attention as production code.

Want to work at a company that puts its testing philosophy into consistent practice? Apply at Phase 2 today.