Category: Best Practice

Testing – Why, When, How

Introduction

Covered

This blog post took a long time to write and is fairly lengthy, but even with that length there are several topics I’d like to cover that I was not able to. This blog post covers how to test for correctness and only correctness. It doesn’t say anything about how to test for performance, usability, or generative testing. I’ll cover those in a future post.

Code

You can find all the code for this blog, and all future blogs, on Github at https://github.com/Jazzepi/p3-blog

Terminology

A quick rundown of terms used in this post. There is a lot of variation in the industry on how people talk about testing. This should clarify up front how I categorize tests, and some other general terms used in this post.

  • Unit Tests: Involve only one component (a class in Java) and are generally very quick to run (on the order of 100s of milliseconds). In JUnit helps us run unit tests, and verify the results.
  • Integration Tests: Involves the composition of multiple subcomponents, but not the entire application stack. Generally interested in exercising the plumbing between those components. Tests that involve exercising your DAOs with a in memory SQL database would be an example. In JUnit is often the test runner for these integration tests, but may or may not need other libraries like rest-assured to provoke a response from the application code.
  • End to End Tests (e2e): Involves the entire application stack from an external user’s point of view. Selenium driven website tests, but can also include HTTP requests against a RESTful API which only expects programmatic interactions (no user facing website). In JUnit is often the test runner for these e2e tests, but will almost certainly need other libraries like rest-assured to provoke a response from the application code.
  • Test Driven Development (TDD): An iterative development style that encourages you to write your tests first, then write your code to make the tests pass.

WHY: You must write tests to preserve credibility

Testing is an essential part of software development because we make promises to our stakeholders and we want to keep those promises. When you tell your customer that your software will do X, Y, and Z and it fails to do Z, then you’ve broken your promise. When you break your promises you lose credibility. Testing is the only true way to prove that your software does what you promise that it will do.

A simple real world example of this is an airbag in a car. Auto manufacturers make a promise to us as a customer that their cars are safe, and in turn the parts manufacturers for the cars make promises to the car makers that their parts are well engineered. The Takata airbag recall is an example of what happens when a company breaks their promise. Not only is Takata on the hook for millions in liability, but people are dead because they didn’t have sufficient quality controls.

In programming, testing is our version of quality control. If you don’t test, you can’t keep your promises and your credibility, along with the credibility of those who use your product as a component, will be damaged. Finally, see the Therac-25 machine for how bad programming can literally kill.

WHEN: Tests are pure overhead

Below is an example of a mocked up user interface. It’s crude, it’s non-functional, but it was cheap to make, and very little will be lost when it is thrown away. Nothing in the below mock will contribute to the delivery of the final product, it is pure overhead, but (and this is key) it is not very much overhead. This mock probably took a few hours to construct. Building a real website with functioning components could take hundreds of man hours. The trade off of a small amount of time spent on getting your ideas correct up front vs a large redesign later on is why we do things like create mocks even though they’re pure overhead.

Just like our above mock tests are pure overhead. Unfortunately, unlike the above mock tests are intimately coupled to the low level implementation details of code. Yes you are testing the API of a given component, and asserting on the values that it returns, but your tests care about what the internals of the API did. If your API change, OR if implementation changes, your tests will have to also change. Therefore if we could be perfectly certain (and we can’t, at least not yet, but people are working on provable software) that our software worked we would never write tests, because we wouldn’t need them to know that our promises will be kept. This is why tests are pure overhead.

WHEN: Write tests when you’ve narrowed the cone of uncertainty to minimize overhead

This below chart is a cone of uncertainty that shows how at the very beginning of the project it’s difficult to make predictions about how long the entire project will take. There are simply too many unknowns. Will this technology really work at scale? Will the service provider we’re using meet it’s SLAs? How many customers will we really have? Is Pinnegar going to call in sick saying “I really lit and will rodqy” at a crucial time? (I was ill and sent in a bizarre message. I still have no idea what I was trying to type.)

Despite the fact that the chart is about predicting how much time something will take, this applies just as equally to whether or not a piece of written code will change over time. If you’re writing a large application, the first component you write will often be refactored, rewritten, and rethought many times as the application integrates more functionality. But the last component you add will not have as many iterative cycles over it simply because you’re very close to the time you’re going to deliver the product to a stakeholder. Even if you wanted to make radical changes it’s just too late in the process to do so.

That said we want to minimize any overhead in our application development process, and this applies to tests just as well as anything else. Therefore I suggest you write your tests only when you have confidence that your design is not going to change. Anytime you change your design, you often times have to throw out any work you’ve spent on tests, and then you have to write them again.

Note that this goes against the fundamentals of TDD which suggest writing your tests first. My problem with this approach is that it assumes that you know what the design of the subcomponent you’re working on in your application apriori. TDD implies that, like Athena from Zeus’s skull, ideas will come emerge fully formed from your mind. TDD proponents also argue that the writing and rewriting of these tests are an acceptable amount of overhead for the advantage of having to go through the tests before writing the code (another point on which I disagree).

In summary, when you’re thinking about when you should be writing tests you should do them at the last possible moment so that they deliver value while minimizing overhead.

How to write tests

Cover as much as you can lower in the testing pyramid

The testing pyramid is the ideal distribution of your tests. The area of the section of the pyramid represents how much of your testing should be done using that methodology. So following the pyramid, we should expect to have far viewer Automated GUI Tests (e2e tests) than unit tests.

This is because the more components you test during a test the longer the tests will take to run, and the more unstable they will become. Because of these two reasons try to get as much coverage out of your unit tests as possible. That said don’t be afraid to cover the same functionality in a unit, integration, and e2e tests. Mainly you should strive to write an exhaustive suite of unit tests, and a smaller, less exhaustive suite of tests on the layers above it.

Isolate your code from dependencies

Unit tests are all about isolating one piece of your application and testing it in absentia of anything else. You should be able to run your unit test anywhere, anytime, without any external dependencies (extra Java libraries are fine).

If your class depends on an external service then you should mock it. Use a mocking framework like Mockito, or write a stub of your own.

If your class depends on time, then freeze time for your unit tests. You can achieve this a number of ways.

For Ruby, there’s a great library called Timecop that allows you to freeze time, and move forward and backwards through it.

For Java, if you’re using Jodatime (which I strongly recommend you use over the pre-Java 8 time libraries) you can use their DateTimeUtils class to set the time. Note that if one of your components care about the TimeZone Joda time caches the timezone provided by the Java standard library so you’ll also need to call  DateTimeZone.setDefault;

For Java, if you’re using the raw System.currentTimeMillis then I recommend that replace it with a TimeSource service which provides the time to your components. This class should normally call through to the System.currentTimeMillis method, but you will be able to mock out the time this way. Here’s a great example.

For Java, if you’re using a component who’s source you do not control that uses raw System.currentTimeMillis then you need to mock it out using PowerMock and Mockito. This is the least-best option.

Don’t be afraid to get a little wet

DRY or don’t repeat yourself is an great rule of thumb for programming in general. If you repeat something that means you need to change it in multiple places if it has to be altered in the future. This is great for code, like an EmailService, that will be used throughout your application in various places. However, when you’re writing code for a test what you’re most interested in is clarity. This means that you should be willing to sacrifice a little DRYness, and do some copy and paste if that makes the test easier to read.

I generally find that if truly understanding a test requires the reader to look through the implementation of more than two private methods (such as setup()), then I should make the test a little more wet, with the express goal of allowing the reader to focus on a single block of code.

Each test should assert the state of only one conceptual thing, but put as many assertions about that concept as you need

You want to know by looking at a unit test name what component failed and what it was doing when it failed. You get the specific values that were in play at the time of the exception in the stack trace the unit test provides.

In the below case we could write two different tests with the same setup, and two different assertions in each one. This would mean that the first failure would not mask the second one. But I find it cumbersome to repeat the same setup multiple times, and moving it into a method just obfuscates the test, and forces the reader to bounce around between multiple sections of the code.

Instead I recommend writing a test with as many assertions as you need as long as those assertions are about the same conceptual idea. In this case we’re testing to make sure that when a child dies, the parent no longer has them in their list. The operation has to pass both these checks to be correctly performed, so we just include them in the same test instead of repeating the setup across both tests.

Note this goes against the popular notion of keeping your tests to one assert. I find this too dogmatic, and tests often fail in two different situations: one while developers are running them in a prospective manner and can easily go through the rerun/fix loop twice them if two asserts are broken, and two when a CI test fails because of flakiness and the first test assert is destined to fail just as badly as the last since there’s a systemic problem in the CI environment.

Source

package com.pinnegar;

import org.junit.Test;

import static org.assertj.core.api.Assertions.assertThat;

public class PersonTest {

    @Test
    public void should_not_have_child_after_death() {
        Person sicklyChild = new Person("Sickly Child 2");
        Person person = new Person("Mike").addChildren(new Person("Child 1"), sicklyChild, new Person("Child 3"));

        sicklyChild.die();

        assertThat(person.getChildren()).hasSize(2);
        assertThat(sicklyChild.getLivingStatus()).isEqualTo(Person.LIVING_STATUS.DEAD);
    }
}

ACT, ARRANGE, ASSERT

Order your tests as much as possible this way, and group the stanzas with blank lines between them. When reviewing a test you’re not familiar with you generally want to ignore the setup, look at what it’s doing, and then see which assertion failed. This allows you to quickly scan a test without reading all the setup. For reference I first learned about this organizational methodology here.

Source

package com.pinnegar;

import org.junit.Before;
import org.junit.Test;

import static org.assertj.core.api.Assertions.assertThat;

public class CalculatorTest {
    @Test
    public void calculator_can_mix_operations() throws Exception {
        //ARRANGE
        Calculator.SubtractingCalculator subtractingCalculator = calculator.subtract(0);
        Calculator.DividingCalculator dividingCalculator = calculator.divide(-4);

        //ACT
        int subtractingAnswer = subtractingCalculator.from(-500);
        int dividingAnswer = dividingCalculator.by(2);

        //ASSERT
        assertThat(subtractingAnswer).isEqualTo(-500);
        assertThat(dividingAnswer).isEqualTo(-2);
    }
}

Write your test names with underscores

Most programming languages use camelCase for method names. That makes a lot of sense when you often times have to type references to those names frequently in other parts of the code. snake_case require you to frequently hold shift and then awkwardly move away from the letters to type the _ while camelCase allows you to keep your fingers on the letters.

I think that’s a great argument against using snake_case in code that will be referenced often in typing. Unit tests are a special case though, in that the title will be written once and read many, many times. For that reason I recommend using snake_case in the names of your test cases instead of camelCase.

Throw exceptions

Always throw checked exceptions from your tests don’t try to catch and handle them there. This prevents you from littering your code with try-catch statements.

If you’re writing a test case that validates that an exception is thrown, then let JUnit validate that for you as shown below. You can also use AssertJ to validate exceptions nicely with Java 8. If you need powerful assertion checking I definitely recommend using AssertJ’s inline method over JUnit’s expected annotation.

Here’s an example of three different testing methods. One doesn’t declare any exceptions to catch, so doesn’t throw them. The middle declares an exception can be thrown, but the underlying code shouldn’t throw one (we expect it to pass). And finally the last one declares an exception can be thrown, and is expected to be thrown during the test (dividing by 0 is a bad idea).

Source

@Test
public void calculator_should_subtract_both_negative_numbers() {
    assertThat(calculator.subtract(-100).from(-3)).isEqualTo(97);
}

@Test
public void calculator_should_divide_by_negative_numbers() throws Exception {
    assertThat(calculator.divide(10).by(-3)).isEqualTo(-3);
}

@Test(expected = IllegalArgumentException.class)
public void calculator_should_throw_exception() throws Exception {
    assertThat(calculator.divide(-100).by(0)).isEqualTo(97);
}

Test diversity

You should always try to run your tests on a variety of platforms. If you’re writing Javascript unit tests, execute them in different browsers. If you’re running e2e tests in a browser, run them in different browsers. If you’ve got a Java program that is destined to be deployed on Linux and Windows, run ALL your tests on both platforms.

Also take advantage of test runners that allow you to randomize the test runs. This will help uncover hidden dependencies between unit tests that you did not anticipate. Be aware that if you set this option on, it may fail an important build like a CI release build, so you should use discretion on where you enable this feature. If your randomizing test runner supports it, make sure that you use seeded randomness and record the seed so that you can replicate the order of failure easily.

Never mock when you don’t have to

Mocks are complicated things. They require a library to setup, and generally you have to provide all the functionality that a normal object would (Mockito mocks have some special knowledge of Java standard library classes and will do things like return empty lists). Therefore you should avoid using them if you can just use a normal Java class. You should, for example, never mock Java’s List interface. Just use a normal ArrayList. That way you won’t end up mocking the .size() function, which is a complete waste of your time.

Never assert on mocks, or the values they return

Anything a mock returns is fake. It cannot be used to tell you something about the real behavior of your code. Therefore asserting on that value is always incorrect. If you ever see the below kind of code you instantly should know something is wrong.

assertThat(mockService.getValue()).isEqualTo(value);

The one exception to this rule is if you’re using a partial mock (sometimes called a spy) which is where you take a real class, and only mock out some of its methods. This is generally the least-best option you have for mocking, as you can’t guarantee that you captured all the functionality of the stubbed out method that you replaced.

Use obviously fake, but meaningful to the test, values when possible

If you’ve got a method call library that splits text on spaces, the way you would test it is by providing it a set of strings and proving that given that you have string “B C D”, you get string “B”, “C”, and “D” out of it. Since the strings B, C, and D can be anything (without spaces) then it makes a lot more sense to test on the string “first second third” or “1 2 3”. Then you can write an assertion like this where the fake values are helpful in debugging the test since 1, 2, and 3 have an obvious order.

package com.pinnegar;

import org.junit.Test;

import static org.assertj.core.api.Assertions.assertThat;

public class RegexTest {
    @Test
    public void test_space_splitter() {
        assertThat("1 2 3".split(" ")).containsExactly("1", "2", "3");
    }
}

Unit Tests: Only test a single class

Only ever test one class in your unit test methods. This keeps them simple and straight forward. If you have three classes Foo, Bar, and Baz then you should have at least three test classes named FooTest, BarTest, and BazTest. You may have more test classes for one of the components if it covers a lot of different functionality, but then you should seriously reconsider refactoring the class, it probably has too many responsibilities.

e2e Tests: Retry, retry, retry

e2e tests are notorious for failing in intermittent ways: something doesn’t load on the page correctly, a shared service is down, the browser is slow, or the network is saturated. You should retry your e2e tests when they fail, and accept the second pass as a pass for the whole suite. However, you should not discard that information. It’s helpful to know which tests are unreliable, and why failures occurred.

e2e Tests: Stress your tests

Run your e2e tests on a regular basis, but also stress test them. When you’re developers are away, use that downtime to run your CI machine on a long loop over your e2e tests. Keep track of the failures over time. You will be able to identify the test that are adding the most instability to your e2e test suite (probably because they’re written wrong) and fix them. It’ll also help you identify when a stability fix actually works.

e2e Tests: Wait, don’t sleep

When running e2e tests you often have to wait on something to be true. You need to wait for components to be present in a browser, or for the database to update with its changes (especially true if you’re not using an ACID compliant database). You should always wait for these components by detecting their presence, not by using a sleep command that forces the test thread to wait. Sleeps are very, very brittle and should be avoided at all costs. I would even recommend modifying your code base to provide special hidden values for the test suite to trigger on rather than adding in a sleep.

Conclusion

Write your tests you need them. Write them in a manner that will ensure that you keep your promises. Write them in a way that keeps them maintainable, readable, and correct throughout their lifetimes.

Image from http://ashishqa.blogspot.com/2012/12/history-of-software-testing.html

Exceptional Exceptions – How to write excellent error messages

Bad error messages increase MTTR

Software errors are an unavoidable part of any software development process. Your application will not always function as planned. Things will go wrong, systems will be down, bugs will creep in, and users will do the wrong thing with your software. The real question is how long will it take you, or the operator who probably isn’t you, to recover from that failure. This is called mean time to recovery (MTTR) and it’s an important measure of the health of any software system. Many factors influence MTTR, but the one I want to focus on for this article are error messages. Note that you want a low MTTR.

Good error messages help operators and developers

When a problem arises, error messages help the software operator understand what has gone wrong. If the problem is part of the operational deployment then the error message will help the operator understand what the root cause of the problem is and quickly remedy it. That error message can also help the operator understand that the problem is not rooted in the operational deployment, but instead is a software issue that needs to be fixed by a developer. Finally, the error message may be an error that crops up during the development process. If that error message is well written and contains the appropriate information it will help the developer quickly fix the problem and improve software delivery times.

What are the essential elements of a good error message?

Report enough context so the reader knows what went wrong

Let’s pretend we have two pieces of Java software, simple-reader and complex-reader. Both of them do some kind of batch processing on XML files.

simple-reader only processes files from a single directory which the operator provides.

complex-reader processes several banks of XML files from disparate sources. This system is a connector between several legacy systems and some more modern ones. It reads these XML records from various sources some. The software fetches some from a FTP server, others from a Hadoop cluster, still more from a shared network drive.

Let’s say that simple-reader throws a FileNotFoundException with the path of the file when it encounters a problem reading from the directory. The operator or developer would receive the path to the file, and be able to remedy the situation

Let’s say that complex-reader also throws a FileNotFoundException with the path of the file when it encounters a problem reading from the directory (be it remote or otherwise). The operator or developer would receive the path to the file, and not know which of the subsystems that complex-reader was trying to fetch the file from when it encountered a problem.

complex-reader needs to be fixed by adding more context to the failure. What subsystem was it speaking to when it failed? What was the path on that subsystem? Was it a network failure or an actual read failure? Could the complex-reader report the number of files it had already processed from that endpoint before it failed? All of this information would help the operator find the real problem and fix it quickly. Probably by identifying a subsystem that’s down, or a networking issue that’s preventing complex-reader from fetching and reading the file.

Comparing and contrasting these two examples it becomes clear that more complex software needs more robust error messages. Give the operator the right amount of context to help them solve their problems.

If possible, help the reader know what to do next

The reader reading an error message should receive a call to action if it isn’t obvious from the message itself. This is especially true if the developer has special knowledge that can be encoded in the error message. You have to be careful with this sort of thing. Good advice encoded in an error message may quickly rot as technology and infrastructures change.

For example, if the complex-reader failed during a connection to read from an FTP site, and this site is often flaky, then the error message may read something like

Tried connecting to the FTP service at [49.132.9.1] [3] times before giving up. Service is often unreliable, try again in a couple of minutes.

This helps the reader know that being unable to connect is not an uncommon problem, that there could be a more serious underlying issue with the networking infrastructure, but that they shouldn’t panic until they’ve run the service after some wait time.

Wrap bad exceptions with in better custom ones

You can’t always control what exceptions are coming out of your software. Often times you use frameworks that throw exceptions while they run and you have no ability to control those. For the cases when you DO have the capacity to catch bad exceptions, the best thing you can do is transform them into more useful ones.

java.net.ConnectException as thrown by Java’s Socket class is a great example of a terrible error message because it doesn’t tell the reader the host and port of the destination system.

package com.mpinnegar;

import java.io.IOException;
import java.net.Socket;

public class Connector {
    public static void main(String[] args) throws IOException {
        new Socket("localhost", 1111).getOutputStream().write("hello world".getBytes());
    }
}

Invoking the above produces this error message

Exception in thread "main" java.net.ConnectException: Connection refused: connect
 at java.net.DualStackPlainSocketImpl.connect0(Native Method)
 at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:589)
 at java.net.Socket.connect(Socket.java:538)
 at java.net.Socket.<init>(Socket.java:434)
 at java.net.Socket.<init>(Socket.java:211)
 at com.mpinnegar.Connector.main(Connector.java:11)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Now let’s make this better by catching the exception and rethrowing it as our own special exception which includes the host and port. In this case I extended Exception, but I tend to extend RuntimeException whenever possible because I think checked exceptions are often more trouble than they’re worth.

package com.mpinnegar;

import java.io.IOException;
import java.net.ConnectException;
import java.net.Socket;
import java.net.UnknownHostException;

public class Connector {
    public static void main(String[] args) throws IOException, BetterConnectException {
        String host = "localhost";
        int port = 1111;
        try {
            new Socket(host, port).getOutputStream().write("hello world".getBytes());
        } catch (ConnectException | UnknownHostException exception) {
            throw new BetterConnectException(host, port, exception);
        }
    }
}

Which produces this error message. Notice that we've preserved the original
stack trace so we don't lose any information about the original exception, but
we've also added our own information

Exception in thread "main" com.mpinnegar.BetterConnectException: Failed trying to connect to host [localhost] on port [1111]
 at com.mpinnegar.Connector.main(Connector.java:15)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.net.ConnectException: Connection refused: connect
 at java.net.DualStackPlainSocketImpl.connect0(Native Method)
 at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:589)
 at java.net.Socket.connect(Socket.java:538)
 at java.net.Socket.<init>(Socket.java:434)
 at java.net.Socket.<init>(Socket.java:211)
 at com.mpinnegar.Connector.main(Connector.java:13)
 ... 5 more

 

Obfuscate sensitive data in your error messages

Logs, and by extension error messages, should never be considered a secure part of an application. They are often read by many different individuals with varying levels of access and authority. They can also live for months or years in archived form.

You should therefore sanitize any sensitive data that you write to an error message or log. The simplest way to do this in Java is to write an overridden .toString() method that doesn’t output any sensitive fields, outputs a small portion of those fields, or that outputs the full field only in some kind of developer mode to help with debugging. Below is an example of a class that obfuscates the user’s credit card number only when the system is setup to do so.

package com.mpinnegar;

import static com.mpinnegar.Sanitizer.sanitize;

public class Person {
    private String firstName;
    private String lastName;
    private String creditCardNumber;

    public Person(String firstName, String lastName, String creditCardNumber) {
        this.firstName = firstName;
        this.lastName = lastName;
        this.creditCardNumber = creditCardNumber;
    }

    //Getters and setters are omitted for brevity
    @Override
    public String toString() {
        return "Person{" +
                "firstName='" + firstName + '\'' +
                ", lastName='" + lastName + '\'' +
                ", creditCardNumber='" + sanitize(creditCardNumber) + '\'' +
                '}';
    }
}


package com.mpinnegar;

public class Sanitizer {
    private static boolean isSanitizing = Boolean.getBoolean(System.getenv("shouldSanitizeData"));
    public static String sanitize(String sensitiveData) {
        if (isSanitizing) {
            return sensitiveData.replaceAll(".", "*");
        } else {
            return sensitiveData;
        }
    }
}

Demarcate your parameterized values

Often you will need to pass values into an error message. This helps the reader understand what the state of the system was when it failed. When passing these error messages in you should demarcate them from the text of the message. I use brackets ([ and ]), but you should use what makes the most sense for you. This is most important when you might output something that could be interpreted as part of the error message itself. Consider the following error message.

Tried printing to the screen, but failed.

What’s lost here is that the error message is actually telling the reader exactly what it tried to print. In this case it’s the empty string, but since there’s no demarcation it’s impossible for a reader to know that. Instead consider this improvement.

Tried printing [] to the screen, but failed.

Validate individual steps and tie your error messages to your code whenever possible

It puts more burden on the programmer, but whenever you can reasonably do so, validate input in a layered fashion so that you can respond with an error message specific to the problem at hand. Consider validating some data a reader has entered into a form which accepts first name, last name, and phone number.

In our example we require all fields except for the phone number, and the first name must be 20 characters or less in length. The last name is unbounded, but must be at least one character, and the phone number must be a 10 digit number after removing everything but the numbers.

Here’s a very simple, and very bad example. It provides almost no feedback to the reader on what went wrong. In our case we have a very simple form, only three fields, but in a real world example you may have tens of fields to deal with.

package com.mpinnegar;

public class BadValidator {

    private static final int FIRST_NAME_FIELD_SIZE_LIMIT = 20;
    private static final int PHONE_NUMBER_FIELD_EXACT_SIZE = 10;
    private String errorMessage;

    boolean validate(String firstName, String lastName, String phoneNumber) {
        if (
                firstName == null || firstName.length() == 0 ||
                lastName == null || lastName.length() == 0 ||
                (phoneNumber != null && phoneNumber.replaceAll("[^0-9]", "").length() != PHONE_NUMBER_FIELD_EXACT_SIZE) ||
                firstName.length() > FIRST_NAME_FIELD_SIZE_LIMIT
            ) {
            errorMessage = "Error found in form data.";
            return false;
        }
        return true;
    }

    public String getErrorMessage() {
        return errorMessage;
    }
}

Now let’s try again applying some of the lessons learned from above about layered validation. Note that it does take more code to do this. You don’t get this for free, but you’re making your application easier to use either when a developer sends a new request that’s bad and gets a reasonable error message back, or when an operator sends a bad request and now knows what they need to fix.

package com.mpinnegar;

public class Validator {

    private static final int FIRST_NAME_FIELD_SIZE_LIMIT = 20;
    private static final int PHONE_NUMBER_FIELD_EXACT_SIZE = 10;
    private String errorMessage;

    boolean validate(String firstName, String lastName, String phoneNumber) {
        if (firstName == null || firstName.length() == 0) {
            errorMessage = missingRequiredValue("firstName");
            return false;
        }
        if (lastName == null || lastName.length() == 0) {
            errorMessage = missingRequiredValue("lastName");
            return false;
        }
        if (phoneNumber != null && phoneNumber.replaceAll("[^0-9]", "").length() != PHONE_NUMBER_FIELD_EXACT_SIZE) {
            errorMessage = "Phone number [" + phoneNumber + "] is invalid. It must be " + PHONE_NUMBER_FIELD_EXACT_SIZE + " characters long considering only the numbers and dropping everything else.";
            return false;
        }
        if (firstName.length() > FIRST_NAME_FIELD_SIZE_LIMIT) {
            errorMessage = "The first name field cannot be longer than " + FIRST_NAME_FIELD_SIZE_LIMIT + " characters. " +
                    "The first name submitted was " + firstName + " which is " + (firstName.length() - FIRST_NAME_FIELD_SIZE_LIMIT) + " characters too long.";
            return false;
        }

        return true;
    }

    private String missingRequiredValue(String valueName) {
        return "Missing [" + valueName + "] which is a required field.";
    }

    public String getErrorMessage() {
        return errorMessage;
    }
}

You can see in the above example how the length of the field is being shared between the error message and the actual code doing the validation. This helps cut down on the rot that can occur where error messages don’t match the code that they’re written against.

Notice that this isn’t perfect. We’ve written in the error message for the phone number that we’re dropping all the non-numeric characters. That’s a textual description of what the regex is doing. There’s nothing that prevents a developer from changing the regex without updating the error message.

Use parameterized logging in Java for your error messages (and other logs)

Your logging framework in Java should support parameterized logging (if you logging framework does not, get a new one). This is when you pass a string with placeholders in it (usually curly braces {}) to the logging framework’s log method, and then you pass a series of objects which replace the placeholders. Consider this log statement which does not use parameterized logging.

logger.debug("You just broke the world by passing me a this bad BigObject [" + objectWithReallyBigToString + "]");

In the above form every time the code passes over this .debug() statement, even if your logger code is not in debug mode, you will pay the cost of the .toString() method on the objectWithReallyBigToString. This is because the code has to resolve the parameter’s value before passing it in.

Instead of that let’s do something better and use the logging framework’s parameterized logging. Notice how we use the {} which is where the logging framework will inject the value returned by objectWithReallyBigToString.toString()

logger.debug("You just broke the world by passing me a this bad BigObject [{}]", objectWithReallyBigToString);

Now because we passed the objectWithReallyBigToString object into the second parameter and let the logging framework build the string itself, we don’t incur the cost of calculating it’s string value each time we pass over that piece of code while we aren’t in debug mode. This can make a BIG performance difference in pieces of the code that are frequently visited.

Conclusion

Whenever you’re programming think about the reader who’s coming along after you whether it be another developer or an operator. They need to be able to quickly ascertain what’s going on. When they come along make sure your error messages don’t look like this.

 

Special thanks to pathslong, MikhailEdoshin, dorkinson, and FranknFreddys for their feedback on this article.