Fuzzing

One of the most important lessons that every developer eventually learns is that all data is suspect. No matter how well specified a data format is, you will eventually have to process input that doesn’t match what you expect. This doesn’t happen for only malicious reasons. Data can be corrupted in transit, in storage, and even in memory. Programs that process data might have bugs, and specifications for data formats always have corner cases that developers interpret in different ways.

Even when developers do the work of writing good unit tests, it’s impossible to think of everything. As you’ve seen, even 100% unit test code coverage is no guarantee that your code is bug free. You need to supplement unit tests with generated data that could break your program in ways you didn’t anticipate. That’s where fuzzing helps.

Fuzzing is a technique for generating random data and submitting it to code to see whether it properly handles unexpected input. The developer can provide a seed corpus or set of known good data, and the fuzzer uses that as a basis for generating bad input. Let’s see how to use the fuzzing support in Go’s testing tools to discover additional test cases.

Assume that you are writing a program to process data files. You are sending a list of strings but want to efficiently allocate memory, so the number of strings in a file is sent as the first line, while the remaining lines are the lines of text. Here’s the sample function for processing this data:

func ParseData(r io.Reader) ([]string, error) {
    s := bufio.NewScanner(r)
    if !s.Scan() {
        return nil, ErrEmpty
    }
    countStr := s.Text()
    count, err := strconv.Atoi(countStr)
    if err != nil {
        return nil, &ParseError{CountStr: countStr}
    }
    out := make([]string, 0, count)
    for range count {
        hasLine := s.Scan()
        if !hasLine {
            return nil, ErrTooFewLines
        }
        line := s.Text()
        out = append(out, line)
    }
    return out, nil
}

You use a bufio.Scanner to read line-by-line from an io.Reader. If there’s no data to be read, an error is returned. You then read the first line and attempt to convert it to an int named count. If you can’t, an error is returned. Next, you allocate the memory for your slice of strings, and read count number of lines from the scanner. If there aren’t enough lines, an error is returned. If all goes well, you return the lines you read.

A unit test has already been written to validate the code:

func TestParseData(t *testing.T) {
    data := []struct {
        name string
        in   []byte
        out  []string
        err  error
    }{
        {
            name: "simple",
            in:   []byte("3\nhello\ngoodbye\ngreetings\n"),
            out:  []string{"hello", "goodbye", "greetings"},
            err:  nil,
        },
        {
            name: "empty_error",
            in:   []byte(""),
            out:  nil,
            err:  ErrEmpty,
        },
        {
            name: "zero",
            in:   []byte("0\n"),
            out:  []string{},
            err:  nil,
        },
        {
            name: "number_error",
            in:   []byte("asdf\nhello\ngoodbye\ngreetings\n"),
            out:  nil,
            err:  &ParseError{CountStr: "asdf"},
        },
        {
            name: "line_count_error",
            in:   []byte("4\nhello\ngoodbye\ngreetings\n"),
            out:  nil,
            err:  ErrTooFewLines,
        },
    }
    for _, d := range data {
        t.Run(d.name, func(t *testing.T) {
            r := bytes.NewReader(d.in)
            out, err := ParseData(r)
            if !errors.Is(err, d.err) {
                t.Errorf("expected %v, got %v", d.err, err)
            }
            if diff := cmp.Diff(d.out, out); diff != "" {
                t.Error(diff)
            }
        })
    }
}

The unit test has 100% line coverage for ParseData, handling all its error cases. You might think the code is ready for production, but let’s see if fuzzing can help you discover errors that you hadn’t considered.

Note

Be aware that fuzzing uses a lot of resources. A fuzz test can allocate (or attempt to allocate) many gigabytes of memory and might write several gigabytes of data to your local disk. If you are running something else on the same machine at the same time as a fuzz test, don’t be surprised if it slows down.

Start by writing a fuzz test:

func FuzzParseData(f *testing.F) {
    testcases := [][]byte{
        []byte("3\nhello\ngoodbye\ngreetings\n"),
        []byte("0\n"),
    }
    for _, tc := range testcases {
        f.Add(tc) // Use f.Add to provide a seed corpus
    }
    f.Fuzz(func(t *testing.T, in []byte) {
        r := bytes.NewReader(in)
        out, err := ParseData(r)
        if err != nil {
            t.Skip("handled error")
        }
        roundTrip := ToData(out)
        rtr := bytes.NewReader(roundTrip)
        out2, _ := ParseData(rtr)
        if diff := cmp.Diff(out, out2); diff != "" {
            t.Error(diff)
        }
    })
}

A fuzz test looks similar to a standard unit test. The function name starts with Fuzz, the only parameter is of type *testing.F, and it has no return values.

Next, set up a seed corpus, which is composed of one or more sets of sample data. The data can run successfully, error out, or even panic. The important things are that you know how your program behaves when this data is provided and that this behavior is accounted for by your fuzz test. These samples are mutated by the fuzzer to generate bad inputs. This example uses only a single field of data for each entry (a slice of bytes), but you can have as many fields as needed. The fields in a corpus entry are currently limited to only certain types:

• Any integer type (including unsigned types, rune, and byte)

• Any floating-point type

• bool

• string

• []byte

Each entry in the corpus is passed to the Add method on the *testing.F instance. In the example, you have a slice of bytes for each entry:

f.Add(tc)

If the function being fuzzed needed an int and a string, the call to Add would look like this:

f.Add(1, "some text")

It’s a runtime error to pass a value of an invalid type to Add.

Next, call the Fuzz method on your testing.F instance. This looks a bit like a call to Run when writing a table test in a standard unit test. Fuzz takes in a single parameter, a function whose first parameter is of type testing.T and whose remaining parameters exactly match the types, order, and count of the values that were passed in to Add. This also specifies the type of data generated by the fuzzing engine during fuzzing. There’s no way for the Go compiler to enforce this constraint, so it’s a runtime error if the convention is not followed.

Finally, let’s look at the body of the fuzz target. Remember, fuzzing is used to find cases where bad input is not handled correctly. Since the input is randomly generated, you can’t write tests that have knowledge of what the output should be. Instead, you have to use test conditions that will be true for all inputs. In the case of ParseData, you can check for two things:

• Does the code return an error for bad input, or does it panic?

• If you convert the slice of strings back to a slice of bytes and re-parse it, do you get the same result?

Let’s see what happens when you run the fuzz test:

$ go test -fuzz=FuzzParseData
fuzz: elapsed: 0s, gathering baseline coverage: 0/137 completed
failure while testing seed corpus entry: FuzzParseData/03f81b404ad91d09
fuzz: elapsed: 3s, gathering baseline coverage: 2/137 completed
--- FAIL: FuzzParseData (2.92s)
    --- FAIL: FuzzParseData (0.00s)
        testing.go:1927: panic: runtime error: makeslice: cap out of range
            goroutine 12 [running]:
            runtime/debug.Stack()
                /opt/homebrew/opt/go/libexec/src/runtime/debug/stack.go:26 +0xc4
            testing.tRunner.func1()
                /opt/homebrew/opt/go/libexec/src/testing/testing.go:1927 +0x218
            panic({0x100ca49c0?, 0x100ce5dc0?})
                /opt/homebrew/opt/go/libexec/src/runtime/panic.go:860 +0x12c
            file_parser.ParseData({0x100ce7a40?, 0x10e411dfcf30?})
                file_parser/file_parser.go:46 +0x134

If you don’t specify the -fuzz flag, your fuzz tests will be treated like unit tests and will be run against the seed corpus. Only a single fuzz test can be fuzzed at a time.

Note

If you want to get the full experience, delete the contents of the testdata/fuzz/FuzzParseData directory. This will cause the fuzzer to generate new seed corpus entries. Since the fuzzer generates random input, your samples might be different from the ones shown. The different entries will likely produce similar errors, though maybe not in the same order.

The fuzz test runs and in less than a second, it reports a panic. Panics are bad, so let’s look at the input that was generated. Every time a test case fails, the fuzzer writes it to the testdata/fuzz/TESTNAME subdirectory in the same package as the failed test, adding a new entry to the seed corpus. This automatically generated unit test runs when go test -fuzz=FuzzParseData is called, and acts as a regression test once you fix your bug.

Here are the contents of the file:

go test fuzz v1
[]byte("-1")

The first line is a header to indicate that this is test data for a fuzz test. The subsequent lines have the data that caused the failure.

Passing the name of the file in the failure message as a -run parameter to go test allows you to isolate this failing case when you rerun the test:

$ go test -run=FuzzParseData/03f81b404ad91d09
--- FAIL: FuzzParseData (0.00s)
    --- FAIL: FuzzParseData/03f81b404ad91d09 (0.00s)
panic: runtime error: makeslice: cap out of range [recovered, repanicked]

The line that’s generating the panic is shown here:

out := make([]string, 0, count)

You are trying to create a slice with negative capacity, which panics. An error type for this situation has been handily defined for you. Add a condition to your code to look for negative numbers before you call make:

if count < 0 {
    return nil, ErrNegative
}

If you run go test -run=FuzzParseData/03f81b404ad91d09, you’ll see that the test now passes. Run the fuzzer again to see whether it finds any more errors:

$ go test -fuzz=FuzzParseData
fuzz: elapsed: 0s, gathering baseline coverage: 0/137 completed
failure while testing seed corpus entry: FuzzParseData/0e4ad8073b1a5373
fuzz: elapsed: 3s, gathering baseline coverage: 2/137 completed
--- FAIL: FuzzParseData (2.95s)
    --- FAIL: FuzzParseData (0.00s)
        file_parser_test.go:87:   []string{
              ... // 3 identical elements
              "",
              "",
            -  "\r",
            +  "",
              "0",
            }

This time, the input file created by the fuzzer looks like this:

go test fuzz v1
[]byte("7\n\n\n\n\n\n\r\r\n0")

The fuzz test is generating blank lines with \r (return) characters. Blank lines are not expected in your input, so add some code to the loop that reads lines of text from the Scanner. You’ll check whether a line consists only of whitespace characters. If it does, return an error:

line = strings.TrimSpace(line)
if len(line) == 0 {
    return nil, ErrBlankLine
}

Validate that you fixed the bug with go test -run=FuzzParseData/0e4ad8073b1a5373, then run the fuzzer again to see whether it finds any more errors:

$ go test -fuzz=FuzzParseData
fuzz: elapsed: 0s, gathering baseline coverage: 0/137 completed
fuzz: elapsed: 3s, gathering baseline coverage: 136/137 completed
fuzz: elapsed: 6s, gathering baseline coverage: 136/137 completed
fuzz: elapsed: 9s, gathering baseline coverage: 136/137 completed
failure while testing seed corpus entry: FuzzParseData/e8bf251ff804c663
fuzz: elapsed: 11s, gathering baseline coverage: 136/137 completed
--- FAIL: FuzzParseData (10.83s)
    fuzzing process hung or terminated unexpectedly: exit status 2

In this case, the go command reports that your program has “hung or terminated unexpectedly”. We don’t want programs to hang or terminate, expectedly or unexpectedly, so let’s look at the generated test case to figure out what went wrong:

go test fuzz v1
[]byte("3000000000000")

The problem is that you are trying to allocate a slice with the capacity to hold 3 trillion strings. That requires quite a bit more RAM than my computer (and probably yours) has. You need to limit the number of expected text elements to a reasonable number. Use the sentinel error ErrTooManyLines and set the maximum number of rows to 1,000 by adding the following code to ParseData after you parse the number of expected rows:

if count > 1000 {
    return nil, ErrTooManyLines
}

Validate that the bug is fixed with go test -run=FuzzParseData/e8bf251ff804c663, and run the fuzzer yet again:

$ go test -fuzz=FuzzParseData
fuzz: elapsed: 0s, gathering baseline coverage: 0/147 completed
fuzz: elapsed: 0s, gathering baseline coverage: 147/147 completed,
    now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 1568753 (522795/sec), new interesting: 1 (total: 148)
fuzz: elapsed: 6s, execs: 3134921 (522087/sec), new interesting: 1 (total: 148)
fuzz: elapsed: 9s, execs: 4652294 (505854/sec), new interesting: 1 (total: 148)
...
fuzz: elapsed: 2m9s, execs: 59383661 (416674/sec), new interesting: 2 (total: 149)
fuzz: elapsed: 2m12s, execs: 60694659 (436957/sec), new interesting: 3 (total: 150)
fuzz: elapsed: 2m15s, execs: 61976072 (427059/sec), new interesting: 3 (total: 150)
^C
PASS

After a few minutes, no more errors are found, so hit Ctrl-C to end fuzzing. You can also use the -fuzztime flag to specify a time limit (-fuzztime=60s for a 60 second run) or a number of iterations (-fuzztime=100000x to run the fuzzer 100,000 times).

Just because the fuzzer didn’t find additional issues doesn’t mean that the code is now bug free. However, fuzzing allowed you to automatically find some significant oversights in the original code. Writing fuzz tests takes practice, as they require a slightly different mindset than writing unit tests. Once you get the hang of them, they become an essential tool for validating how your code handles unexpected user input. Once you have a library of fuzz tests, run them periodically on your CI infrastructure, using -fuzztime to timebox the work.