Determining how fast (or slow) code runs is surprisingly difficult. Rather than trying to figure it out yourself, use the benchmarking support that’s built into Go’s testing framework. Let’s explore it with a function in the examples/bench directory:
func FileLen(f string, bufsize int) (int, error) {
file, err := os.Open(f)
if err != nil {
return 0, err
}
defer file.Close()
count := 0
for {
buf := make([]byte, bufsize)
num, err := file.Read(buf)
count += num
if err != nil {
break
}
}
return count, nil
}
This function counts the number of characters in a file. It takes in two parameters, the name of the file and the size of the buffer that you are using to read the file (you’ll see the reason for the second parameter in a moment).
Before seeing how fast it is, test your library to make sure that it works:
func TestFileLen(t *testing.T) {
result, err := FileLen("testdata/data.txt", 1)
if err != nil {
t.Fatal(err)
}
if result != 65204 {
t.Error("Expected 65204, got", result)
}
}
The unit tests passes, so now it is time to see how long it takes your file-length function to run. The goal is to find out what size buffer you should use to read from the file.
Note
Before you spend time going down an optimization rabbit hole, be sure that you need to optimize. If your program is already fast enough to meet your responsiveness requirements and is using an acceptable amount of memory, your time is better spent on adding features and fixing bugs. Your business requirements determine what “fast enough” and “acceptable amount of memory” mean.
In Go, benchmarks are functions in your test files that start with the word Benchmark and take in a single parameter of type testing.B. This type includes all the functionality of a testing.T as well as additional support for benchmarking. Let’s start by looking at a benchmark that uses a buffer size of 1 byte:
func BenchmarkFileLen1(b *testing.B) {
for b.Loop() {
_, err := FileLen("testdata/data.txt", 1)
if err != nil {
b.Fatal(err)
}
}
}
Go 1.24 introduced the Loop method on *testing.B. It manages three things that you had to do by hand in earlier versions of Go. The first is looping the proper number of times. The testing framework calls your benchmark functions repeatedly until it is sure that the timing results are accurate. Using b.Loop ensures this is done properly.
The second thing is b.Loop ensures that any code before or after the loop is automatically excluded from the benchmark measurement.
The final thing the b.Loop method does is it prevents optimization. Compilers are pretty smart. They can automatically remove calculations or function calls whose values are never used. This is fantastic when running a program in production, but it’s not so great if you are calling a function in a benchmark to see how fast it is and don’t care about the result. The compiler might remove the function call from the benchmark, which is not helpful.
In order to address this, the Go compiler has a special rule when it sees that it is compiling a benchmark with a for b.Loop loop. In that one case, it makes sure that nothing in the body of the for loop is removed from the compiled code.
You run a benchmark by passing the -bench flag to go test. This flag expects a regular expression to describe the name of the benchmarks to run. Use -bench=. to run all benchmarks. A second flag, -benchmem, includes memory allocation information in the benchmark output. All tests are run before the benchmarks, so you can benchmark code only when tests pass.
Here’s the output for the benchmark on my computer:
BenchmarkFileLen1-16 44 24379298 ns/op 128 B/op 3 allocs/op
Running a benchmark with memory allocation information produces output with five columns. Here’s what each one means:
BenchmarkFileLen1-16
The name of the benchmark, a hyphen, and the value of GOMAXPROCS for the benchmark.
44
The number of times that the test ran to produce a stable result.
24379298 ns/op
The amount of time it took to run a single pass of this benchmark, in nanoseconds (there are a billion nanoseconds in a second).
128 B/op
The number of bytes allocated during a single pass of the benchmark.
3 allocs/op
The number of times bytes had to be allocated from the heap during a single pass of the benchmark. This will always be less than or equal to the number of bytes allocated.
As I covered in “Reducing the Garbage Collector’s Workload”, we care about the number of bytes allocated on the heap and the number of times that our program allocates from the heap, because heap allocations create more work for the garbage collector. The more time your program spends garbage collecting, the less time it spends doing the work you wrote it to do.
Now that you have results for a buffer of 1 byte, let’s see what the results look like when you use buffers of different sizes:
func BenchmarkFileLen(b *testing.B) {
for _, v := range []int{1, 10, 100, 1000, 10000, 100000} {
b.Run(fmt.Sprintf("FileLen-%d", v), func(b *testing.B) {
for b.Loop() {
_, err := FileLen("testdata/data.txt", v)
if err != nil {
b.Fatal(err)
}
}
})
}
}
Just as you launched table tests using t.Run, you’re using b.Run to launch benchmarks that vary based only on input. Here are the results of this benchmark on my computer:
BenchmarkFileLen/FileLen-1-16 49 24230445 ns/op 128 B/op 3 allocs/op
BenchmarkFileLen/FileLen-10-16 489 2458019 ns/op 128 B/op 3 allocs/op
BenchmarkFileLen/FileLen-100-16 4214 269902 ns/op 73376 B/op 657 allocs/op
BenchmarkFileLen/FileLen-1000-16 29570 40574 ns/op 68736 B/op 70 allocs/op
BenchmarkFileLen/FileLen-10000-16 72368 16581 ns/op 82048 B/op 11 allocs/op
BenchmarkFileLen/FileLen-100000-16 62640 19032 ns/op 213121 B/op 5 allocs/op
Parts of these results are expected and parts are surprising. As you increase the size of the buffer, it takes fewer loops to read all the data and your code runs faster, until the buffer is bigger than the file. When the buffer is bigger than the size of the file, extra allocations slow the output. If you expect files of roughly this size, a buffer of 10,000 bytes works best.
In “Allocating Slices”, I mentioned that slices can sometimes be allocated on the stack instead of the heap; this benchmark provides a demonstration. The 1 byte and 10 byte buffers uses little heap memory because they are allocated on the stack. The benchmem flag only shows expensive heap allocations, not cheap stack allocations. For larger buffers, the heap is used.
But you can make a change that improves the numbers more. You are reallocating the buffer every time you get the next set of bytes from the file. That’s unnecessary. If you move the byte slice allocation before the loop and rerun your benchmark, you see an improvement:
BenchmarkFileLen/FileLen-1-16 48 24040199 ns/op 128 B/op 3 allocs/op
BenchmarkFileLen/FileLen-10-16 490 2435033 ns/op 128 B/op 3 allocs/op
BenchmarkFileLen/FileLen-100-16 4680 255315 ns/op 240 B/op 4 allocs/op
BenchmarkFileLen/FileLen-1000-16 34837 34776 ns/op 1152 B/op 4 allocs/op
BenchmarkFileLen/FileLen-10000-16 93732 12751 ns/op 10368 B/op 4 allocs/op
BenchmarkFileLen/FileLen-100000-16 84337 14236 ns/op 106624 B/op 4 allocs/op
The number of allocations is now consistent and small: just three or four for every buffer size. What is interesting is that you now can make trade-offs. If you are tight on memory, you can use a smaller buffer size and save memory at the expense of performance.
If benchmarking reveals that you have a performance or memory problem, the next step is figuring out exactly what the problem is. Go includes profiling tools that gather CPU and memory usage data from a running program as well as tools that help you visualize and interpret the generated data. You can even expose a web service endpoint to gather profiling information remotely from a running Go service. This data can even be used at compile time to improve performance automatically. I talk about instrumentating code and understanding profile data in the related discussion.
benchstat
How do you keep track of performance changes between versions of code and how do you know that the differences are significant? You could eyeball the output and try to remember, but it’s better to use benchstat, a tool built by the Go team that automates the process. Add benchstat to your Go module using:
go get -tool golang.org/x/perf/cmd/benchstat
When capturing benchmarks for benchstat, you need to run them multiple times and save the output to disk. The -count flag controls the number of times each benchmark runs; set it to 10 by default:
go test -bench=BenchmarkFileLen$ -count=10 -benchmem > base.txt
# move the allocation from inside the for loop to before the for loop
go test -bench=BenchmarkFileLen$ -count=10 -benchmem > new.txt
Compare the runs using go tool benchstat base.txt new.txt. Here are the results I obtained by running benchstat for benchmarks on last the two versions of the FileLen function:
$ go tool benchstat base.txt new.txt
goos: darwin
goarch: arm64
pkg: example.com/studynotes/bench
cpu: Apple M3 Max
base.txt new.txt new.txt
sec/op sec/op vs base
FileLen/FileLen-1-16 23.83m ± 1% 23.80m ± 0% ~ (p=0.796 n=10)
FileLen/FileLen-10-16 2.433m ± 1% 2.412m ± 0% -0.85% (p=0.000 n=10)
FileLen/FileLen-100-16 263.3µ ± 2% 249.4µ ± 1% -5.31% (p=0.000 n=10)
FileLen/FileLen-1000-16 38.93µ ± 1% 33.80µ ± 0% -13.18% (p=0.000 n=10)
FileLen/FileLen-10000-16 16.13µ ± 2% 12.69µ ± 2% -21.35% (p=0.000 n=10)
FileLen/FileLen-100000-16 19.02µ ± 2% 14.15µ ± 1% -25.60% (p=0.000 n=10)
geomean 238.1µ 210.5µ -11.63%
base.txt new.txt new.txt
B/op B/op vs base
FileLen/FileLen-1-16 128.0 ± 0% 128.0 ± 0% ~ (p=1.000 n=10) ¹
FileLen/FileLen-10-16 128.0 ± 0% 128.0 ± 0% ~ (p=1.000 n=10) ¹
FileLen/FileLen-100-16 73040.0 ± 0% 240.0 ± 0% -99.67% (p=0.000 n=10)
FileLen/FileLen-1000-16 66.125Ki ± 0% 1.125Ki ± 0% -98.30% (p=0.000 n=10)
FileLen/FileLen-10000-16 80.12Ki ± 0% 10.12Ki ± 0% -87.36% (p=0.000 n=10)
FileLen/FileLen-100000-16 208.1Ki ± 0% 104.1Ki ± 0% -49.97% (p=0.000 n=10)
geomean 10.35Ki 1.277Ki -87.66%
¹ all samples are equal
base.txt new.txt new.txt
allocs/op allocs/op vs base
FileLen/FileLen-1-16 3.000 ± 0% 3.000 ± 0% ~ (p=1.000 n=10) ¹
FileLen/FileLen-10-16 3.000 ± 0% 3.000 ± 0% ~ (p=1.000 n=10) ¹
FileLen/FileLen-100-16 654.000 ± 0% 4.000 ± 0% -99.39% (p=0.000 n=10)
FileLen/FileLen-1000-16 69.000 ± 0% 4.000 ± 0% -94.20% (p=0.000 n=10)
FileLen/FileLen-10000-16 11.000 ± 0% 4.000 ± 0% -63.64% (p=0.000 n=10)
FileLen/FileLen-100000-16 5.000 ± 0% 4.000 ± 0% -20.00% (p=0.000 n=10)
geomean 16.78 3.634 -78.34%
¹ all samples are equal
The first table shows the difference in CPU time, the second the difference in number of bytes allocated on the heap, and the third the difference in number of times the memory had to be allocated from the heap. Each of the numbers is given a variance range (such as ± 1%); lower is better. The p value ranges from 0 to 1, telling you the likelihood that the comparison is valid. Zero means you can trust the result, one means you can’t, and a number inbetween indicates the degree of trust. If the difference between the base case and the new case is not statistically significant, you will see a high value for p; feel free to ignore it. In other cases, a p value less than 0.05 is ideal.
If the variance or the p value is high, ensure that your computer wasn’t busy running other tasks while you were collecting benchmark data. You can also use the -count flag to increase the number of benchmark runs or the -benchtime flag to force each benchmark to run for a set number of seconds (-benchtime=2s for a two second run) or to run a set number of times (-benchtime=100000x to force each benchmark to run 100,000 times).