Slices as Buffers

When reading data from an external resource (like a file or a network connection), many languages use code like this:

r = open_resource()
while r.has_data() {
  data_chunk = r.next_chunk()
  process(data_chunk)
}
close(r)

The problem with this pattern is that every time you iterate through that while loop, you allocate another data_chunk, even though each chunk is used only once. This creates lots of unnecessary memory allocations (much like the Unmarshal function). Garbage-collected languages handle those allocations for you automatically, but the work still needs to be done to clean them up when you’re done processing.

Even though Go is a garbage-collected language, writing idiomatic Go means avoiding unneeded allocations. Rather than returning a new allocation each time you read from a data source, create a slice of bytes once and use it as a buffer to read data from the data source:

file, err := os.Open(fileName)
if err != nil {
    return err
}
defer file.Close()
data := make([]byte, 100)
for {
    count, err := file.Read(data)
    process(data[:count])
    if err != nil {
        if errors.Is(err, io.EOF) {
            return nil
        }
        return err
    }
}

Remember, you can’t change the length or capacity of a slice when it is passed into a function. However, you can change its contents up to the current length. In this code, you create a buffer of 100 bytes. Each time through the loop, you copy the next block of bytes (up to 100) into the slice. You then pass the populated portion of the buffer to process. If an error happens, it is returned. The exception is io.EOF, a special-case error that indicates there is no more data to read; when io.EOF is returned as the error, there is no more data and the function returns nil.