Now that I’ve talked about slices, we can look at strings again. You might think that a string in Go is made out of runes, but that’s not the case. Under the covers, Go uses a sequence of bytes to represent a string. These bytes don’t have to be in any particular character encoding, but several Go library functions (and the for-range loop that I discuss in this topic) assume that a string is composed of a sequence of UTF-8-encoded code points.
Note
According to the language specification, Go source code is always written in UTF-8. Unless you use hexadecimal escapes in a string literal, your string literals are written in UTF-8.
Just as you can extract a single value from an array or a slice, you can extract a single value from a string by using an index expression:
var s string = "Hello there"
var b byte = s[6]
Like arrays and slices, string indexes are zero-based; in this example, b is assigned the numeric value of the seventh position in s, which is 116 (the UTF-8 value of a lowercase t).
The slice expression notation that you used with arrays and slices also works with strings:
var s string = "Hello there"
var s2 string = s[4:7]
var s3 string = s[:5]
var s4 string = s[6:]
This assigns “o t” to s2, “Hello” to s3, and “there” to s4. You can try out this code on The Go Playground.
While it’s handy that Go allows you to use slicing notation to make substrings and use index notation to extract individual entries from a string, you should be careful when doing so. Since strings are immutable, they don’t have the modification problems that slices of slices do. There is a different problem, though. A string is composed of a sequence of bytes, while a code point in UTF-8 can be anywhere from one to four bytes long. The previous example was entirely composed of code points that are one byte long in UTF-8, so everything worked out as expected. But when dealing with languages other than English or with emojis, you run into code points that are multiple bytes long in UTF-8:
var s string = "Hello \U0001F30E"
var s2 string = s[4:7]
var s3 string = s[:5]
var s4 string = s[6:]
“o 🌎.”
In this example, s3 will still be equal to “Hello.” The variable s4 is set to the world emoji. But s2 is not set to “o 🌎.” Instead, you get “o <img src="../../../go/lessons/images/emoji/replacement.png"/>].” That’s because you copied only the first byte of the world emoji’s code point, which is not a valid code point on its own.
Go allows you to pass a string to the built-in len function to find the length of the string. Given that string index and slice expressions count positions in bytes, it’s not surprising that the length returned is the length in bytes, not in code points:
var s string = "Hello \U0001F30E"
fmt.Println(len(s))
This code prints out 10, not 7, because it takes four bytes to represent the world emoji in UTF-8. You can run these world emoji examples on The Go Playground.
Warning
Even though Go allows you to use slicing and indexing syntax with strings, you should use it only when you know that your string contains only characters that take up one byte.
Because of this complicated relationship among runes, strings, and bytes, Go has some interesting type conversions between these types. A single rune or byte can be converted to a string:
var a rune = 'x'
var s string = string(a)
var b byte = 'y'
var s2 string = string(b)
Warning
A common bug for new Go developers is to try to make an int into a string by using a type conversion:
var x int = 65
var y = string(x)
fmt.Println(y)
This results in y having the value “A,” not “65.” The go vet tool blocks a type conversion to string from any integer type other than rune or byte.
A string can be converted back and forth to a slice of bytes or a slice of runes. Try Example 2-9 on The Go Playground.
Example 2-9. Converting strings to slices
var s string = "Hello, \U0001F30E"
var bs []byte = []byte(s)
var rs []rune = []rune(s)
fmt.Println(bs)
fmt.Println(rs)
When you run this code, you see the following:
[72 101 108 108 111 44 32 240 159 140 141]
[72 101 108 108 111 44 32 127757]
The first output line has the string converted to UTF-8 bytes. The second has the string converted to runes.
Most data in Go is read and written as a sequence of bytes, so the most common string type conversions are back and forth with a slice of bytes. Slices of runes are uncommon.
UTF-8
UTF-8 is the most commonly used encoding for Unicode. Unicode uses four bytes (32 bits) to represent each code point, the technical name for each character and modifier. Given this, the simplest way to represent Unicode code points is to store four bytes for each code point. This is called UTF-32. It is mostly unused because it wastes so much space. Due to Unicode implementation details, 11 of the 32 bits are always zero. Another common encoding is UTF-16, which uses one or two 16-bit (2-byte) sequences to represent each code point. This is also wasteful; much of the content in the world is written using code points that fit into a single byte. And that’s where UTF-8 comes in.
UTF-8 is clever. It lets you use a single byte to represent the Unicode characters whose values are below 128 (which includes all of the letters, numbers, and punctuation commonly used in English), but expands to a maximum of four bytes to represent Unicode code points with larger values. The result is that the worst case for UTF-8 is the same as using UTF-32. UTF-8 has some other nice properties. Unlike UTF-32 and UTF-16, you don’t have to worry about little-endian versus big-endian. It also allows you to look at any byte in a sequence and tell if you are at the start of a UTF-8 sequence or somewhere in the middle. That means you can’t accidentally read a character incorrectly.
The only downside is that you cannot randomly access a string encoded with UTF-8. While you can detect if you are in the middle of a character, you can’t tell how many characters in you are. You need to start at the beginning of the string and count. Go doesn’t require a string to be written in UTF-8 but strongly encourages it. You’ll see how to work with UTF-8 strings in upcoming chapters.
Fun fact: UTF-8 was invented in 1992 by Ken Thompson and Rob Pike, two of the creators of Go.
Rather than use the slice and index expressions with strings, you should extract substrings and code points from strings using the functions in the strings and unicode/utf8 packages in the standard library. In this topic, you’ll see how to use a for-range loop to iterate over the code points in a string.
Standard Library Functions for string and bytes
The standard library contains many functions for working with strings in the aptly-named strings package. If you want to search, compare, split, or trim a string, chances are the standard library provides what you need. Before writing your own implementation, check to make sure that the work hasn’t been done for you.
The functions defined in the strings package have doppelgangers in the bytes package. Data from a network connection or disk is often loaded into memory as a slice of bytes. Rather than converting the bytes to a string, you can work directly on the bytes slice.
You may also notice that some of the functions defined in the bytes package have analogues in the slices package. If you are working with a byte slice, use the functions in the bytes package, since they are optimized for byte slices.