Say you have a web server that appends a line to a log file every time it serves a request. For example, using the NGINX default access log format, one line of the log might look like this:
216.58.210.78 - - [27/Jun/2025:17:55:11 +0000] "GET /css/typography.css HTTP/1.1"
200 3377 "https://martin.kleppmann.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
(That is actually one line; it’s split into multiple lines here for readability.) There’s a lot of information in that line. To interpret it, you need to look at the definition of the log format, which is as follows:
$remote_addr - $remote_user [$time_local] "$request"
$status $body_bytes_sent "$http_referer" "$http_user_agent"
So, this one line of the log indicates that on June 27, 2025, at 17:55:11 UTC, the server received a request for the file /css/typography.css from the client IP address 216.58.210.78. The user was not authenticated, so $remote_user is set to a hyphen (-). The response status was 200 (i.e., the request was successful), and the response was 3,377 bytes in size. The web browser was Chrome 137, and it loaded the file because it was referenced in the page at the URL https://martin.kleppmann.com/.
Though log parsing might seem contrived, it’s a critical part of the operations of many modern technology companies and is used for everything from ad pipelines to payment processing. Indeed, it was a driving force behind the rapid adoption of MapReduce and the “big data” movement.
Simple Log Analysis
Various tools can take these log files and produce pretty reports about your website traffic, but for the sake of exercise, let’s build our own, using basic Unix tools. For example, say you want to find the five most popular pages on your website. You can do this in a Unix shell as follows:
cat /var/log/nginx/access.log | 1
awk '{print $7}' | 2
sort | 3
uniq -c | 4
sort -r -n | 5
head -n 5 6
1
Read the log file. (Strictly speaking, cat is unnecessary here, as the input file could be given directly as an argument to awk. However, the linear pipeline is more apparent when it’s written like this.)
2
Split each line into fields by whitespace, and output only the seventh field from each line, which happens to be the requested URL. In our example line, this URL is /css/typography.css.
3
Alphabetically sort the list of requested URLs. The reason for sorting is to ensure that, if a URL has been requested n times, the sorted file will contain the same URL repeated n times in a row.
4
The uniq command filters out repeated lines in its input by checking whether two adjacent lines are the same. The -c option tells it to also output a counter: for every distinct URL, it reports how many times that URL appeared in the input.
5
The second sort sorts by the number (-n) at the start of each line, which is the number of times the URL was requested. It then returns the results in reverse (-r) order—with the largest number first.
6
Finally, head outputs just the first five lines (-n 5) of input and discards the rest.
The output of that series of commands looks something like this:
4189 /favicon.ico
3631 /2016/02/08/how-to-do-distributed-locking.html
2124 /2020/11/18/distributed-systems-and-elliptic-curves.html
1369 /
915 /css/typography.css
Although the preceding command line likely looks a bit obscure if you’re unfamiliar with Unix tools, it is incredibly powerful. It will process gigabytes of log files in a matter of seconds, and you can easily modify the analysis to suit your needs. For example, if you want to omit CSS files from the report, you can change the awk argument to $7 !~ /.css$/ {print $7}, and if you want to count top client IP addresses instead of top pages, you can change the awk argument to {print $1}, and so on.
We don’t have space in these notes to explore Unix tools in detail, but they are very much worth learning about. Many data analyses can be done in a few minutes using a combination of awk, sed, grep, sort, uniq, and xargs, and they perform surprisingly well.
Chain of Commands Versus Custom Program
Instead of the chain of Unix commands, you could write a simple program to do the same thing. For example, in Python, it might look something like this:
from collections import defaultdict
counts = defaultdict(int) 1
with open('/var/log/nginx/access.log', 'r') as file:
for line in file:
url = line.split()[6] 2
counts[url] += 1 3
top5 = sorted(((count, url) for url, count in counts.items()),
reverse=True)[:5] 4
for count, url in top5: 5
print(f"{count} {url}")
1
Initialize counts to be a hash table that keeps a counter for the number of times we’ve seen each URL. The initial value of each counter is 0.
2
Get the requested URL, which is the seventh whitespace-separated field, from each line of the log (the array index is 6 because Python’s arrays are zero-indexed).
3
Increment the counter for the URL in the current line of the log.
4
Sort the hash table contents by counter value (descending), and take the top five entries.
5
Print out those top five entries.
This program is not as concise as the chain of Unix commands, but it’s fairly readable, and which of the two you prefer is in part a matter of taste. However, besides the superficial syntactic differences between the two, there is a big difference in the execution flow, which becomes apparent if you run this analysis on a large file.
Sorting Versus In-Memory Aggregation
The Python script keeps an in-memory hash table of URLs, where each URL is mapped to the number of times it has been seen. The Unix pipeline example does not have such a hash table, but instead relies on sorting a list of URLs in which multiple occurrences of the same URL are simply repeated.
Which approach is better? It depends on the number of URLs you have. For most small to mid-sized websites, you can probably fit all distinct URLs, and a counter for each URL, in (say) 1 GB of memory. In this example, the working set of the job (the amount of memory to which the job needs random access) depends only on the number of distinct URLs. If there are a million log entries for a single URL, the space required in the hash table is still just one URL plus the size of the counter. If this working set is small enough, an in-memory hash table works fine—even on a laptop.
On the other hand, if the job’s working set is larger than the available memory, the sorting approach has the advantage that it can make efficient use of disks. It’s the same principle we discussed in “Log-Structured Storage”: chunks of data can be sorted in memory and written out to disk as segment files, and then multiple sorted segments can be merged into a larger sorted file. Mergesort has sequential access patterns that perform well on disks (see “Sequential Versus Random Writes on SSDs”).
The sort utility in GNU Coreutils (Linux) automatically handles larger-than-memory datasets by spilling to disk and automatically parallelizes sorting across multiple CPU cores. This means that the simple chain of Unix commands we saw earlier easily scales to large datasets, without running out of memory. The bottleneck is likely to be the rate at which the input file can be read from disk.
A limitation of Unix tools is that they run on a single machine. Datasets that are too large to fit in memory or on the local disk present a problem—and that’s where distributed batch processing frameworks come in.