Monotonic clocks don’t require synchronization, but time-of-day clocks need to be set according to an NTP server or other external time source in order to be useful. Unfortunately, our methods for getting a clock to tell the correct time aren’t nearly as reliable or accurate as you might hope—hardware clocks and NTP can be fickle beasts. To give just a few examples:
-
The quartz clock in a typical computer is not very accurate: it drifts (runs faster or slower than it should). Clock drift varies depending on the temperature of the machine. Google assumes a clock drift of up to 200 ppm (parts per million) for its servers, which is equivalent to a 6 ms drift for a clock that is resynchronized with a server every 30 seconds, or a 17-second drift for a clock that is resynchronized once a day. This drift limits the best possible accuracy you can achieve, even if everything is working correctly.
-
If a computer’s clock differs too much from an NTP server, it may refuse to synchronize or be forcibly reset. Any applications observing the time before and after this reset may see time go backward or suddenly jump forward.
-
If a node is accidentally firewalled off from NTP servers, the misconfiguration may go unnoticed for some time, during which the drift may add up to large discrepancies between that and other nodes’ clocks. Anecdotal evidence suggests that this does happen in practice.
-
NTP synchronization can be only as good as the network delay, so its accuracy is limited when you’re on a congested network with variable packet delays. One experiment showed that a minimum error of 35 ms is achievable when synchronizing over the internet, though occasional spikes in network delay can lead to errors of around a second. Depending on the configuration, large network delays can cause the NTP client to give up entirely.
-
Some NTP servers are wrong or misconfigured, reporting time that is off by hours. NTP clients mitigate such errors by querying several servers and ignoring outliers. Nevertheless, it’s somewhat worrying to bet the correctness of your systems on the time that you were told by a stranger on the internet.
-
Leap seconds result in a minute that is 59 seconds or 61 seconds long, which messes up timing assumptions in systems that are not designed with leap seconds in mind. The fact that leap seconds have crashed many large systems shows how easy it is for incorrect assumptions about clocks to sneak into a system. The best way of handling leap seconds may be to make NTP servers “lie,” by performing the leap second adjustment gradually over the course of a day (this is known as smearing), although actual NTP server behavior varies in practice. Leap seconds will no longer be used from 2035 on, so this problem will fortunately go away.
-
In VMs, the hardware clock is virtualized, which raises additional challenges for applications that need accurate timekeeping. When a CPU core is shared among VMs, each VM is paused for tens of milliseconds while another VM is running. From an application’s point of view, this pause manifests itself as the clock suddenly jumping forward when the VM continues running. An NTP client running inside the VM does not know when a pause occurs, so it may report the clock accuracy incorrectly.
-
If you run software on devices that you don’t fully control (e.g., mobile or embedded devices), you probably cannot trust their hardware clocks at all. Some users deliberately set their device’s hardware clock to an incorrect date and time, for example, to cheat in games. As a result, the clock might be set to a wildly inaccurate time.
Achieving very good clock accuracy is possible if you care about it sufficiently to invest significant resources. For example, the MiFID II European regulation for financial institutions requires all high-frequency trading funds to synchronize their clocks to within 100 microseconds of UTC, in order to help debug market anomalies such as “flash crashes” and to help detect market manipulation.
Such accuracy can be achieved with special hardware (GPS receivers and/or atomic clocks), the Precision Time Protocol (PTP), and careful deployment and monitoring. Relying on GPS alone can be risky because GPS signals can easily be jammed. In some locations (e.g., close to military facilities), this happens frequently. Some cloud providers have begun offering high-accuracy clock synchronization for their virtual machines, but clock synchronization still requires a lot of care. If your NTP daemon is misconfigured or a firewall is blocking NTP traffic, the clock error due to drift can quickly become large.