Data races and the limits of ThreadSanitizer in C and Go

Make informed decisions on software infrastructure.

Data races and the limits of ThreadSanitizer in C and Go

Have you ever wondered what a data race is? How a race detector works? If there are bugs in your race detector? We take a look.

By Phil EatonSeptember 6, 2026Artwork by Andrea Chersia Focus

You are getting early access to this article as a subscriber. Your support makes articles like this possible. Thank you.

To gain confidence in code we write tests. But when you add concurrency to the mix the bugs become nondeterministic. To gain additional confidence in concurrent code we might enable a race detector that tells us that a race happened in some piece of code during a run. This is useful because our tests might still have passed even though a race happened. The existence of the race, whether our tests otherwise fail or not, means that a latent bug absolutely exists.

Most major language implementations that have any race detector (Clang, GCC, Go, Swift, OCaml) use LLVM’s ThreadSanitizer. ThreadSanitizer (TSan) is not well documented. It has gone through three major iterations and while you can find the algorithm for TSan version two (released in 2012), the author of TSan suggests we just read the source to understand version three (released in 2021). Perhaps someone will contribute new docs.

There is a rich history of algorithms for detecting data races like Eraser (which influenced TSan version one), FastTrack (has ideas in common with TSan versions two and three), RaceTrack by Microsoft, and so on. Each algorithm has its own limitations, as TSan does too. And Clang, GCC, and Go do emit generic entry points for race detector libraries, even if TSan is the only one in serious use today.

In this article we’ll walk through the basics of data races. Then we’ll implement an idealized interpreter in Python for multi-threaded C code alongside FastTrack-style vector clocks to show how TSan roughly works. Then we’ll show how architecture choices make it possible to overload TSan in a few dimensions, causing it to miss obvious data races. For example, TSan cannot reliably report data races while crossing a 255 total thread boundary. This is not a particularly rare situation, considering web services implemented in a language like Go with one goroutine per request.

Some of these issues were discussed in Chapter 6 of Farzam Dorostkar’s 2025 PhD thesis, “Identifying and mitigating implementation-induced data race detection blind spots in ThreadSanitizer v3”. The thesis is partly in French, but Chapter 6 is in English.

There are also a few Go-specific scenarios that go undetected by TSan. We'll cover a design decision in sync.Pool that hides any race between two goroutines whose pooled objects, even in unrelated pools, hash to the same slot.

All of this is not to say TSan is a bad tool or that the authors should have done a better job. I would not be happy without TSan access. Nonetheless, it’s good to know its limitations.

Let’s dig in!

Background#

The C11 standard says, "[the] execution of a program contains a data race if it contains two conflicting actions in different threads, at least one of which is not atomic, and neither happens before the other. Any such data race results in undefined behavior." Go says, “[a] data race is defined as a write to a memory location happening concurrently with another read or write to that same location, unless all the accesses involved are atomic data accesses as provided by the sync/atomic package.” A data race is also, at least in theory, something we can automatically detect without additional guidance.

Grab GCC and Go.

$ sudo apt-get install -y gcc golang-go

In this example, two threads both read a counter, do some logic, and update the counter.

#include <pthread.h>
#include <stdlib.h>

long counter;

void* bump(void* n) {
  for (long i = 0; i < *(long*)n; i++) {
    long v = counter;
    // some logic
    counter = v + 1;
  }

  return 0;
}

int main(void) {
  pthread_t p, q;
  long n = atol(getenv("N") ?: "100");
  pthread_create(&p, 0, bump, &n);
  pthread_create(&q, 0, bump, &n);
  pthread_join(p, 0);
  pthread_join(q, 0);
  return (counter == n * 2) ? 0 : 1;
}

counter_data_race.c

In “normal” conditions we might not notice the race. We might have this run in CI and it would appear never to fail.

$ gcc counter_data_race.c -o counter_data_race
$ ./counter_data_race; echo $?
0
$ ./counter_data_race; echo $?
0
$ ./counter_data_race; echo $?
0
$ ./counter_data_race; echo $?
0

If we “stress” the code we might spot it.

$ N=1000 ./counter_data_race; echo $?
0
$ N=10000 ./counter_data_race; echo $?
1

But we have TSan! Just rebuild with -fsanitize=thread (and -g for useful line numbers).

$ gcc -fsanitize=thread -g counter_data_race.c -o counter_data_race
$ ./counter_data_race; echo $?
==================
WARNING: ThreadSanitizer: data race (pid=704862)
  Read of size 8 at 0x56110c2ce018 by thread T2:
    #0 bump /root/counter_data_race.c:8 (counter_data_race+0x129e) (BuildId: bdadc0a05776410367f6f1ee7400694f7a3267ab)

  Previous write of size 8 at 0x56110c2ce018 by thread T1:
    #0 bump /root/counter_data_race.c:10 (counter_data_race+0x12c0) (BuildId: bdadc0a05776410367f6f1ee7400694f7a3267ab)

  Location is global 'counter' of size 8 at 0x56110c2ce018 (counter_data_race+0x4018)

  Thread T2 (tid=704865, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/counter_data_race.c:20 (counter_data_race+0x1394) (BuildId: bdadc0a05776410367f6f1ee7400694f7a3267ab)

  Thread T1 (tid=704864, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/counter_data_race.c:19 (counter_data_race+0x1375) (BuildId: bdadc0a05776410367f6f1ee7400694f7a3267ab)

SUMMARY: ThreadSanitizer: data race /root/counter_data_race.c:8 in bump
==================
ThreadSanitizer: reported 1 warnings
66

Which is pretty great.

Let’s try out the above example ported to Go.

package main

import (
	"os"
	"strconv"
	"sync"
)

var counter int64

func bump(n int64, wg *sync.WaitGroup) {
	defer wg.Done()
	for i := int64(0); i < n; i++ {
		v := counter
		// some logic
		counter = v + 1
	}
}

func main() {
	n := int64(100)
	if s := os.Getenv("N"); s != "" {
		if v, err := strconv.ParseInt(s, 10, 64); err == nil {
			n = v
		}
	}
	var wg sync.WaitGroup
	wg.Add(2)
	go bump(n, &wg)
	go bump(n, &wg)
	wg.Wait()
	if counter != n*2 {
		os.Exit(1)
	}
}

counter_data_race.go

Go behaves similarly. When it does fail, it looks like a flaky test.

$ go build -o counter_data_race_go counter_data_race.go
$ ./counter_data_race_go; echo $?
0
$ ./counter_data_race_go; echo $?
0
$ N=100 ./counter_data_race_go ; echo $?
0
$ N=100 ./counter_data_race_go ; echo $?
0
$ N=10000 ./counter_data_race_go ; echo $?
1
$ N=10000 ./counter_data_race_go ; echo $?
0
$ N=10000 ./counter_data_race_go ; echo $?
0

But we’ll catch the race immediately when we enable TSan with -race.

$ go build -race -o counter_data_race_go counter_data_race.go
$ ./counter_data_race_go; echo $?
==================
WARNING: DATA RACE
Read at 0x0000005bda00 by goroutine 8:
  main.bump()
      /root/counter_data_race.go:14 +0x8d
  main.main.gowrap2()
      /root/counter_data_race.go:28 +0x44

Previous write at 0x0000005bda00 by goroutine 7:
  main.bump()
      /root/counter_data_race.go:14 +0xa5
  main.main.gowrap1()
      /root/counter_data_race.go:27 +0x44

Goroutine 8 (running) created at:
  main.main()
      /root/counter_data_race.go:28 +0x224

Goroutine 7 (finished) created at:
  main.main()
      /root/counter_data_race.go:27 +0x17c
==================
Found 1 data race(s)
66

Interestingly the race detector even catches this race when we limit Go to only using a single thread.

$ GOMAXPROCS=1 ./counter_data_race_go
==================
WARNING: DATA RACE
Read at 0x0000005bda00 by goroutine 7:
  main.bump()
      /root/counter_data_race.go:14 +0x95
  main.main.gowrap1()
      /root/counter_data_race.go:29 +0x44

Previous write at 0x0000005bda00 by goroutine 8:
  main.bump()
      /root/counter_data_race.go:16 +0xad
  main.main.gowrap2()
      /root/counter_data_race.go:30 +0x44

Goroutine 7 (running) created at:
  main.main()
      /root/counter_data_race.go:29 +0x116

Goroutine 8 (finished) created at:
  main.main()
      /root/counter_data_race.go:30 +0x19c
==================
Found 1 data race(s)

But this is because TSan provides hooks for whatever thread implementation you want. Go must tell TSan a new thread is created each time it creates a goroutine. As another example, QEMU has a coroutine implementation with TSan support. TSan comes with builtin support for pthreads.

We can of course fix the data race incorrectly, in a way no race detector can spot, turning the data race into a general race.

#include <pthread.h>
#include <stdlib.h>

long counter;
pthread_mutex_t m =  PTHREAD_MUTEX_INITIALIZER;

void* bump(void* n) {
  for (long i = 0; i < *(long*)n; i++) {
    pthread_mutex_lock(&m);
    long v = counter;
    pthread_mutex_unlock(&m);
    // some logic
    pthread_mutex_lock(&m);
    counter = v + 1;
    pthread_mutex_unlock(&m);
  }

  return 0;
}

int main(void) {
  pthread_t p, q;
  long n = atol(getenv("N") ?: "100");
  pthread_create(&p, 0, bump, &n);
  pthread_create(&q, 0, bump, &n);
  pthread_join(p, 0);
  pthread_join(q, 0);
  return (counter == n * 2) ? 0 : 1;
}

counter_general_race.c

And the Go version.

package main

import (
        "os"
        "strconv"
        "sync"
)

var counter int64
var m sync.Mutex

func bump(n int64, wg *sync.WaitGroup) {
        defer wg.Done()
        for i := int64(0); i < n; i++ {
                m.Lock()
                v := counter
                m.Unlock()
                // some logic
                m.Lock()
                counter = v + 1
                m.Unlock()
        }
}

func main() {
        n := int64(100)
        if s := os.Getenv("N"); s != "" {
                if v, err := strconv.ParseInt(s, 10, 64); err == nil {
                        n = v
                }
        }
        var wg sync.WaitGroup
        wg.Add(2)
        go bump(n, &wg)
        go bump(n, &wg)
        wg.Wait()
        if counter != n*2 {
                os.Exit(1)
        }
}

counter_general_race.go

Build and run both and we’ll still see the bug (periodically), but the race detector no longer tells us about it.

$ go build -race -o counter_general_race_go counter_general_race.go
$ gcc -fsanitize=thread -g counter_general_race.c -o counter_general_race
$ ./counter_general_race_go; echo $?
1
$ ./counter_general_race_go; echo $?
0
$ ./counter_general_race; echo $?
0
$ ./counter_general_race; echo $?
1

And while we must worry about programs like this, the focus of this article is on data races.

But a single concurrent read and write is enough, we don't even need loops and counters. Take this program for example.

#include <pthread.h>

int x, seen, failed;

void *t1(void *arg) {
    x = 1;
    return NULL;
}

void *t2(void *arg) {
    seen = x;
    return NULL;
}

int main(void) {
    pthread_t a, b;
    pthread_create(&a, NULL, t1, NULL);
    pthread_create(&b, NULL, t2, NULL);
    pthread_join(a, NULL);
    pthread_join(b, NULL);
    failed = seen != 1; // programmer intended that t2 observed t1's write
    return failed; // exit 0 when it did
}

no_sync.c

We’ll build an interpreter in Python to model this program. And then we’ll build a FastTrack- / TSan-style race detector to catch the race.

Keep in mind, we are not building an accurate model of C! We’re implementing an idealized subset of C.

Let’s dig in.

Interpreting an idealized subset of C#

Our interpreter won’t care about variable declarations or top-level statements at all. It will parse functions and statements inside functions, reading a C program from stdin.

We’ll do very little error handling at all.

import re
import sys


def parse_statement(code):
  t = re.findall(r"\w+|[=!]=|\S", code)
  if t[0] == "pthread_mutex_lock":  # pthread_mutex_lock ( & MUTEX ) ;
    return "mutex_lock", (t[3],)
  if t[0] == "pthread_mutex_unlock":  # pthread_mutex_unlock ( & MUTEX ) ;
    return "mutex_unlock", (t[3],)
  if t[0] == "pthread_create":  # pthread_create ( & HANDLE , _ , ENTRY , _ ) ;
    return "thread_create", (t[3], t[7])
  if t[0] == "pthread_join":  # pthread_join ( HANDLE , _ ) ;
    return "thread_join", (t[2],)
  if t[0] == "return" and len(t) <= 3:  # return NAME ;   or   return ;
    return "return_", (t[1] if len(t) == 3 else "",)
  if len(t) == 4 and t[1] == "=":  # NAME = NUMBER ;   or   NAME = NAME ;
    return "write", (t[0], t[2])
  if len(t) == 6 and t[1] == "=" and t[3] in ("==", "!="):
    return "compare", (t[0], t[2], t[3], t[4])  # NAME = NAME OP NAME ;
  if all(word.isidentifier() or word == "," for word in t[:-1]):
    return None  # TYPE NAME , NAME ;  a declaration does not run
  raise SyntaxError(code)


def parse(source):
  functions = {}
  function_name = None
  for lineno, line in enumerate(source.splitlines(), 1):
    code, _, _ = line.partition("//")
    code = code.strip()
    if not code:
      continue
    named = re.match(r"\w+\s+\*?(\w+)\([^)]*\)\s*\{", code)
    if named:
      function_name = named.group(1)
      functions[function_name] = []
    elif code == "}" and function_name is not None:
      body = functions[function_name]
      if not body or body[-1][0] != "return_":
        raise SyntaxError(f"{function_name} does not end with a return")
      function_name = None
    elif function_name is not None:
      parsed = parse_statement(code)
      if parsed:
        functions[function_name].append((*parsed, code, lineno))
  return functions

interpreter.py

Now for the interpreter. We’ll set it up with state for managing threads and locks, function bodies, and variable values.

class Interpreter:
  def __init__(self, functions):
    self.functions = functions
    self.memory = {}
    self.mutex_thread_holders = {}
    self.threads = {"main": list(functions.get("main", []))}
    self.finished_threads = set()
    self.exit_code = 0

interpreter.py

Next we’ll implement writing a value into a variable (we’ll map by variable names not by addresses), comparing two values, and returning from a function.

def write(self, thread, dst, source):
    self.memory[dst] = self._read(source)

  def _read(self, name):
    if name.isdigit():
      return int(name)
    if name in ("", "NULL"):
      return 0
    return self.memory.get(name, 0)

  def compare(self, thread, dst, left, operator, right):
    same = self._read(left) == self._read(right)
    self.memory[dst] = int(same if operator == "==" else not same)

  def return_(self, thread, expression):
    self.finished_threads.add(thread)
    if thread == "main":
      self.exit_code = self._read(expression)
    self.threads.pop(thread, None)

interpreter.py

Then we’ll implement support for creating a thread (which marks that thread as live, schedulable) and joining a thread (blocking until the thread exits). When an instruction raises the Blocked exception, the instruction will keep being retried until it succeeds.

class Blocked(Exception):
    pass

  def thread_create(self, thread, handle, entrypoint):
    self.threads[handle] = list(self.functions[entrypoint])

  def thread_join(self, thread, handle):
    if handle in self.threads:
      raise self.Blocked

interpreter.py

Next for mutex locking, we’ll block if another thread has registered the mutex, otherwise we register it ourselves. Unlocking will remove the current thread as the registered current owner of the mutex.

def mutex_lock(self, thread, mutex):
    if self.mutex_thread_holders.get(mutex) not in (None, thread):
      raise self.Blocked # another thread is holding it
    self.mutex_thread_holders[mutex] = thread

  def mutex_unlock(self, thread, mutex):
    if self.mutex_thread_holders.get(mutex) != thread:
      raise AssertionError(f"{thread} unlocks {mutex} without holding it")
    del self.mutex_thread_holders[mutex]

interpreter.py

Then we'll add a step function and a run function, to keep running all threads until every live thread has returned.

def step(self, thread, statement):
    operation, arguments, *_ = statement
    handler = getattr(self, operation, None)
    if handler is None:
      raise NameError(f"{operation} is not implemented")
    return handler(thread, *arguments)

  def run(self):
    while self.threads:
      # Schedules the first thread not currently Blocked.
      for thread, queue in list(self.threads.items()):
        try:
          self.step(thread, queue[0])
        except self.Blocked:
          continue
        queue.pop(0)
        break

interpreter.py

Lastly, tie together parsing and running.

if __name__ == "__main__":
  interpreter = Interpreter(parse(sys.stdin.read()))
  interpreter.run()
  sys.exit(interpreter.exit_code)

interpreter.py

With the interpreter done, recall our minimal example program.

#include <pthread.h>

int x, seen, failed;

void *t1(void *arg) {
    x = 1;
    return NULL;
}

void *t2(void *arg) {
    seen = x;
    return NULL;
}

int main(void) {
    pthread_t a, b;
    pthread_create(&a, NULL, t1, NULL);
    pthread_create(&b, NULL, t2, NULL);
    pthread_join(a, NULL);
    pthread_join(b, NULL);
    failed = seen != 1; // programmer intended that t2 observed t1's write
    return failed; // exit 0 when it did
}

no_sync.c

Let’s run it with our interpreter.

$ python3 interpreter.py < no_sync.c; echo $?
0

And with GCC.

$ gcc no_sync.c
$ ./a.out; echo $?
0

And now with TSan.

$ gcc -g -fsanitize=thread no_sync.c
$ ./a.out; echo $?
==================
WARNING: ThreadSanitizer: data race (pid=1041294)
  Read of size 4 at 0x556f51049014 by thread T2:
    #0 t2 /root/no_sync.c:11 (a.out+0x12bd) (BuildId: a50e60b6ecef90977bab7057915cbe47fdf5aecc)

  Previous write of size 4 at 0x556f51049014 by thread T1:
    #0 t1 /root/no_sync.c:6 (a.out+0x1274) (BuildId: a50e60b6ecef90977bab7057915cbe47fdf5aecc)

  Location is global 'x' of size 4 at 0x556f51049014 (a.out+0x4014)

  Thread T2 (tid=1041297, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/no_sync.c:18 (a.out+0x134d) (BuildId: a50e60b6ecef90977bab7057915cbe47fdf5aecc)

  Thread T1 (tid=1041296, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/no_sync.c:17 (a.out+0x1330) (BuildId: a50e60b6ecef90977bab7057915cbe47fdf5aecc)

SUMMARY: ThreadSanitizer: data race /root/no_sync.c:11 in t2
==================
ThreadSanitizer: reported 1 warnings
66

Let’s catch this race ourselves! But first let’s talk about how GCC (and Clang and Go and everyone) instruments code for TSan. And then we’ll instrument our own interpreter too.

Instrumentation#

Under the hood, -fsanitize=thread does two things. First, it rewrites your code, including that memory accesses get additional calls to __tsan_read* or __tsan_write* functions, and function bodies get additional calls to __tsan_func_entry and __tsan_func_exit at their beginning and end. (Go does similar rewriting but has its own API.) Second, it links against libtsan which provides those symbols.

The compiler does not do anything about libraries like pthreads. TSan itself handles wrapping calls to pthreads synchronization methods. If you use a threading library other than pthreads you may have to teach TSan about it. pthreads was popular enough for TSan to come with builtin support.

If we compile with -fsanitize=thread and don’t link with libtsan, we can instead link against our own runtime implementing TSan symbols and run our own race detection algorithm.

It will be simpler in our implementation where we can just write a new entrypoint that intercepts interpreter calls and hooks into our race detector. So let’s talk about the race detector itself.

Eraser#

Eraser is a historic algorithm for race detection based on observing held locks. With Eraser, every address in memory moves through one of four states: Virgin (hasn’t been accessed yet), Exclusive (accessed by only a single thread so far), Shared (read by more than one thread, written by only one thread), and SharedModified (when a new thread writes an Exclusive address or any thread writes a Shared address).

When an accessed address is Shared or SharedModified, you note all locks currently held by the thread, storing a running intersection of locks historically held every time the address is accessed. If the address is SharedModified and at some point the intersection is empty, you report a data race.

read/write from first thread

Shared-Modified
note + report

stateDiagram-v2
    SharedModified: Shared-Modified<br/>note + report
    Shared: Shared<br/>note only

    [*] --> Virgin
    Virgin --> Exclusive: write
    Exclusive --> Exclusive: read/write from first thread
    Exclusive --> Shared: read from new thread
    Exclusive --> SharedModified: write from new thread
    Shared --> Shared: read
    Shared --> SharedModified: write

The basic Eraser algorithm is heavily prone to false positives. The FastTrack (vector clocks for race detection) paper for example notes that it is not aware of synchronization idioms like barrier synchronization or thread join. And it cannot catch our no_sync.c race that uses no locks at all.

TSan v1 was based on Eraser-style locksets plus vector clocks to reduce false positives. TSan v2 dropped the lockset detection entirely. TSan v2 and v3 are not FastTrack, though they are very similar. So we’re going to start with the idealized FastTrack idea and later we’ll talk about TSan’s specific differences.

FastTrack and vector clocks#

In FastTrack, each thread has a monotonic time counter (its clock). Each thread also stores a vector clock containing the time it is aware of for every other thread. And there is a global mapping of addresses to which thread last read this address and when, and to which thread last wrote this address and when, both according to those threads’ own clocks.

import re
import sys

class FastTrackRaceDetector:
  def __init__(self):
    self.clocks = {"main": {"main": 1}}
    self.primitive_last_published = {}
    self.last_write = {}
    self.last_read = {}
    self.line = 0
    self.source = []
    self.races = 0

detector.py

Synchronization primitives store the vector clock of the thread that last finished with the primitive (e.g. unlocking a mutex, exiting a thread). You also increment the thread’s clock at this point.

def release(self, thread, obj):
    self.primitive_last_published[obj] = dict(self.clocks[thread])
    self.clocks[thread][thread] += 1

detector.py

When you engage a primitive (e.g. locking a mutex, joining a thread) you take the maximum of each time in your vector clock and the primitive’s vector clock. This is how ordering travels between threads.

def acquire(self, thread, obj):
    for author, tick in self.primitive_last_published.get(obj, {}).items():
      if tick > self.clocks[thread].get(author, 0):
        self.clocks[thread][author] = tick  # learn what the publisher knew

detector.py

A new thread inherits its parent’s vector clock, and both parent and child tick here as well.

def start(self, parent, child):
    self.clocks[child] = dict(self.clocks[parent])
    self.clocks[child][child] = self.clocks[child].get(child, 0) + 1
    self.clocks[parent][parent] += 1

detector.py

When you access memory, you check the global mapping for the last write to that address: thread B, time B. If your view of thread B's time is lower than time B, you never learned about that write, and you found a data race. If you are writing at that address, you also check the global mapping for the last read: thread A, time A.

def access(self, thread, var, is_write):
    time = self.clocks[thread][thread]
    against = [self._compare(thread, self.last_write.get(var), "write")]
    if is_write:
      against.append(self._compare(thread, self.last_read.get(var), "read"))
      self.last_write[var] = (thread, time, self.line)
    else:
      self.last_read[var] = (thread, time, self.line)
    kind = "write" if is_write else "read"
    for earlier in against:
      if earlier:
        print(f"data race on {var}")
        print(self._quote(kind, self.line, thread, var))
        print(self._quote(*earlier, var, previous=True))

  def _compare(self, thread, recorded, kind):
    if recorded is None or recorded[0] == thread:
      return None
    author, time, line = recorded
    if self.clocks[thread].get(author, 0) >= time:
      return None
    self.races += 1
    return kind, line, author

  def _quote(self, kind, line, thread, var, previous=False):
    kind = f"previous {kind}" if previous else kind
    text = self.source[line - 1] if line <= len(self.source) else ""
    found = re.search(rf"\b{var}\b", text)
    column = found.start() if found else 0
    return (
      f"  {kind} at line {line} by {thread}\n"
      f"  {line:>4} | {text}\n"
      f"  {'':>4} | {' ' * column}^"
    )

detector.py

This is not a complete description of FastTrack, but it gets us to our goal.

Lastly, we need to intercept our interpreter and hook into these race detector methods at each point we discussed.

def link(interp, detector):
  original_write, original_compare = interp.write, interp.compare
  original_step = interp.step

  def write(thread, dst, source):
    original_write(thread, dst, source)
    if source.isidentifier():
      detector.access(thread, source, is_write=False)
    detector.access(thread, dst, is_write=True)

  def compare(thread, dst, left, operator, right):
    original_compare(thread, dst, left, operator, right)
    for name in (left, right):
      if name.isidentifier():
        detector.access(thread, name, is_write=False)
    detector.access(thread, dst, is_write=True)

  def step(thread, statement):
    operation, arguments, _text, line = statement
    detector.line = line
    original_step(thread, statement)
    if operation == "mutex_lock" or operation == "thread_join":
      detector.acquire(thread, arguments[0])
    elif operation == "mutex_unlock":
      detector.release(thread, arguments[0])
    elif operation == "thread_create":
      detector.start(thread, arguments[0])
    elif operation == "return_":
      detector.release(thread, thread)

  interp.write = write
  interp.compare = compare
  interp.step = step
  return interp

detector.py

And give ourselves a new entrypoint for the detector.

if __name__ == "__main__":
  from interpreter import Interpreter, parse
  program = sys.stdin.read()
  functions = parse(program)
  detector = FastTrackRaceDetector()
  detector.source = program.splitlines()
  interp = link(Interpreter(functions), detector)
  interp.run()
  sys.exit(66 if detector.races else 0)

detector.py

Now try it out against no_sync.c!

$ python3 detector.py < no_sync.c
data race on x
  read at line 11 by b
    11 |     seen = x;
       |            ^
  previous write at line 6 by a
     6 |     x = 1;
       |     ^

That’s pretty cool.

Where TSan falls over#

So first off, TSan over-orders. The Eraser paper was clear about this when it came out in 1997. Its Figure 2 described essentially the following program.

#include <pthread.h>

int x, failed;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

void *first(void *arg) {
  x = 1;
  pthread_mutex_lock(&m);
  pthread_mutex_unlock(&m);
  return NULL;
}

void *second(void *arg) {
  pthread_mutex_lock(&m);
  pthread_mutex_unlock(&m);
  x = 2;
  return NULL;
}

int main(void) {
  pthread_t t1, t2;
  pthread_create(&t1, NULL, first, NULL);
  pthread_create(&t2, NULL, second, NULL);
  pthread_join(t1, NULL);
  pthread_join(t2, NULL);
  failed = x != 2;
  return failed;
}

figure2.c

Run our detector against it and it will not crash.

$ python3 detector.py < figure2.c; echo $?
0

And likewise, TSan will (most likely) not crash.

$ gcc -g -fsanitize=thread figure2.c -o ./figure2
$ ./figure2; echo $?
0

Run it 1,000 times and TSan will occasionally catch it.

$ n=0; for i in $(seq 1 1000); do TSAN_OPTIONS=abort_on_error=0:exitcode=0 ./figure2 2>&1 | grep -q 'data race' && n=$((n+1)); printf '\r%d/%d' "$n" "$i"; done; echo
14/1000

And yes it’s good that occasionally TSan will happen to catch this, but also the vast majority of the time it won’t. While this is a bit of an edge case, it’s pretty simple code that is not impossible to imagine existing after a refactor.

But also just because Eraser called out this example and can handle it itself doesn’t mean TSan is a worse algorithm. In general TSan is much more useful than an Eraser-based detector would be. This is just an example of a race TSan cannot (easily) see.

Let’s move on to an even bigger tradeoff TSan has made.

Budgets#

Unlike our idealized race detector, TSan has some strict goals about maximum impact on your workload. In order to do that it has to limit its own resources. And there are three dimensions we’ll take a look at and show we can overload: 1) how many active threads TSan can track, 2) how many synchronization releases it can track, and 3) how many accesses it remembers per 8-byte granule of memory. We’ll show TSan miss scenarios in each case, in both C and Go. And we’ll walk through a Go-specific missed races with sync.Pool.

Too many threads#

TSan tracks a total of 255 threads before wrapping around and reusing that space to track new threads. If you have a long-running writer thread (say, a thread that reads from a configuration file) that updates a shared field without a mutex, enough threads created to get you past 253 (past the main thread and the writer thread), and then a new thread comes along that reads from the shared field, TSan cannot tell you about it.

In C.

#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>

int x;

static void *longlivedwriter(void *a) {
    x = 1;
    pause(); // stays alive, holding its slot
    return 0;
}

static void *worker(void *a) {
    // does some work
    return 0;
}

int main(int c, char **v) {
    pthread_t t, h;
    pthread_create(&t, 0, longlivedwriter, 0);
    pthread_detach(t); // detach, not join, a join would order it
    usleep(50000); // let the write land
    for (long i = 0; i < (c > 1 ? atol(v[1]) : 253); i++) {
        pthread_create(&h, 0, worker, 0);
        pthread_join(h, 0);
    }
    int sink = x; // races with the write above, -O0 avoid eliding
    return 0;
}

toomanythreads.c

By default it will only use up 254 threads and TSan will notice it.

$ gcc -g -fsanitize=thread toomanythreads.c
$ ./a.out
==================
WARNING: ThreadSanitizer: data race (pid=969888)
  Read of size 4 at 0x55d81b01e014 by main thread:
    #0 main /root/toomanythreads.c:27 (a.out+0x141d) (BuildId: 2423602ee49a18a91cdc9046647c6f2da6b8c9e2)

  Previous write of size 4 at 0x55d81b01e014 by thread T1:
    #0 longlivedwriter /root/toomanythreads.c:8 (a.out+0x12f4) (BuildId: 2423602ee49a18a91cdc9046647c6f2da6b8c9e2)

  As if synchronized via sleep:
    #0 usleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:390 (libtsan.so.2+0x587d1) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanythreads.c:22 (a.out+0x1397) (BuildId: 2423602ee49a18a91cdc9046647c6f2da6b8c9e2)

  Location is global 'x' of size 4 at 0x55d81b01e014 (a.out+0x4014)

  Thread T1 (tid=969890, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanythreads.c:20 (a.out+0x1375) (BuildId: 2423602ee49a18a91cdc9046647c6f2da6b8c9e2)

SUMMARY: ThreadSanitizer: data race /root/toomanythreads.c:27 in main
==================
ThreadSanitizer: reported 1 warnings
66

But if you ask it to create one more thread (pass in 254), it will end up wrapping TSan’s thread tracker and TSan will no longer catch the race.

$ gcc -fsanitize=thread toomanythreads.c
$ ./a.out 254; echo $?
0

Go does still use TSan, but Go’s runtime itself seems to use up an additional slot so the wrap happens 1 goroutine sooner.

package main

import (
	"os"
	"strconv"
	"time"
)

var X, Sink int

func longlivedwriter(forever chan struct{}) {
	X = 1
	<-forever // stays alive, holding its slot
}

func worker(done chan struct{}) {
	// does some work
	close(done)
}

func main() {
	n := 252
	if len(os.Args) > 1 {
		n, _ = strconv.Atoi(os.Args[1])
	}

	go longlivedwriter(make(chan struct{}))

	time.Sleep(50 * time.Millisecond) // let the write land

	for i := 0; i < n; i++ {
		done := make(chan struct{})
		go worker(done)
		<-done // finished before the next is created
	}

	Sink = X // races with the write above
}

toomanythreads.go

Give it a try.

$ go run -race toomanythreads.go
==================
WARNING: DATA RACE
Read at 0x0000005baa00 by main goroutine:
  main.main()
      /root/toomanythreads.go:37 +0x1d4

Previous write at 0x0000005baa00 by goroutine 7:
  main.longlivedwriter()
      /root/toomanythreads.go:12 +0x35
  main.main.gowrap1()
      /root/toomanythreads.go:27 +0x17

Goroutine 7 (running) created at:
  main.main()
      /root/toomanythreads.go:27 +0x105
==================
Found 1 data race(s)
exit status 66
1

But if you take up one more goroutine, TSan will again miss the race.

$ go run -race toomanythreads.go 253; echo $?
0

Next up, overloading synchronization releases!

Synchronization releases#

TSan tracks up to 255 threads (with one reserved). But the counter in each thread’s slots is only 14 bits. When the counter increments enough times, TSan is forced to move on to another thread slot to keep accounting. And a single thread that is, for example, unlocking the same mutex enough times can use up the entire thread slot-counter budget and miss a race.

In C.

#include <pthread.h>
#include <unistd.h>

int x;

static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

static void *longlivedwriter(void *a) {
    x = 1;
    pause(); // stay alive
    return 0;
}

int main(int c, char **v) {
    long n = (c > 1 ? 4200000 : 4100000);
    pthread_t t;
    pthread_create(&t, 0, longlivedwriter, 0);
    pthread_detach(t); // detach, not join
    usleep(50000); // let the write land
    for (long i = 0; i < n; i++) { // spend the budget
        pthread_mutex_lock(&m);
        pthread_mutex_unlock(&m);
    }
    int sink = x; // races with the writer, must be -O0 not to get elided
    return 0;
}

toomanyreleases.c

Give it a try at just under the budget and watch TSan catch it.

$ gcc -g -O0 -fsanitize=thread toomanyreleases.c 
$ ./a.out
==================
WARNING: ThreadSanitizer: data race (pid=970953)
  Read of size 4 at 0x562c8903b040 by main thread:
    #0 main /root/toomanyreleases.c:24 (a.out+0x13e0) (BuildId: 1ce81d8296c5cf89397de829823493d91b66f1cc)

  Previous write of size 4 at 0x562c8903b040 by thread T1:
    #0 longlivedwriter /root/toomanyreleases.c:9 (a.out+0x12f4) (BuildId: 1ce81d8296c5cf89397de829823493d91b66f1cc)

  As if synchronized via sleep:
    #0 usleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:390 (libtsan.so.2+0x587d1) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanyreleases.c:19 (a.out+0x139a) (BuildId: 1ce81d8296c5cf89397de829823493d91b66f1cc)

  Location is global 'x' of size 4 at 0x562c8903b040 (a.out+0x4040)

  Thread T1 (tid=970955, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanyreleases.c:17 (a.out+0x1378) (BuildId: 1ce81d8296c5cf89397de829823493d91b66f1cc)

SUMMARY: ThreadSanitizer: data race /root/toomanyreleases.c:24 in main
==================
ThreadSanitizer: reported 1 warnings

But if we bump the number of releases and use up the whole budget, TSan won’t report.

$ ./a.out bump; echo $?
0

Interestingly, Go’s mutexes are relatively more expensive. Every mutex lock/unlock pair takes up 3 releases in TSan, so we exhaust the budget 3x sooner than pthreads.

package main

import (
	"os"
	"strconv"
	"sync"
	"time"
)

var X, Sink int

func longlivedwriter(forever chan struct{}) {
	X = 1
	<-forever
}

func main() {
	n := 1_300_000
	if len(os.Args) > 1 {
		n, _ = strconv.Atoi(os.Args[1])
	}

	go longlivedwriter(make(chan struct{}))

	time.Sleep(50 * time.Millisecond) // let the write land

	var mu sync.Mutex
	for i := 0; i < n; i++ { // spend the process release budget
		mu.Lock()
		mu.Unlock()
	}

	Sink = X // races with the write above
}

toomanyreleases.go

Watch it catch the race at first.

$ go run -race toomanyreleases.go; echo $?
==================
WARNING: DATA RACE
Read at 0x0000005baa00 by main goroutine:
  main.main()
      /root/toomanyreleases.go:33 +0x170

Previous write at 0x0000005baa00 by goroutine 7:
  main.longlivedwriter()
      /root/toomanyreleases.go:13 +0x35
  main.main.gowrap1()
      /root/toomanyreleases.go:23 +0x17

Goroutine 7 (running) created at:
  main.main()
      /root/toomanyreleases.go:23 +0x105
==================
Found 1 data race(s)
exit status 66

But then not catch the race when we do more releases.

$ go run -race toomanyreleases.go 1400000; echo $?
0

The last budget we’ll focus on is how TSan tracks access history by address.

Access history#

TSan groups addresses into 8-byte granules, with a budget of four cells per granule to record accesses within the granule. A cell is claimed by each access that differs from what is already there in thread-slot or in which bytes it touched; repeats from the same thread to the same bytes reuse the cell they hold. Once four cells are claimed, the next distinct access evicts one, even when TSan's own overlap check has just proven that access cannot race with what it evicts.

#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>

_Alignas(8) char cell[8];

static void *longlivedwriter(void *a) {
    cell[0] = 1;
    pause(); // stays alive, holding its slot
    return 0;
}

static void *neighbours(void *a) {
    for (long i = 0; i < (long)a; i++)
        cell[1 + i] = i; // same 8 bytes, different byte: cannot race
    return 0;
}

int main(int c, char **v) {
    long n = c > 1 ? atol(v[1]) : 3;
    pthread_t t, f;
    pthread_create(&t, 0, longlivedwriter, 0);
    pthread_detach(t); // detach, not join
    usleep(50000); // let the write land
    pthread_create(&f, 0, neighbours, (void *)n);
    pthread_join(f, 0);
    int sink = cell[0]; // races with the write above, -O0 to avoid eliding
    return 0;
}

toomanycells.c

Trigger only 4 cell writes and TSan catches it.

$ gcc -g -O0 -fsanitize=thread toomanycells.c
$ ./a.out
==================
WARNING: ThreadSanitizer: data race (pid=971449)
  Read of size 1 at 0x5580c8875018 by main thread:
    #0 main /root/toomanycells.c:27 (a.out+0x1472) (BuildId: 5566f97956b9da04da0d67758e898915b3fec6e2)

  Previous write of size 1 at 0x5580c8875018 by thread T1:
    #0 longlivedwriter /root/toomanycells.c:8 (a.out+0x12f4) (BuildId: 5566f97956b9da04da0d67758e898915b3fec6e2)

  As if synchronized via sleep:
    #0 usleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:390 (libtsan.so.2+0x587d1) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanycells.c:24 (a.out+0x1427) (BuildId: 5566f97956b9da04da0d67758e898915b3fec6e2)

  Location is global 'cell' of size 8 at 0x5580c8875018 (a.out+0x4018)

  Thread T1 (tid=971451, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 main /root/toomanycells.c:22 (a.out+0x1405) (BuildId: 5566f97956b9da04da0d67758e898915b3fec6e2)

SUMMARY: ThreadSanitizer: data race /root/toomanycells.c:27 in main
==================
ThreadSanitizer: reported 1 warnings

Trigger a 5th write and TSan will miss the race.

And again in Go.

package main

import (
        "os"
        "strconv"
        "time"
)

var Cell struct {
        _ [0]int64 // 8-byte alignment: B is exactly one shadow granule
        B [8]byte
}
var Sink byte

func longlivedwriter(forever chan struct{}) {
        Cell.B[0] = 1
        <-forever // stays alive
}

func neighbours(n int, done chan struct{}) {
        for i := 0; i < n; i++ {
                Cell.B[1+i] = byte(i) // same 8 bytes, different byte: cannot race
        }
        close(done)
}

func main() {
        n := 3
        if len(os.Args) > 1 {
                n, _ = strconv.Atoi(os.Args[1])
        }

        go longlivedwriter(make(chan struct{}))

        time.Sleep(50 * time.Millisecond) // let the write land

        done := make(chan struct{})
        go neighbours(n, done)
        <-done

        Sink = Cell.B[0] // races with the write above
}

toomanycells.go

Run it with 4 accesses and it’s caught.

$ go run -race toomanycells.go ; echo $?
==================
WARNING: DATA RACE
Read at 0x0000005baa00 by main goroutine:
  main.main()
      /root/toomanycells.go:41 +0x1bd

Previous write at 0x0000005baa00 by goroutine 7:
  main.longlivedwriter()
      /root/toomanycells.go:16 +0x35
  main.main.gowrap1()
      /root/toomanycells.go:33 +0x17

Goroutine 7 (running) created at:
  main.main()
      /root/toomanycells.go:33 +0x105
==================
Found 1 data race(s)
exit status 66
1

And at 5 accesses it’s lost.

$ go run -race toomanycells.go 4; echo $?
0

There are a few more resources like this that you might be able to overload to miss races in TSan. And again the point of this isn’t to be adversarial here, none of these cases are that unlikely.

Let’s move on to one last category of missed races, with Go’s sync.Pool.

sync.Pool#

A thread-safe object pool exists so that multiple threads (goroutines) can reuse objects instead of allocating new ones. But reusing objects across threads looks like a data race. So the pool talks to the race detector directly, telling the race detector it has acquired and released some address, even with no lock.

It is not safe to tell the race detector you’re acquiring and releasing the pooled object’s own address because synchronization may be separately associated with that object (a mutex inside of the object) that the pool would mess with. So Go creates a hash of the object’s address and uses the result to pick one of 128 addresses it sets aside for use here, from sync/pool.go:

We don't use the actual pointer stored in x directly, for fear of conflicting with other synchronization on that address. Instead, we hash the pointer to get an index into poolRaceHash.

The problem is that any two objects, even in unrelated pools, can end up sharing the same hash and create ordering between goroutines where there is none, and any data race between these goroutines is unreported.

package main

import (
        "fmt"
        "os"
        "sync"
        "time"
        "unsafe"
)

var X, Sink int

type ob struct{ _ [32]byte }

// replicates the hash Go computes
func bucket(p *ob) uint32 {
        return uint32((uint64(uint32(uintptr(unsafe.Pointer(p))))*0x85ebca6b)>>16) % 128
}

func putter(p *sync.Pool, o *ob, wg *sync.WaitGroup) {
        defer wg.Done()
        X = 1
        for i := 0; i < 8; i++ {
                p.Put(o)
        }
}

func getter(p *sync.Pool, wg *sync.WaitGroup) {
        defer wg.Done()
        time.Sleep(20 * time.Millisecond) // let the Put land first
        if p.Get() == nil {
                fmt.Fprintln(os.Stderr, "skip")
                return
        }
        Sink = X // races with the write in putter
}

func main() {
        same := len(os.Args) > 1

        // search for an object that collides, or not
        var o1, o2 *ob
        for o2 == nil {
                c := &ob{}
                if o1 == nil {
                        o1 = c
                } else if (bucket(c) == bucket(o1)) == same {
                        o2 = c
                }
        }

        var p1, p2 sync.Pool // entirely unrelated pools
        for i := 0; i < 8; i++ {
                p2.Put(o2) // something for the getter to take
        }

        var wg sync.WaitGroup
        wg.Add(2)
        go putter(&p1, o1, &wg)
        go getter(&p2, &wg)
        wg.Wait()
}

poolrace.go

Give it a try, by default it will look for objects that don’t collide and the race will be reported.

$ go run -race poolrace.go
==================
WARNING: DATA RACE
Read at 0x00000060b9c0 by goroutine 8:
  main.getter()
      /root/poolrace.go:35 +0xb0
  main.main.gowrap2()
      /root/poolrace.go:60 +0x44

Previous write at 0x00000060b9c0 by goroutine 7:
  main.putter()
      /root/poolrace.go:22 +0x84
  main.main.gowrap1()
      /root/poolrace.go:59 +0x4f

Goroutine 8 (running) created at:
  main.main()
      /root/poolrace.go:60 +0x37c

Goroutine 7 (finished) created at:
  main.main()
      /root/poolrace.go:59 +0x2cb
==================
Found 1 data race(s)
exit status 66

But if we look for an object that collides, the race goes missing.

$ go run -race poolrace.go collide; echo $?
0

This 128-slot table is particularly small. Perhaps because this is a price that every binary pays, whether -race is on or not.

Parting thoughts#

Race detectors are incredibly useful. They do also have edge cases, and the edge cases are not really documented, at least not with TSan.

If you want extra credit, look for the missing race detection when using chan struct{}. And, again, there are a few more budget limits to explore in TSan itself. But this article got too long as it is.

Happy hunting.

Noticed a mistake? Have a question or comment? Write to the editor.