Golang's goroutine leak profiler

Go 1.27 adds a goroutineleak profile to runtime/pprof. It reports goroutines that are blocked and can never be unblocked, and it is cheap and precise enough to run against production services. Vlad Saioc's post on the Golang blog walks through using it, how it piggybacks on the garbage collector, and what it cannot see. The feature comes out of a research collaboration between Aarhus University, Washington University in St. Louis and Uber, published as "Dynamic Partial Deadlock Detection and Recovery via Garbage Collection" (Saioc et al., ASPLOS 2025) goroutine-leak-profiles.

The gap it fills

A leaked goroutine is one that is blocked on something whose unblocking condition will never be met. Leaks pile up as memory, both the goroutine's own and whatever it references, and as GC CPU time, which the post says gets worse when GOMEMLIMIT is set. go-channel-bug-patterns describes the usual shapes and notes that Golang's built-in deadlock detector only fires when every goroutine is blocked, so a handful of stuck goroutines in a running service goes unnoticed.

The existing tools all work at test time. goleak flags goroutines still alive when a test ends, and the synctest package from Go 1.25 lets a test control the ordering of concurrent events. Neither says anything about a production process doing things the tests never exercised. A plain goroutine profile does run in production, but it cannot tell a leak from many goroutines legitimately waiting during a traffic spike, and a leak of a few goroutines never stands out in it at all.

Using it

A service that already imports net/http/pprof gets the profile for free at /debug/pprof/goroutineleak. The post's running example is a worker fan-out that returns early on the first error:

func processWorkItems(ws []workItem) ([]workResult, error) {
    ch := make(chan result)
    for _, w := range ws {
        go func() {
            res, err := processWorkItem(w)
            ch <- result{res, err}
        }()
    }

    var results []workResult
    for range len(ws) {
        r := <-ch
        if r.err != nil {
            return nil, r.err // remaining senders block forever
        }
        results = append(results, r.res)
    }
    return results, nil
}

Collected with curl and opened in go tool pprof, the profile attributes every leaked goroutine to the line that blocks:

$ curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof
$ go tool pprof leak.prof
Type: goroutineleak
(pprof) list processWorkItems
Total: 116
ROUTINE ======================== main.processWorkItems.func1 in .../main.go
         0        116 (flat, cum)   100% of Total
         .        116     33:                   ch <- result{res, err}

The count grows the longer the program runs. The fix is a buffer of len(ws) on the channel, so every sender can complete after the receiver has gone.

How it detects a leak

The idea rests on reachability. If a goroutine is blocked on a channel or lock that no other goroutine holds a reference to, nothing can ever wake it. The post generalizes that into an inductive definition of liveness: a goroutine is live if it is not blocked on a concurrency primitive, or if at least one primitive blocking it is referenced by another live goroutine. Everything that is not live is leaked.

Computing that is a reachability problem, and the runtime already solves one on every GC cycle. Golang's collector is a concurrent tri-color mark-and-sweep (now in its Green Tea variant), and leak detection changes it in five steps:

1. A normal cycle treats every goroutine and every global as a mark root. The leak-detecting cycle uses only the unblocked goroutines as roots, because those are live by definition.
2. Marking proceeds unchanged, so it now marks only memory reachable from live goroutines.
3. At the end of marking, the runtime inspects each blocked goroutine that is not yet a root. If any primitive blocking it got marked, it becomes a root and marking resumes. This is the inductive step of the definition.
4. When no more goroutines can be added, every goroutine that never became a root is flagged as leaked.
5. Marking resumes one last time with the leaked goroutines added as roots, so the cycle retains exactly what a regular cycle would have.

The last step matters: detection does not reclaim anything. Leaked goroutines and their memory stay where they were, and the profiler then takes an ordinary goroutine profile filtered down to the flagged ones.

What it misses

Three limits follow from the design.

A primitive that stays reachable from a global variable or from a runnable goroutine keeps every goroutine blocked on it live, even if nothing will ever touch it again. The post calls this memory overreach, and the only mitigation it offers is tighter scoping of references to channels and locks.

Detection covers only Golang's first-class primitives, for correctness reasons: channel sends and receives (including on nil channels), select without a default case (including the empty select {}), and sync.Mutex, RWMutex, WaitGroup and Cond. A goroutine blocked on file or network IO, a raw system call, or a hand-rolled spin lock is never reported, unless the custom primitive is itself built on the ones above.

A leak is visible only after it has happened. The profiler does not predict leaks, so a flaky program still has to leak in front of it. The post recommends combining it with goleak and synctest in tests rather than choosing one.

Cost

Memory overhead is small bookkeeping. CPU is the real price, and the post illustrates the worst case with a leak-free "daisy chain": runnable G₀ references primitive P₁ that blocks G₁, G₁ references P₂ that blocks G₂, and so on. Proving each link live requires having marked everything reachable from the previous one, so marking is effectively serialized along the chain. The end-of-round inspection also rechecks all blocked goroutines each time, which is O(n²) steps per cycle in the number of goroutines. The authors expect to optimize the second cost; the first is intrinsic to the approach.

Two facts keep this tolerable. The GC still runs concurrently with user code by default. And a leak, once present, stays present for the rest of the process's life, so a profiling setup can sample rarely (the post suggests every 4 hours) without missing anything that a more frequent schedule would have caught.

Patterns it catches

The post's playground set runs from trivial to cross-package, and several come from real fixes in large Golang projects:

• a double send, where a missing return after an error-path send makes the goroutine send twice to a receiver that reads once;
• an early return or a ctx.Done() branch in a select that abandons an unbuffered sender, fixed with a buffer of 1;
• range over a channel that is never closed, which leaks every worker, plus the parent sender if the worker count is zero;
• a Start/Stop contract hidden behind an interface, where a caller that never calls Stop strands the background loop;
• a break out of a lock loop without Unlock (CockroachDB #584), and WaitGroup.Wait() placed inside the spawn loop instead of after it (Moby #25384);
• channel-ordering races, where Stop and run finish their handshake and leave a concurrent Status sender stuck (etcd #6857);
• mutual blocking between a mutex and a channel, where one goroutine holds a lock while sending and the would-be receiver waits on the same lock (Kubernetes #6632, Moby #28462).

The profile points at the blocking line, not at the cause. For the double send it highlights the second send, and the missing return is still for the reader to find. The mixed lock-and-channel cases are the ones where that pointer is most useful, because the two leaked goroutines show up together at the send and at the Lock() call.

Like ThreadSanitizer for data races, a clean goroutineleak profile is evidence, not proof: it says nothing about goroutines stuck in IO or behind a primitive that some global still references.