# Tokio gives progress, not ordering

Pranitha's account of a memory spike in an event-driven Rust service, written as a prequel to her post about allocator behaviour — this is the application-side investigation that came *before* they worked out the memory was tied to the allocator.

The service read events off a queue (Kafka, Redis Streams or NATS) and processed each one by fanning out. Each event carried a ~4KB payload and up to 1000 user tokens; each token needed an outbound call, and the responses were fanned back in:

```rust
loop {
    let event: Event = fetch_next_event().await;
    tokio::spawn(async move {
        let mut tasks = JoinSet::new();
        for token in &event.user_tokens {
            tasks.spawn(async move { process(token, data).await });
        }
        let mut responses = Vec::with_capacity(event.user_tokens.len());
        while let Some(res) = tasks.join_next().await {
            responses.push(res)
        }
        generate_response_event(event, responses);
    });
}
```

The fan-out is unbounded, which the author assumed was fine in practice. The tasks are short-lived — a few milliseconds each, then dropped — so tasks spawned early should finish early. Individual outbound calls completing out of order was expected; earlier *events* finishing before later ones seemed like a safe assumption.

## What the logs showed

During a burst of 1000 events at ~1000 tokens each — roughly 1M tasks — the logs interleaved like this:

```
finished: event 779
finished: event 976

started: event 900, user 42
started: event 900, user 261
started: event 1, user 974
started: event 1, user 831
```

Event 1's token tasks were getting their *first poll* after events 779 and 976 had already completed. The burst still finished inside its time budget, so this wasn't a latency problem. The surprise was the drift between submission order and first poll.

## Why the scheduler does that

Tokio's multi-threaded runtime has a fixed set of worker threads, a local queue per worker with a capacity of 256 tasks, and a global queue shared across workers. On local-queue overflow a worker moves half its tasks to the global queue. Workers prefer their own local queue, check the global queue occasionally, and steal from other workers when idle.

Once a task is independently schedulable, the runtime has no idea which event created it. The 1000 token tasks from one event are mixed in with token tasks from every other event, with the parent event tasks parked on `JoinSet`, and with tasks waking from I/O readiness — all of them just runnable work competing for a queue slot. Overflow and work stealing then reorder pickup relative to submission. The distinction the author lands on:

```
task created != task polled != task completed
```

The memory consequence follows directly. Every task carries state, individually small, and peak memory tracks how many tasks are *live at once*, not how many are running. A few token tasks from early events surviving to the end of the burst kept their parent event tasks alive, which kept the 4KB payloads and token vectors alive with them.

## The fairness guarantee has a precondition

Tokio's [documented fairness](https://docs.rs/tokio/latest/tokio/runtime/index.html#detailed-runtime-behavior) holds under a bounded number of tasks, assuming no task blocks a worker thread. The original code had no bound anywhere in the chain — events were read as fast as they arrived, each spawning a task that spawned up to 1000 more. Tokio accepts whatever it is given and keeps making progress on all of it; the boundedness its guarantee assumes has to come from the application.

What they actually wanted was *event-level* fairness — all token tasks belonging to one event polled and finished near their submission time. A `Semaphore` capping how many events enter processing at once gave them that, with the permit count found by trial and error, since the right number depends on how long each task takes to return from a poll. Throughput was unaffected: the burst still finished on time, with peak memory down significantly.

## Why it was easy to miss

Nothing was wrong. No task was dropped, no request was slow, the throughput target was met. It surfaced only as a memory spike with no corresponding cause anywhere in the application logic — the mismatch was between the author's mental model (an event is a unit of work that starts and finishes) and the runtime's (a task is a task).

That is the general shape of the problem: if you have an application-level unit of fairness you expect the runtime to honour, the runtime does not know it exists. [[safety-in-an-unsafe-world]] is the argument that libraries should encode their invariants in types so that violating them fails to compile; this is the counter-case, where the invariant belongs to the caller and there is no type Tokio could have offered to express it. The practical version is a question to ask before every `tokio::spawn`: what is the maximum number of live tasks this can produce, and what does each of them hold while it waits?