# needle

Needle is a 26M-parameter function-calling model distilled from Gemini 3.1 by [Cactus Compute](https://github.com/cactus-compute/cactus). It runs at 6000 toks/sec prefill and 1200 decode on Cactus's mobile inference runtime — the target devices are phones, watches, glasses. Weights and dataset-generation code are open on Hugging Face (`Cactus-Compute/needle`).

## Architecture

Cactus calls the underlying shape a "Simple Attention Network" — a cross-attention encoder-decoder where the encoder is FFN-free:

```
d=512, 8 heads / 4 KV heads, BPE vocab 8192
- Encoder × 12: ZCRMSNorm → Self-attn (GQA+RoPE) → Gated Residual.  No FFN.
- Decoder × 8:  ZCRMSNorm → Masked Self-attn (RoPE) → Cross-attn (to encoder) → Gated Residual.
- Embedding shared between input text and decoder input.
- Output projection tied to embedding.
- Output is [EOS]<tool_call> + answer; softmax over the shared vocab.
```

ZCRMSNorm (zero-centered RMSNorm) and gated residuals are the two unusual choices; the encoder dropping FFNs is the other. The detailed write-up is in [`docs/simple_attention_networks.md`](https://github.com/cactus-compute/needle/blob/main/docs/simple_attention_networks.md).

Training:

- **Pretrain** — 16 × TPU v6e for 27 hours, 200B tokens (PleIAs/SYNTH).
- **Post-train** — 2B tokens of single-shot function-call data, 45 minutes.

## What it's for

Single-shot function calls on personal-AI devices: weather, alarms, app actions. The README is candid that this is a *targeted* model — it beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, LFM2.5-350m on single-shot function calling, but conversational models with more capacity outperform Needle in dialogue. Small models are finicky; the recommended path is finetuning on your own tools via the bundled playground.

## Usage

```bash
git clone https://github.com/cactus-compute/needle.git
cd needle && source ./setup
needle playground   # web UI at 127.0.0.1:7860, weights auto-download
```

Python API:

```python
from needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer

params, config = load_checkpoint("checkpoints/needle.pkl")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer()

result = generate(
    model, params, tokenizer,
    query="What's the weather in San Francisco?",
    tools='[{"name":"get_weather","parameters":{"location":"string"}}]',
    stream=False,
)
# [{"name":"get_weather","arguments":{"location":"San Francisco"}}]
```

CLI:

```
needle playground        # web UI: test + finetune on your own tools
needle finetune data.jsonl
needle run --query "..." --tools '...'
needle train
needle pretrain
needle eval --checkpoint <path>
needle generate-data     # synthesise training data via Gemini
needle tpu <action>      # TPU mgmt (docs/tpu.md)
```

The finetuning loop in `needle playground` calls Gemini to generate data, trains, evaluates, and bundles the result — the explicit pitch is "click a button, get a personal-AI function caller."

## Where it fits

Needle is on the small-on-device end of the local-LLM thread that runs through [[lucumr-local-models|Ronacher's local-models post]] and [[titit-local-ai|Neward's local-AI case]]. Those argue why local matters; Needle is one shape of *how* — distil a frontier model into 26M parameters and target a specific task instead of trying to compress a generalist. The on-device family it ships into is the same one [[claude-emotion-concepts]] and frontier-model research are increasingly far from — there is a real divide between "research lab compressing intelligence" and "phone OS shipping a function caller", and Needle sits firmly on the phone side.

The Cactus inference runtime ([github.com/cactus-compute/cactus](https://github.com/cactus-compute/cactus)) is the deployment substrate; without it the 6000/1200 toks/sec numbers don't transfer.

## Limitations

- Single-shot function call only. Multi-turn agentic behaviour is out of scope; the post-train data shape doesn't support it.
- Beats other tiny models on its niche, loses to mid-size models on anything else.
- "Small models can be finicky" is the README's own caveat — expect to finetune.
- 26M params + 8K BPE vocab means English-only by training; multilingual support would require new tokenisation and pretraining.
- Single-team project (Cactus Compute), four days old at ingest — on the watchlist for adoption / finetuned-checkpoint distribution.

Repo: https://github.com/cactus-compute/needle (MIT, 275 stars, single-shot function-calling on-device).