Map

needle

Toolbox toolboxpythonllmon-device-aifunction-callingdistillationwatchlistPythonMIT ↳ show in map Markdown
title
needle
type
toolbox
summary
26M-parameter function-calling model distilled from Gemini 3.1 for phones, watches and glasses
tags
python, llm, on-device-ai, function-calling, distillation, watchlist
repo
https://github.com/cactus-compute/needle
language
Python
license
MIT
created
2026-05-13
updated
2026-05-13

Needle is a 26M-parameter function-calling model distilled from Gemini 3.1 by Cactus Compute. It runs at 6000 toks/sec prefill and 1200 decode on Cactus's mobile inference runtime — the target devices are phones, watches, glasses. Weights and dataset-generation code are open on Hugging Face (Cactus-Compute/needle).

Architecture

Cactus calls the underlying shape a "Simple Attention Network" — a cross-attention encoder-decoder where the encoder is FFN-free:

d=512, 8 heads / 4 KV heads, BPE vocab 8192
- Encoder × 12: ZCRMSNorm → Self-attn (GQA+RoPE) → Gated Residual.  No FFN.
- Decoder × 8:  ZCRMSNorm → Masked Self-attn (RoPE) → Cross-attn (to encoder) → Gated Residual.
- Embedding shared between input text and decoder input.
- Output projection tied to embedding.
- Output is [EOS]<tool_call> + answer; softmax over the shared vocab.

ZCRMSNorm (zero-centered RMSNorm) and gated residuals are the two unusual choices; the encoder dropping FFNs is the other. The detailed write-up is in docs/simple_attention_networks.md.

Training:

  • Pretrain — 16 × TPU v6e for 27 hours, 200B tokens (PleIAs/SYNTH).
  • Post-train — 2B tokens of single-shot function-call data, 45 minutes.

What it's for

Single-shot function calls on personal-AI devices: weather, alarms, app actions. The README is candid that this is a targeted model — it beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, LFM2.5-350m on single-shot function calling, but conversational models with more capacity outperform Needle in dialogue. Small models are finicky; the recommended path is finetuning on your own tools via the bundled playground.

Usage

git clone https://github.com/cactus-compute/needle.git
cd needle && source ./setup
needle playground   # web UI at 127.0.0.1:7860, weights auto-download

Python API:

from needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer

params, config = load_checkpoint("checkpoints/needle.pkl")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer()

result = generate(
    model, params, tokenizer,
    query="What's the weather in San Francisco?",
    tools='[{"name":"get_weather","parameters":{"location":"string"}}]',
    stream=False,
)
# [{"name":"get_weather","arguments":{"location":"San Francisco"}}]

CLI:

needle playground        # web UI: test + finetune on your own tools
needle finetune data.jsonl
needle run --query "..." --tools '...'
needle train
needle pretrain
needle eval --checkpoint <path>
needle generate-data     # synthesise training data via Gemini
needle tpu <action>      # TPU mgmt (docs/tpu.md)

The finetuning loop in needle playground calls Gemini to generate data, trains, evaluates, and bundles the result — the explicit pitch is "click a button, get a personal-AI function caller."

Where it fits

Needle is on the small-on-device end of the local-LLM thread that runs through Ronacher's local-models post and Neward's local-AI case. Those argue why local matters; Needle is one shape of how — distil a frontier model into 26M parameters and target a specific task instead of trying to compress a generalist. The on-device family it ships into is the same one claude-emotion-concepts and frontier-model research are increasingly far from — there is a real divide between "research lab compressing intelligence" and "phone OS shipping a function caller", and Needle sits firmly on the phone side.

The Cactus inference runtime (github.com/cactus-compute/cactus) is the deployment substrate; without it the 6000/1200 toks/sec numbers don't transfer.

Limitations

  • Single-shot function call only. Multi-turn agentic behaviour is out of scope; the post-train data shape doesn't support it.
  • Beats other tiny models on its niche, loses to mid-size models on anything else.
  • "Small models can be finicky" is the README's own caveat — expect to finetune.
  • 26M params + 8K BPE vocab means English-only by training; multilingual support would require new tokenisation and pretraining.
  • Single-team project (Cactus Compute), four days old at ingest — on the watchlist for adoption / finetuned-checkpoint distribution.

Repo: https://github.com/cactus-compute/needle (MIT, 275 stars, single-shot function-calling on-device).

Linked from