Cactus Needle Agentic LLM for tiny devices test

Summary

Cactus Needle Agentic LLM for tiny devices test

WORK IN-PROGRESS
This is a test version. so, USE THIS AT YOUR OWN RISK.

Environment

build all and tested on GNU/Linux

GNU/Linux: Ubuntu 24.04_x64 LTS
g++: 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1) 
Zig: zig-x86_64-linux-0.17.0-dev.1676+c9dc9b798

Build and Run

-----------------------------------------------------
Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in 28MB of RAM. It is built on our Simple Attention Network findings, compressed to CQ2-bit with Cactus Quants, and baked into its own engine. On the benchmarks below, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, at 5x to 70x smaller, and 2 bits against their f16. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series. With a peak session RAM around 28MB, Needle reaches microcontrollers like the ESP32-P4; others have reported running it on an ESP32-S3 in about 11MB.

 - Self-contained: model baked into the binary, no runtime, no downloads, no network.
 - Runs everywhere: ARM64, x86-64, ARMv7, RISC-V, and WebAssembly, on Apple, Windows, Linux, Android, Raspberry Pi.
 - Simple contract: tool calls come back as structured data, text in, JSON out; a byte-level grammar compiled from your schemas constrains every token.
 - Confidence-gated: every response carries a calibrated confidence score from a learned head; set a threshold, act above it, escalate below it.
 - Tool retrieval: declare a large catalogue and a built-in retrieval head renders only the top five tools per turn, with the grammar constrained to that subset.
 - Bounded memory: a 256-token sliding window with the tools pinned as KV sinks, so total memory stays near 28MB no matter how long the conversation runs.

Source, engine, and training code: github.com/cactus-compute/needle.



Reference:
 - https://huggingface.co/Cactus-Compute/needle2
 - https://huggingface.co/Cactus-Compute/needle2/tree/main/linux-x86_64
 - https://huggingface.co/Cactus-Compute/needle2/tree/main/android-arm64
 - https://github.com/cactus-compute/needle


Dependencies:
    libneedle.a
    libneedle.h
    needle2.cact

    $ wget -O needle2.cact https://huggingface.co/Cactus-Compute/needle2/resolve/main/needle2.cact?download=true

    // x86_64
    $ wget -O libneedle.a https://huggingface.co/Cactus-Compute/needle2/resolve/main/linux-x86_64/libneedle.a?download=true
    $ wget -O libneedle.h https://huggingface.co/Cactus-Compute/needle2/resolve/main/linux-x86_64/needle.h?download=true

    // android-arm64
    $ wget -O libneedle.a https://huggingface.co/Cactus-Compute/needle2/resolve/main/android-arm64/libneedle.a?download=true
    $ wget -O libneedle.h https://huggingface.co/Cactus-Compute/needle2/resolve/main/android-arm64/needle.h?download=true

    (optional: prebuilt executable binary)
    // x86_64
    $ wget -O needle https://huggingface.co/Cactus-Compute/needle2/resolve/main/linux-x86_64/needle?download=true
    // android-arm64
    $ wget -O needle https://huggingface.co/Cactus-Compute/needle2/resolve/main/android-arm64/needle?download=true


Build:
    $ sudo apt-get install build-essential

    // DO NOT USE {
        // clang++
        $ sudo apt-get update && sudo apt-get install clang libc++-dev libc++abi-dev
        $ clang++ -o test_needle_lib test_needle_lib.cpp -L. -lneedle -stdlib=libc++ -lpthread -std=c++17 

        // g++: LLVM libc++
        $ sudo apt-get install libc++-dev libc++abi-dev
        $ g++ -o test_needle_lib test_needle_lib.cpp -L. -lneedle -lc++ -lc++abi -lpthread -std=c++17 
    // DO NOT USE }


    // (Recommended)
    // Zig: x86_64 (Ubuntu 24.04 LTS)
    $ wget https://ziglang.org/builds/zig-x86_64-linux-0.17.0-dev.1676+c9dc9b798.tar.xz
    $ tar xJvf zig-x86_64-linux-0.17.0-dev.1676+c9dc9b798.tar.xz
    $ zig c++ -o test_needle_lib test_needle_lib.cpp -L. -lneedle -target x86_64-linux -std=c++17
    $
    $ sudo apt-get install jq
    $ ./test_needle_lib | jq .
    or
    $ bash build.sh
    $ ./test_needle_lib | jq .


    //! NOT TESTED
    // (Recommended)
    // Zig: for Android (AArch64) (.so file for JNI)
    $ export NDK_PATH="$HOME/Android/Sdk/ndk/25.1.8937393"
    $ zig c++ \
        -o test_needle_lib.so test_needle_lib.cpp \
        -L. -lneedle \
        -target aarch64-linux-android \
        -shared -std=c++17 \
        -I"$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include" \
        -I"$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/aarch64-linux-android" \
        -L"$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/29" \
        -lnotexist_dummy_to_force


Run:
    $ sudo apt-get install jq
    $ ./test_needle_lib | jq .
    or
    $ bash build.sh
    $ ./test_needle_lib | jq .


Results:
    build: executable binary ...
    [+] build [SUCCESS]
    {
      "type": "call",
      "success": true,
      "error": null,
      "error_code": null,
      "reason": null,
      "function_calls": [
        {
          "name": "set_lights",
          "arguments": {
            "room": "living room",
            "state": "on",
            "brightness": 30
          }
        }
      ],
      "reasoning": null,
      "confidence": 0.9984,
      "prefill_tps": 187.4,
      "decode_tps": 116.0,
      "peak_ram_mb": 21.5,
      "validation": {
        "ungrounded": [],
        "negation": false
      }
    }
-----------------------------------------------------