Crabby Buddy

A simple command-line interface for chatting with local LLMs, built in Rust. It runs GGUF models locally via llama.cpp (through the llama-cpp-2 bindings) — no API, no network.

Features

• Real-time AI chat through the terminal
• Runs local GGUF models on CPU/GPU (GPU offload enabled by default)
• Persistent sessions with fast KV-cache resume, plus an incognito mode
• Interactive arrow-key picker for models and sessions
• Vi-style line editing and input history

Installation

From source

git clone https://github.com/AshLink95/CrabbyBuddy.git
cd CrabbyBuddy
cargo build --release

For an optimized build on Linux, consider:

RUSTFLAGS="-C target-cpu=native -C lto=fat" cargo build --release --target x86_64-unknown-linux-gnu

[!note] --features The build will be CPU only unless you use this flag.
If you're using a nvidia GPU, append --features cuda. For any other gpu, use --features vulkan

With these flags, the binary should be at:

CrabbyBuddy/target/x86_64-unknown-linux-gnu/release/crabbybuddy

Setup

Crabby Buddy reads models and stores sessions under your local data directory:

$XDG_DATA_HOME/crabbybuddy/     (typically ~/.local/share/crabbybuddy/)
├── models/     # put your .gguf model files here
└── sessions/   # persistent sessions are saved here automatically

Drop one or more .gguf files into models/ before running.

Usage

# Start chatting
crabbybuddy

# Show help
crabbybuddy --help

On launch you'll pick a model, then choose a session: go incognito (nothing saved), an existing saved session, or new chat (creates a persistent session). Navigate with the arrow keys (or j/k) and press Enter to select.

Type your messages and press Enter. Type \bye! or press Ctrl-C to exit.