InkLeaf: Real-time Hand Writing to Document Compiler for RNote

This project is a lightweight tool that converts handwritten Rnote papers into well-structured documents (Markdown/LaTeX/Typst) using VLMs. Its core harness detects differences between autosaves and triggers the VLM to update the corresponding region in the target document.

Install

Install through uv:

uv tool install git+

Configure API key:

TODO: configure API key?

Usage

Start watching a file:

inkleaf MyDocument.rnote # Defaults to same path, MyDocument.md

You can specify the document format, document path and media path:

inkleaf Mydocument.rnote \
        --output ./docs/Document.tex \ # Output to latex
        --media_path ./docs/media/  \ # Store images in this path

Templates and Instructions

inkleaf Mynote.rnote \
        --output ./ObsidianVault/MyNote.md \ 
        --template ./docs/ObsidianNoteTemplate.md \ # Template, with custom instructions (Optional)

You can specify a structure for the file and leave instructions for the agent as well. The VLM will see all these as part of the system instruction and will initialize a blank document of the specified structure. Here is an example custom template, with Obsidian-favoured markdown.

# Obsidian Note Template

---
title: "My Task 1"
date: "2024-11-22"
description: "My task 1 description"
---

> [!abstract] TL;DR
> ## Lorem Ipsum

Agent generates a summary for document. 


> [!info] Information
> ## Subtitle

...

> [!definition] Definition
> ## Subtitle

...

> [!hint] Hint
> ## Hint for Agents

Prefer double `$$` but in a single line with equation.

$$ y= kx+b $$

Prefer `\aligned` multi-line block for long equations.

$$\begin{aligned}
\left|S - S_n \right| = & \left|\sum_{k=1}^{\infty} (-1)^k a_k - \sum_{k=1}^{n}(-1)^k a_k \right| \\
= & \left| \sum_{k=n+1}^{\infty} (-1)^k a_k \right| \\
= & \left| a_{n+1} - a_{n+2} + a_{n+3} - a_{n+4} + a_{n+5} - ...  \right| \\
= & \left| a_{n+1} - (a_{n+2} - a_{n+3}) - (a_{n+4} - a_{n+5}) - ...\right| \\
(\text{where } & (a_{n+i} - a_{n+i+1}) \geq 0) \\
\left| S- S_n\right| \leq & \quad \boxed{a_{n+1}}
\end{aligned}$$

Real-time Preview

You can always use your favorite document editor (VSCode / Obsidian) to preview the markdown file as it builds.

This project provides a web dashboard to highlight the parts that are updated during each edit. The dashboard is based on markdown-it + KaTeX and accessible through the browser and defaults to https://localhost:8080.

inkleaf Mynote.rnote \
        --output ./ObsidianVault/MyNote.md \ 
        --dashboard --port 6657

How it works

Detecting Changes

RNote saves to a compressed json file, this program detects changes to the file, which happens every 60-120s depending on your autosave config.

.rnote = JSON -> gzip
data.engine_snapshot
  ├── document.config.format  { width: 1123, height: 1587, dpi, orientation, ... }
  ├── document               { x, y, width, height }   # doc origin + extents
  ├── document.config.layout  "infinite" | "fixed-size" | "continuous-vertical" | ...
  ├── stroke_components      [ {value: <stroke>|null, version: u32}, ... ]   # slotmap
  ├── chrono_components      [ {value: {t: u32, layer: {...}}|null, version}, ... ]
  └── chrono_counter         u32

Every stroke is labelled with an ID, and detecting additional / removed/modified strokes are all simple set operations. With another bounding-box check, we can highlight the rectangular areas that are added, removed, or modified.

And RNote provides a CLI interface to quickly export notes to an image:

rnote-cli export ...

• -o, --output-file <FILE> — format inferred from extension; single input only
• -f, --output-format <FMT> — svg | pdf | xopp; output path = input path with extension swapped
• -o and -f are mutually exclusive; exactly one required
• --page-order — horizontal-first (default) | vertical-first

Updating Area

Every page that contains changed content (if spanning multiple pages) is converted to PNG:

• Gridlined: Gridlines of a specific style are added to help VLMs identify the space.
• Highlighted: The regions that are added/removed/modified will be highlighted using different colors.
• Changes to Document: The VLM inspects the change and applies edits to the target document.

Preserving Drawings

If you create illustrations in your note, the VLM will utilize toolcalls to extract the drawing region from the note and paste it as an svg image in the converted markdown. However, this relies on the VLM's vision backbone to precisely determine the bounding box based on provided gridlines, which is difficult for low-cost models.

The alternative is to circle your drawing with a huge rectangle in RNote, you can make the stroke think enough that it doesn't bother you. Whenever these boxes are detected, the VLM gets a hint to use this as the precise bounding box.

Handling Media

TODO: Extracting media from an RNote file.