SPARQL Anything for non-RDF origin data

[ddeboer]

Pipeline step that can work with non-RDF data and convert it to RDF for subsequent steps.

[ddeboer]

Updated take: after working through this against #81, I think the right framing is that this is a specific application of CommandExecutor, not a separate executor. Leaving the motivation below since it’s still useful context for why non-RDF ingest matters.

---

Concrete motivation from porting netwerk-digitaal-erfgoed/geonames-rdf to LDE: that repo today is a shell pipeline (download.sh + map.sh) that wraps SPARQL Anything to turn the GeoNames TSV dump (∼12M rows) into RDF. The bulk of its complexity is workarounds for SPARQL Anything’s memory profile at that scale:

• fx:ondisk + fx:ondisk.reuse false to keep memory bounded
• split -l 1000000 to chunk the TSV before invoking the jar
• A per-chunk loop that re-prepends headers and concatenates *.csv.ttl at the end
• An awk step that fabricates adm1/adm2 foreign-key columns with a NONE sentinel, purely so the CONSTRUCT can join without OPTIONAL

These are all CLI invocations chained together. The win from LDE isn’t in re-implementing any of them — it’s in making the chain declarative and composable.

What the user authors: the full SPARQL CONSTRUCT query (including its SERVICE <x-sparql-anything:> block with fx:ondisk, fx:csv.delimiter, fx:null-string, etc.). LDE doesn’t parse, splice, or vet that query. Same contract as SPARQL stages today — the framework executes the query, it doesn’t introspect the SPARQL.

What LDE provides: a CommandExecutor (per #81) that spawns java -jar sparql-anything.jar -q <query> and streams N-Triples from stdout into the next stage. Chunking is also a CommandExecutor step (split -l N, or jq -c for NDJSON, or xmlstarlet sel for XML) — not something LDE’s ingest stage needs to know about. The format problem stays in SPARQL Anything; the chunking problem stays in whichever CLI tool fits the format.

This collapses the geonames-rdf shell pipeline to a 1:1 translation: each shell invocation becomes one CommandExecutor stage, with proper dependency tracking and observability layered on top.

Recommend closing this in favour of #81 — SparqlAnythingExecutor is a usage example of CommandExecutor, not a separate executor worth building. Earlier framings in this thread (format-specific drivers inside LDE, iterating vs slurp models, a curated fx:properties API surface) are obsolete under the CommandExecutor approach: drivers don’t exist, iteration is whatever the user’s chunker does, and fx: config lives inside the user’s query where it belongs.

[ddeboer]

One more thing the stage needs to get right: the full SPARQL Anything configuration has to be passthrough-able by the client. Not a curated subset.

For the GeoNames case the workable config is specifically:

fx:ondisk          "/tmp/sparql-anything-geonames" ;
fx:ondisk.reuse    false ;
fx:null-string     "" ;
fx:csv.delimiter   "\t" ;
fx:csv.headers     true ;

Drop any one of these and the pipeline either OOMs (ondisk, ondisk.reuse), produces wrong triples (null-string, csv.headers), or fails to parse the input at all (csv.delimiter). And the matrix is much bigger across the other SPARQL Anything formats — JSON path expressions, XML namespace bindings, HTML selectors, Bibtex, archive traversal, etc. The stage can’t reasonably enumerate which fx: params it supports; whichever ones it filters out are exactly the ones the next user will need.

So the contract should be: the client supplies the full fx:properties block (or equivalent), and the stage hands it to SPARQL Anything unchanged. The stage’s job is iteration and batching, not vetting source-specific tuning knobs. Same principle as how endpoint: works for SPARQL stages today — LDE doesn’t try to validate the remote store config, it just uses it.

That also makes non-OOM behaviour the client’s lever to pull rather than something LDE has to solve generically: if the source needs ondisk, the client sets ondisk; if it fits in memory, they don’t.

[ddeboer]

Lessons from a hardened SPARQL Anything prototype

netwerk-digitaal-erfgoed/geonames-rdf is effectively a hand-rolled version of the step this issue describes: it ingests non-RDF input (the GeoNames TSV dump) and converts it to RDF with SPARQL Anything for downstream use. We recently hardened it after a production incident, and the lessons translate almost directly into requirements for a generic LDE step. Sharing them here so the design starts from what we already learned the hard way.

Reference files: map.sh (the conversion loop) and config/places.rq (the CONSTRUCT + CSV options).

1. A "no partial success" contract per item

The incident: SPARQL Anything (v1.1.0) deletes its output file on a crash (upstream #547). Our loop did not check exit codes, so a crashed chunk left no .ttl, the downstream concatenation silently skipped it, and ~1M rows vanished on a green build.

For an LDE step this matters because the transform stage already does fan-out item selection, so it has the same exposure per item. The step must:

• check each invocation’s exit status and treat a non-zero exit as a hard stage failure;
• treat missing or empty output as failure too (not an empty-but-valid result);
• never let the pipeline reach a publish step with a partial result – "some items succeeded" fails the stage.

2. Completeness gating belongs in the pipeline – and LDE can do it structurally

Our shell prototype counts features by grepping the serialized Turtle for a type token. That is crude precisely because there is no clean input-row-to-output-feature relation (the CONSTRUCT filters rows), and because grepping a serialization is brittle.

In LDE you operate on a parsed RDF graph (N3/RDF-JS), so the same check becomes exact: count ?s a <type> triples, or instance counts per class. This is the natural home for a completeness gate, and the neighbours already exist:

• @lde/pipeline-void can emit the counts;
• a small regression step could compare this run’s counts against the previous run’s.

One nuance worth baking into the design: a static floor (“fail if below N”) mostly re-detects the crash case that the per-item checks already catch earlier and more precisely. We tried one in the prototype and then dropped it for exactly this reason – its trigger range almost entirely overlapped the per-item checks. The aggregate gate only earns its place as a tight relative check (e.g. fail on a >1–2% drop versus the previous run), because that is what catches the residual mode the per-item checks cannot see: an item that exits 0 with non-empty output but silently fewer triples. The prototype currently relies on the per-item checks; a relative regression gate is the recommended aggregate and a natural fit for @lde/pipeline-void plus a stored previous-run count.

3. CSV/parser options are step-API requirements, not source quirks

The production bug was a literal " inside an unquoted TSV: Apache Commons CSV (SPARQL Anything’s parser) treats " as a quote char by default and threw a parse exception, crashing the chunk. We worked around it with fx:csv.quote-char "�" (a control char that never appears in the data). Alongside it we rely on fx:null-string, fx:csv.delimiter, and fx:csv.headers.

These are not GeoNames quirks – they are the knobs any CSV/TSV source needs. The step should surface them as typed options and document the literal-quote trap. Relevant upstream issues to design around:

• SPARQL-Anything #625 – fx:csv.no-quote (until then, the control-char workaround is the pattern);
• SPARQL-Anything #624 – configurable batch size (fx:slice.size);
• SPARQL-Anything #547 – delete-output-on-crash (the reason section 1 matters).

4. Large inputs need chunking + on-disk storage

To avoid OOM on the full dump we split the input into ~1M-row chunks and run each individually with fx:ondisk and fx:ondisk.reuse false. A generic step needs a chunking strategy baked in rather than bolted on – and this dovetails with LDE’s fan-out model: each chunk is one fanned-out item, which gives you the per-item failure contract from section 1 for free.

The one-line takeaway

A green pipeline that publishes incomplete data is worse than a red one. The step ordering should make "prove completeness before publish" first-class: convert → count/validate (VoID + relative regression) → only then publish. Everything in map.sh is the shell-script rehearsal of that.

[ddeboer]

Cross-link: #81 (CommandExecutor) subsumes this, and is the better home for most of the above. The failure contract (section 1) is generic and belongs in CommandExecutor; only the CSV options and N-Triples parsing are specific enough for a SparqlAnythingExecutor. Mapped the four lessons onto those layers in #81.

[ddeboer]

Implementation plan (geonames-rdf → LDE, convert-only first)

Concrete plan from the design session. Architecture (the CommandExecutor/CommandConverter split and layering) lives in #81 — this is the geonames-specific execution scope.

Scope: convert step only. Replace map.sh’s SPARQL Anything loop with an LDE component producing the RDF output. Out of scope for this pass: download.sh (fetch/unzip/awk foreign keys/admin codes) and harvest.yml (schedule + S3 upload) — both stay in netwerk-digitaal-erfgoed/geonames-rdf, too geonames-specific to merge into LDE.

What geonames uses: the converter shape, not an executor. geonames convert is file-in → file-out with no in-Node post-processing (places.rq does the whole transform inside SPARQL Anything), so it uses a SparqlAnythingConverter (writes a file, not an Executor, not inside Pipeline → Stage → Executor). The streaming SparqlAnythingExecutor is deferred (#81); build the process-runner core so it tucks on later without a refactor.

Chunking stays in download.sh. The 1M-row split (+ header replication) that prevents SPARQL Anything OOM stays where it works; the converter consumes the pre-split chunk files. Generalizing "header-aware splitter + run-CLI-per-chunk" is a CommandConverter/fan-out concern for later (#81), not this pass.

Output format: N-Triples. The published artifact moves from geonames.ttl to geonames.nt — concatenating chunk outputs is trivially safe in N-Triples (no prefix/document-structure dependency that catting Turtle has). This is a geonames-rdf-repo change landing first, ahead of the LDE migration; the load/index step (Fuseki, production loader) must be updated to pick up .nt.

Build order:

1. (geonames-rdf repo) Switch output to N-Triples.
2. (LDE) Process-runner core: spawn via @lde/task-runner, fail-on-nonzero contract, per-chunk orchestration.
3. (LDE) SparqlAnythingConverter on the core (file output, NT).
4. Repoint geonames convert at the LDE converter; verify parity (~13.3M gn:Feature, ~137.7M triples).
5. Deferred: streaming SparqlAnythingExecutor + the generic CommandExecutor/CommandConverter extraction (CommandExecutor for external CLI tools as pipeline steps #81), when a second consumer / post-processing need appears.

[ddeboer]

Design note – the SPARQL Anything → Pipeline connection will be a Reader.

@lde/sparql-anything's SparqlAnythingConverter (#511) is convert-only and standalone by scope: it produces an N-Triples file, deliberately outside Pipeline → Stage → Reader. That is #511's boundary, not the end state.

The pipeline integration is a Reader: it wraps the converter to yield AsyncIterable<Quad>, so non-RDF origin data enters the pipeline the same way every other source does – a Reader is the non-RDF → Quad boundary, and its output is always Quad. This keeps the pipeline's invariant intact (a stage's input is always Quad; see ADR 2 and ADR 13), with the conversion hidden inside the reader rather than as a separate pre-pipeline step.

So: convert-to-file (#511, done) → wrap as a Reader (this issue) → any pipeline consumes it as quads.