LDE – Linked Data Elements

Shared building blocks for the full Linked Data lifecycle.

📖 Documentation – guide, package reference and design decisions.

Every organization working with Linked Data ends up building the same infrastructure from scratch: endpoint management, data import, transformation pipelines, dataset discovery.

LDE covers the full Linked Data lifecycle – from discovery and ingestion through transformation to publication – as an open-source toolkit of composable building blocks for Node.js.

Data transformations are expressed as plain SPARQL queries: portable, transparent and free of vendor lock-in.

Key capabilities

• Discover datasets from DCAT-AP 3.0 registries.
• Download and import data dumps to a local SPARQL endpoint for querying.
• Transform datasets with pure SPARQL CONSTRUCT queries: composable stages with fan-out item selection.
• Analyze datasets with VoID statistics and SPARQL monitoring.
• Publish results to SPARQL endpoints or local files.
• Serve RDF data over HTTP with content negotiation (Fastify plugin).

Standards

Standard	Usage
DCAT-AP 3.0 (EU)	Dataset discovery and registry queries
SPARQL 1.1	Data transformations, dataset queries and endpoint management
SHACL	Per-class sampling (@lde/pipeline-shacl-sampler), validation (@lde/pipeline-shacl-validator) and documentation generation (@lde/docgen)
VoID	Statistical analysis of RDF datasets (@lde/pipeline-void)
RDF/JS	Internal data model (N3)
LDES (EU)	Event stream consumption and publication (planned)

Quick example

import {
  Pipeline,
  Stage,
  SparqlConstructReader,
  SparqlItemSelector,
  SparqlUpdateWriter,
  ManualDatasetSelection,
} from '@lde/pipeline';

const pipeline = new Pipeline({
  datasetSelector: new ManualDatasetSelection([dataset]),
  stages: [
    new Stage({
      name: 'per-class',
      itemSelector: new SparqlItemSelector({
        query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
      }),
      readers: new SparqlConstructReader({
        query:
          'CONSTRUCT { ?class a <http://example.org/Class> } WHERE { ?s a ?class }',
      }),
    }),
  ],
  writers: new SparqlUpdateWriter({
    endpoint: new URL('http://localhost:7200/repositories/lde/statements'),
  }),
});

await pipeline.run();

Packages

Discovery – Find and retrieve dataset descriptions from registries
@lde/dataset	Core dataset and distribution objects
@lde/dataset-registry-client	Retrieve dataset descriptions from DCAT-AP 3.0 registries
Processing – Transform, enrich and analyse datasets with SPARQL pipelines
@lde/pipeline	Build pipelines that query, transform and enrich Linked Data
@lde/pipeline-shacl-sampler	Per-class sampling stages derived from SHACL shapes
@lde/pipeline-shacl-validator	SHACL validation for pipeline stages
@lde/pipeline-void	VoID statistical analysis for RDF datasets
@lde/distribution-downloader	Download distributions for local processing
@lde/distribution-health	Derive distribution usability from reachability and RDF validity
@lde/distribution-probe	Probe distributions for availability and metadata
@lde/iiif-validator	Validate that a URL resolves to a valid IIIF Presentation Manifest
@lde/sparql-importer	Import data dumps to a local SPARQL endpoint for querying
Publication – Serve and document your data
@lde/fastify-rdf	Fastify plugin for RDF content negotiation and request body parsing
@lde/docgen	Generate documentation from RDF such as SHACL shapes
@lde/search	The search core: projects RDF into engine-agnostic search documents (framing + a declarative field spec)
@lde/search-api-graphql	Engine- and domain-agnostic GraphQL surface for search: builds an executable GraphQL schema from a SearchSchema at runtime and serves it as a framework-agnostic fetch handler with a self-contained playground
@lde/search-api-server	The served search API as a bootable process and prebuilt Docker image: mounts a schema-declaration module, binds the GraphQL handler to a Typesense engine, and serves /graphql plus /health from environment config
@lde/search-indexer	The search indexer as a bootable process and prebuilt Docker image: mounts the same schema-declaration module as the API server, selects datasets from a registry, and rebuilds the Typesense collections from environment config
@lde/search-pipeline	Applies the @lde/search projection inside an @lde/pipeline run, fanning each document out to the transactional engine writer for its type’s collection (owns no projection itself)
@lde/search-typesense	Typesense engine adapter: transactional Blue/green (build fresh, swap atomically) and In-place (update the live index) rebuild writers
@lde/text-normalization	Text folding (diacritic stripping and transliteration) for search index and query normalization
Monitoring – Observe pipeline runs and endpoint health
@lde/distribution-monitor	Monitor DCAT distributions (SPARQL endpoints and data dumps) with periodic probes
@lde/pipeline-console-reporter	Console progress reporter for pipelines
Infrastructure – Manage SPARQL servers and run tasks
@lde/local-sparql-endpoint	Quickly start a local SPARQL endpoint for testing and development
@lde/sparql-server	Start, stop and control SPARQL servers
@lde/sparql-qlever	QLever SPARQL adapter for importing and serving data
@lde/wait-for-sparql	Wait for a SPARQL endpoint to become available
@lde/task-runner	Task runner core classes and interfaces
@lde/task-runner-docker	Run tasks in Docker containers
@lde/task-runner-native	Run tasks natively on the host system

Architecture

See the architecture documentation for how the packages fit together, including the full dependency diagram.

Who uses LD Elements

 Netwerk Digitaal Erfgoed – Dutch national digital heritage infrastructure, commissioned by the Ministry of Education, Culture and Science

 DC4EU – Dutch Collections for Europe, the accredited national aggregator connecting Dutch heritage institutions with Europeana

Linked Open Limburg – Coöperatie Erfgoed Limburg’s regional heritage programme, built on LDE’s pipeline components

Comparison

See when to use LDE for how LDE compares to TriplyETL, rdf-connect and OpenLDES/LDI.

Development

Prerequisites: Node.js (LTS) and npm.

npm install
npx nx run-many -t build
npx nx run-many -t test
npx nx affected -t lint test typecheck build  # only changed packages

The documentation site lives in docs/; serve it locally with npm run docs:dev.

See CONTRIBUTING.md for the full development workflow.

License

MIT – see LICENSE.

Acknowledgements

LD Elements originated at the Dutch national infrastructure for digital heritage (NDE).