Industry track: GRMP, a platform to monitor linked data resources

GRMP: An open-source platform for automatic monitoring of linked data resources on the web, by Stijn Denis, Marc Portier, Cedric Decruw and Stijn Deknudt, has been accepted in the SEMANTiCS 2026 industry track.

When publishing linked data-related resources through the web, it is strongly advised to actively monitor them and ensure they remain accessible, up-to-date and continue to provide data in the intended manner. Public registries and dashboards do exist, but they require active maintenance themselves lest they become outdated and provide incorrect information regarding the status of these resources. In order to support the management of their own LDES servers and SPARQL endpoints and monitor the status of external web resources they consume, the scientific institute VLIZ (Vlaams Instituut voor de Zee – Flanders Marine Institute) developed GRMP (Graph Resources Monitoring Platform) in cooperation with Flemish IT and software company Sirus. GRMP is an open-source and highly modular platform which allows for the automatic monitoring of linked data resources and web resources in general. GRMP is container-based and primarily consists of an orchestrator which manages and directs test suite containers based on configurations defined in one or more YAML files. Due to its modular nature, users can rapidly develop new test suites in any desired programming language, provided that they have been containerized and adhere to a few guidelines allowing for proper communication with the orchestrator. Additionally, the environment on top of which the system runs can be freely chosen, ranging from local deployments to integration with automation workflows like GitHub Actions. When running within GitHub Actions, additional features are available, such as automatic issue creation resulting from failed tests.

This is directly relevant to everyone running an LDES feed among our examples: keeping a feed reachable, well-formed and up-to-date over time is exactly the kind of maintenance burden that motivated LDES’s append-only, cache-friendly design in the first place, and GRMP tackles the operational side of that promise. VLIZ, incidentally, is also the publisher of the Marine Regions Gazetteer LDES, one of the longest-running production feeds among our examples.

A demo worth watching: streaming construction data with LDES

Also accepted at SEMANTiCS 2026 is a demo on Streaming Interoperable Construction Data with LDES, by Carlos Ramonell and Mathias Bonduel, which comes with a genuinely nice explainer video:

You can find the full entry, keywords and any accompanying materials on the conference’s demo page.

Both contributions are a good reminder that LDES keeps finding new domains — from marine biodiversity data to construction information management — united by the same simple idea: publish an append-only, cacheable stream once, and let every consumer replicate and stay in sync at their own pace.

See you at SEMANTiCS 2026!

An RDF message can contain several triples or quads that belong together and should be interpreted as one unit. Making those boundaries explicit helps software preserve the intended meaning as data moves between systems.

The work draws inspiration from Piotr Sowiński’s keynote at the LDES workshop during SEMIC 2025, where Jelly and LDES were discussed as technologies that could share a common foundation. Pieter’s conference report captures that conversation.

For LDES, this is a promising avenue for future work: a shared message model could help connect event streams with other RDF processing and transport tools. These are possibilities to explore, not requirements introduced into LDES by this draft.

Read the RDF Messages draft and join the discussion in the RDF Stream Processing Community Group repository.

Pieter Colpaert presented the updated specification after a year of development. The workshop also featured catalog harvesting, a cultural heritage server, water data, a .NET implementation, and RDF-Connect pipelines. Together, these contributions showed how a common publication interface can support different domains and software stacks.

Piotr Sowiński’s workshop keynote connected his work on Jelly with the emerging RDF Messages collaboration, opening a discussion about shared foundations for exchanging RDF streams.

For publishers and consumers, the next step is practical: read the LDES specification, explore the implementation reports, and try an example feed.

Read Pieter’s SEMIC 2025 report for the workshop recap, or visit the SEMIC conference archive for the event materials.

Let’s introduce a simple example. Consider this table, representing who follows who on a social network:

This can be represented by the following knowledge graph:

flowchart LR
    bob[Bob]
    alice[Alice]
    jeff[Jeff]
    bob -- follows --> alice
    jeff -- follows --> bob
    jeff -- follows --> alice

This graph can be broken down in simple statements describing objects and their relations, called triples. The example consists of the following triples:

flowchart LR
    bob[Bob]
    jeff[Jeff]
    jeff -- follows --> bob

flowchart LR
    bob[Bob]
    alice[Alice]
    bob -- follows --> alice

flowchart LR
    alice[Alice]
    jeff[Jeff]
    jeff -- follows --> alice

Generating a Knowledge Graph

Various specifications and software tools exist to build knowledge graphs. A widely adopted specification is the RDF Mapping Language (RML, website) which uses a declarative approach. One can describe how sources can be accessed and how data records can be extracted. Next, rules describe how these records can be turned into triples. Finally, there are rules describing where the generated graph can be written to. This is called a target.

In RML the rules to convert the table to the knowledge graph look like:

@prefix ex: <http://example.com/> .
@prefix rml: <http://w3id.org/rml/> .

<http://example.com/base/TriplesMap1> a rml:TriplesMap;

  rml:logicalSource [ a rml:LogicalSource;
      rml:referenceFormulation rml:CSV;
      rml:source [ a rml:RelativePathSource;
          rml:root rml:MappingDirectory;
          rml:path "data.csv"
        ]
    ];

  rml:subjectMap [
      rml:template "http://example.com/{Person}"
    ] ;

  rml:predicateObjectMap [
      rml:predicate ex:follows
      rml:objectMap [
          rml:template "http://example.com/{Follows}"
        ];
   ].

Let’s not dive into detail too much, but what this mapping does is:

• The @prefix statements map an abbreviation to a IRI.
• The rml:logicalSource part sais that the data comes from a file called data.csv (the table of the example)
• The rml:subjectMap assigns a unique IRI to a Person; this is required for knowledge graphs represented in RDF
• The rml:predicateObjectMap makes a connection to another person in the Follows column, and assigns a label or property ex:follows to the connection indicating the type of relation.

Because this is a bit verbose, another language YARRRML was designed to make such mapping rules a bit more intuitive. The same result could be achieved with the following rules in YARRRML:

prefixes:
  ex: http://example.com/

mappings:
  followers:
    sources: [data.csv~csv]
    subject: http://example.com/$(Person)
    predicateobjects:
      - [ex:follows, http://example.com/$(Person)]

A YARRRML processor takes these rules and generates the corresponding RML rules.

All about change: LDES

Now that we know how to generate a Knowledge Graph, let’s add a challenge. After a while Alice starts to follow Jeff. The knowledge graph now looks like this:

flowchart LR
    bob[Bob]
    alice[Alice]
    jeff[Jeff]
    bob -- follows --> alice
    jeff -- follows --> bob
    jeff -- follows --> alice
    alice -- follows --> jeff

A possible approach would be to just rebuild the whole knowledge graph. No need to change the mapping rules. And suppose we published this graph, we would have to notify the consumers that the graph changed. They would have to download the whole graph to have the latest changes. This works well for small graphs, but is not very scalable.

Enter Linked Data Event Streams (LDES), a mechanism to publish changing knowledge graphs on the web. In short, LDES can be described as a collection of immutable data. New versions of the data can be added to the collection, forming a stream of updates. This allows consumers of the data to keep up-to-date with minimal effort.

Furthermore, LDES allows fragmentation of the data making search more efficient by providing a search tree view. Fragments that are unlikely to change can be cached, e.g. using HTTP caching.

flowchart TD
    index["/"]
    2024[2024]
    2025[2025]
    12[December]
    01[January]
    02[February]
    data_1[ ]
    data_2[ ]
    data_3[ ]
    data_4[ ]
    data_5[ ]
    data_6[ ]

    index --> 2024
    index --> 2025
    2024 --> 12
    2025 --> 01
    2025 --> 02
    12 --> data_1
    12 --> data_2
    01 --> data_3
    01 --> data_4
    01 --> data_5
    02 --> data_6

This figure illustrates a LDES. The blocks at the bottom represent data objects (called members). In this case time-based hierarchical fragmentation is used. The fragments representing a month point to its members. Fragments can be linked (using the TREE hypermedia controls).

Generating a changing Knowledge Graph

So now we have the tools to generate a knowledge graph, and we know how to publish a knowledge graph that changes. But if the data changes, how to generate the changes only without rebuilding the whole graph?

IncRML describes how this can be achieved using RML. It introduces some functions that can, during the mapping process, detect changes in the data. One can specify when a change in the data triggers the generation of new triples (thus updates of the knowledge graph). Three types of changes are defined: create, update and delete. As a bonus, a target is added that can write the changes as a LDES. It requires a certain effort to get it right: every type of change (for the same data) needs a new set of rules, with the appropriate function arguments configured.

To lower this effort, we recently introduced a YARRRML extension that lowers the complexity by generating the appropriate RML mappings and functions.

Back to the example. We started with a csv file having three rows:

Alice decides to follow Jeff, so the file gets updated:

We somehow know that the data source is updated, but we are not notified about what exactly changed. The only way to know is to read the file again, then we see a new record is created. This is called an implicit change.

Another possibility is that only the changes are advertised, which is called explicit changes. For example, a Kafka topic could stream only updates in JSON format:

{
  "change": "create",
  "Person": "Alice",
  "Follows": "Jeff"
}

But let’s take the implicit change as an example. We can specify this in our YARRRML mappings like this:

prefixes:
  ex: http://example.com/

mappings:
  followers:
    sources: [data.csv~csv]
    subject: http://example.com/$(Person)
    predicateobjects:
      - [ex:follows, http://example.com/$(Person)]
    changeDetection:
      create:
        explicit: false

When the first version of the data is mapped to a knowledge graph, the three triples displayed at the top of this post are generated. When the second version is mapped, only one triple gets generated:

flowchart LR
    alice[Alice]
    jeff[Jeff]
    alice -- follows --> jeff

If we add an LDES target, everything will be mapped as a stream of changes:

prefixes:
  ex: http://example.com/

mappings:
  followers:
    sources: [data.csv~csv]
    subject: 
      - value: http://example.com/$(Person)
        targets:
          access: out-ldes.ttl
          type: localfile
          serialization: turtle
          ldes:
            id: https://my-ldes.org/social-network-ldes
    predicateobjects:
      - [ex:follows, http://example.com/$(Person)]
    changeDetection:
      create:
        explicit: false

When mapping the first version of the data with an LDES target, the knowledge graph looks like:

:Bob :follows :Alice .
:Jeff :follows :Bob,
               :Alice .

my-ldes:social-network-ldes
  a ldes:EventStream;
  tree:member :Bob,
              :Jeff .

The second run with the updated data generates following triples:

:Alice :follows :Jeff .

my-ldes:social-network-ldes
  a ldes:EventStream;
  tree:member :Alice .

When merged together the complete knowledge graph is formed:

:Bob :follows :Alice .
:Jeff :follows :Bob,
               :Alice .
:Alice :follows :Jeff .

my-ldes:social-network-ldes
  a ldes:EventStream;
  tree:member :Bob,
              :Jeff,
              :Alice .

This is only a simple example. A few more elaborate examples are available on GitHub. The corresponding RML examples can be found here.

“Does LDES scale?” becomes more useful when we ask what is growing and what consumers need. The answer guides decisions about pages, caching, retention, and processing capacity.

A shared source, independent views

LDES lets a publisher expose data and its changes through a common interface. Consumers can copy the available members, follow updates, and maintain their own databases, search indexes, or APIs. Query work can then happen on those consumer views.

A typical publication splits the stream into linked pages. A time-based view might lead from years to months and then to pages of members. A new consumer traverses the available history; a returning consumer follows the part relevant to its next updates. The LDES specification defines the contract between those publishers and consumers.

That design gives us several ways to manage growth. Each comes with a tradeoff.

More data: make history affordable

Start with the bytes you actually transfer. HTTP compression can reduce the cost of textual RDF, but the saving depends on the data and format. Measure compressed transfer size alongside parsing time: smaller responses do not automatically mean faster processing.

Page size is another balancing act. Small pages can make following recent changes efficient, but require more requests during a full copy. Large pages can reduce request overhead while making clients download more data than they need. Try both a first-time harvest and an incremental update when choosing a size.

Retention is a separate decision. A sensor publisher might keep recent observations available and publish daily aggregates for longer-term analysis. A collection publisher might retain only a subset of record versions in one view. An archive can preserve more history for consumers that need it.

Document that choice. A client that returns after a long pause can only catch up from a view if the necessary data is still available. If it has missed the retention window, it may need to rebuild its view from an appropriate snapshot or another source. Aggregates also answer different questions from raw observations; they should be described as such.

Faster updates: keep the backlog under control

There are three different measurements here: how quickly members arrive, how quickly a consumer processes them, and how long a change takes to reach an application.

A consumer needs enough sustained throughput to keep up, with spare capacity to recover after interruptions. Count the whole pipeline: downloading, parsing RDF, extracting members, applying transformations, and writing to the destination. A fast parser does not compensate for a slow database write.

For example, suppose a source publishes 100 members per second and a consumer can process 150. After an hour offline, it has 360,000 members to catch up on. While new members keep arriving, only 50 members per second of capacity remains for that backlog. Under those simplified, constant-rate assumptions, catching up takes another two hours.

Polling affects freshness too. A two-minute polling interval alone cannot promise a two-minute end-to-end delay: publication, caching, network time, and downstream processing all contribute. Choose a polling strategy that respects the publisher’s HTTP cache information and leaves room for processing. Conditional requests can avoid retransmitting unchanged responses. See the HTTP caching specification.

A throughput figure from one implementation is a benchmark for that setup, not a limit imposed by LDES. Record the dataset, hardware, page layout, network conditions, and destination alongside any number you publish.

More consumers: reuse the same responses

If many clients need the same pages, a shared HTTP cache or CDN can serve them without every request reaching the publisher. Consumers still maintain and query their own views; the publisher concentrates on making a reusable source available.

Distinguish immutable members from immutable pages. A page may acquire new members or links even though existing members never change. Only give a response a long cache lifetime when its contents support that promise.

For a finished page, the HTTP immutable directive tells a cache that the response will not change during its freshness lifetime. It does not mean “never request this URL again.” Mutable pages need cache lifetimes that fit the desired update delay. RFC 8246 explains that distinction.

Check the cache hit rate and the requests that still reach your origin. Authentication, personalised responses, and frequently changing pages can reduce the opportunity for shared caching. More subscribers will still use bandwidth somewhere; caching helps distribute that work.

More datasets: reduce integration work

A common stream interface makes fetching data reusable. Understanding the data remains part of the job.

Shared vocabularies and application profiles let a consumer recognise familiar structures across publishers. SHACL shapes can help express and validate those expectations. A catalog harvester built around a shared profile has less source-specific work to do than one built around every publisher’s individual response format.

The SEMIC implementation reports show this approach for catalog metadata and cultural heritage. They are useful starting points for deciding what your publishers and consumers should agree on.

What to measure before you grow

Run three realistic scenarios: a consumer starting from nothing, a consumer following current updates, and a consumer returning after an outage. Measure transferred bytes, request counts, processing throughput, destination write time, and the delay until a member becomes useful in the application.

Then increase the pressure you expect to face: more history, a burst of updates, more clients, or another dataset. Watch where the work accumulates and adjust that part of the system.

LDES gives publishers and consumers a shared basis for this work. Good scaling comes from matching that design to the data, the available infrastructure, and the freshness consumers actually need.

For instance, in Flanders, a mobility dataspace on traffic counts was established to address the fragmented nature of traffic measurements. The story goes that one organization that needed access to traffic counts on a road segment decided to re-count traffic on a segment that was already counted by another organization. Setting up the data sharing would have been more tedious than installing rather expensive hardware all over again. By creating an ecosystem where stakeholders agree on data models, discovery methods, usage control policies, and data exchange APIs, the need for redundant data collection is eliminated.

This concept is applicable not only to fast data but also to slow-moving data, such as base registries or personal data sharing (which is outside the scope of this post). Examples include the federation of data portals toward the European data portal, the creation of a pan-European railway infrastructure database (Rojas, 2021), creating an overview of all art-works in European museums with Europeana.

Each dataspace can decide independently how to achieve interoperability and query answering. However, with LDES, we provide a specific approach. As depicted in the figure below, we utilize an architectural style known as Event Sourcing (see Microsoft’s and Amazon’s descriptions about it). This involves describing changes at the source and creating query-able views tailored to consumer needs and usage control policies. For example, an ANPR camera can write raw number plate data to an event source, which a trusted intermediary participant transforms and republishes according to the agreed-upon data model of the traffic measurements dataspace as aggregated and anonymized traffic counts.

Again, the story can be adapted for any other kind of dataset. For example, for data portals, we have created the DCAT-AP Feeds specification that specifies how a data portal can change, so that aggregators like data.europa.eu can build their derived views incrementally knowing what types of changes it can expect.

For each type of dataset there are always intermediary participants that would like to aggregate datasets, transform them into a model that can be used into a certain dataspace, or provide specialized APIs on top of them. Intermediary participants can self-regulate such services based on demand: i.e., if an autocompletion API would not be used anymore, it can be brought offline, still allowing others in the ecosystem to take over certain functionalities. The LDES protocol therefore is focused on enabling intermediary participants, and thus lowering the burden on dataset providers as they do not need to focus on offering all these services on their own.

This requires the participants themselves, at the minimum, to be able to replicate and always stay in-sync with the source of the data. This is why we designed the simplest LDES views to be chronological search trees. A client can decide what time extent it wants to replicate, and can then on each synchronization job always skip to the latest data.

When you look at an LDES page, such as the one of the European Railway Infrastructure dataset, you would not recognize this to be a “streaming” interface. The streaming aspects will only come visible from the moment you use an LDES client. There are LDES client implementations in Java and in TypeScript. Clients will start from the beginning of the stream and incrementally, while time passes, emit all new updates it can find by following links, polling the page, or even subscribing to a pubsub channel. Those updates can then be pipelined using tooling such as Apache Nifi, LDIO, RDF-Connect or Piveau Consus.

The event stream itself is thus being described in the page: it is defined as an append-only log of objects described using a set of RDF triples. Those clients will frequently poll the page based on the calculated expiration based on their caching headers. Based on our earlier research (Van de Vyvere, 2020), polling remains more powerful and cost-efficient for this use case than push-based protocols such as Websockets or Server-Sent Events (SSE) until you want all subscribers to be informed about a change with a latency that is less than ~10 seconds. In that case, publishers can offer a push-based channel on top of their polling interface that also publishes the history of the stream, although this has not yet been needed in use cases we handled thusfar.

Interoperability and queryability within one dataspace are thus ensured by transforming a source dataset into the right data model, and making the right services available through intermediary participants. However, in reality, a dataset is going to be useful for multiple dataspaces, requiring interoperability with multiple ecosystems. If more dataspaces rely on Linked Data and Event Streaming, interoperability across thematic dataspaces would become possible: a data provider could offer the LDES in multiple data models, or intermediary participants can derive multiple views from the same source.

That’s how Linked Data Event Streams (LDES) enables the exchange of interoperable datasets within and across dataspaces by applying event sourcing on Web APIs. Publishing an LDES yourself is light-weight by design to facilitate adoption. The responsibility of knowing where you are in the stream is fully up to the subscribers, which can be done with the open-source LDES clients.

The publishing of an LDES allows data consumers to always stay in sync with a data source. You can watch the video fragment below to learn more about the ecosystem that the event stream provides.

The specification describes the structure of the LDES itself. With that information, you, as a human, can read the event stream. However, what if you do not want to read it yourself, but instead want one of your applications to ingest the events in the Event Stream? The question is then: “How does an application consume events of a Linked Data Event Stream”.

In the following sections of this blog, I will first elaborate on how to read one with some simple example code. Next, I will provide some tools that are available to ingest any kind of LDES.

All code examples and toolings are written in Javascript/Typescript, more specifically Node.js.

Custom code

In this section, I’ll provide some code that can read an LDES.

Local

Below is a simple LDES which contains two members. Each member has a title, a description and the time at its creation. In the triple example:EventStream a ldes:Evenstream ., we see that the URI http://www.example.org/EventStream is of type ldes:EventStream.

From this URI, the members can be reached due to the tree:member property.

@prefix example: <http://www.example.org/>.
@prefix ldes: <http://w3id.org/ldes#>.
@prefix tree: <https://w3id.org/tree#>.
@prefix dct: <http://purl.org/dc/terms/>.
@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.

example:EventStream a ldes:Evenstream;
	tree:member example:member1, example:member2.

example:member1 dct:title "The first member";
	dct:description "The first member of the Linked Data Event Stream";
	dct:created "2023-01-02T10:00:00Z"^^xsd:dateTime.
	
example:member2 dct:title "The second member";
	dct:description "The second member of the Linked Data Event Stream";
	dct:created "2023-01-02T11:00:00Z"^^xsd:dateTime.

Let’s extract the events from this LDES now in code. First, download the following npm packages, which allow you to load this example LDES in memory as a N3 Store.

# Some packages to install
npm i rdf-store-stream rdf-parse streamify-string

Using the above libraries, the following piece of code parses the LDES into a Store.

// imports to read the LDES in an N3 Store
const { storeStream } = require("rdf-store-stream");
const rdfParser = require("rdf-parse").default;
const streamifyString = require('streamify-string');

// the LDES example as a string
const ldesText = `@prefix example: <http://www.example.org/>.
@prefix ldes: <http://w3id.org/ldes#>.
@prefix tree: <https://w3id.org/tree#>.
@prefix dct: <http://purl.org/dc/terms/>.
@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.

example:EventStream a ldes:Evenstream;
	tree:member example:member1, example:member2.

example:member1 dct:title "The first member";
	dct:description "The first member of the Linked Data Event Stream";
	dct:created "2023-01-02T10:00:00Z"^^xsd:dateTime.
	
example:member2 dct:title "The second member";
	dct:description "The second member of the Linked Data Event Stream";
	dct:created "2023-01-02T11:00:00Z"^^xsd:dateTime.`;

// parse the LDES RDF into an N3 Store 
const textStream = streamifyString(ldesText);
const quadStream = rdfParser.parse(textStream, { contentType: 'text/turtle' });
const ldesStore = await storeStream(quadStream);

Okay great! Now let us read the members of the LDES.

When the LDES URI is not known, we can query the RDF graph to look for the ldes:EventStream class.

const LDESSubjects = ldesStore.getSubjects("http://www.w3.org/1999/02/22-rdf-syntax-ns#type", "https://w3id.org/ldes#EventStream");

However, we know the URI already, so we can use that to extract all the metadata of the LDES itself.

// extract the LDES metadata
const metadataQuads = ldesStore.getQuads("http://www.example.org/EventStream");

for (const quad of metadataQuads) {
  console.log(quad);
}

// extract the members of the LDES
const memberSubjects = ldesStore.getObjects("http://www.example.org/EventStream", "https://w3id.org/tree#member")
for (const membersubject of memberSubjects) {
  const memberQuads = ldesStore.getQuads(membersubject);
  console.log(membersubject.value);
  console.log(memberQuads)
}

Online

In this part, we are going to work with an LDES which is hosted on the web. With this part, I want to show that the above custom code to retrieve all members in the LDES will work as long as it is valid RDF and adheres to the structure of an LDES and that it does not matter where it is hosted.

Let’s continue with our example.

This means the first thing that has to be done is to host this LDES on an HTTP server.

Install an HTTP server

npm i http-server

Set up the HTTP server

To set up the server, execute the following following command:

npx http-server

The above command starts an HTTP server and hosts all the files from the directory where the command has been run at URL http://localhost:8080/.

Saving the example LDES as example-LDES.ttl and storing it in that directory results in our own hosted LDES at URL http://localhost:8080/example-LDES.ttl.

// send an HTTP fetch request to the hosted example LDES URL
const response = await fetch("http://localhost:8080/example-LDES.ttl")

// convert its body to string
const ldesText = await response.text()

// parse the LDES RDF into an N3 Store 
const textStream = streamifyString(ldesText);
const quadStream = rdfParser.parse(textStream, { contentType: 'text/turtle' });
const ldesStore = await storeStream(quadStream);

And at this point, we can interact with the LDES in the same way that was explained in the first part.

Limitations

When the LDES is hosted as one page, the custom code will work fine to consume all the events in the event stream. However, when the LDES is fragmented over multiple pages (using TREE hypermedia), the simple approach is not sufficient anymore.

I could explain how to extract a tree:view from the LDES and how to dereference the relations to retrieve all the members. Though, I would not want to reinvent the wheel as there are already libraries that offer this kind of functionality: the LDES client.

LDES client

The LDES client is a prototype (developed by Brecht Van de Vyvere) and is used to retrieve all the members of an LDES by passing an URL. This library parses the metadata of the LDES and (when present) dereferences all the relations of the view. Thus it abstracts away the technicalities of the LDES structure (and the TREE hypermedia) so that end users can focus on consuming the members in a streaming way.

Let’s use it! First, we have to install the package @treecg/actor-init-ldes-client

npm i @treecg/actor-init-ldes-client

After installation, we can use the code below to retrieve the members of an LDES. The LDES client only deals with LDESs on the web. So for simplicity’s sake, I re-used the LDES we hosted ourselves.

We start by creating an engine. Then we invoke the createReadStream function with an url and several options. Here we ask the engine to disable synchronization and as a representation, we want Quads, which actually will give us members following the Member Interface. Disabling synchronization means that once all relations are followed and all members are read, the engine is done and closes. When enabled, the engine acts a service and would (depending on the pollingInterval) fetch certain pages to look for new members in the LDES.

It is important to note that this will generate a stream of members. To consume any members of the LDES, we add a listener (data) so we can do something (in the example below just logging) with a member.

const newEngine = require('@treecg/actor-init-ldes-client').newEngine;

let url = "http://localhost:8080/example-LDES.ttl"
const LDESClient = new newEngine();
const eventstream = LDESClient.createReadStream(url, {disableSynchronization: true, representation: "Quads"});

eventstream.on('data', (member) => {
  console.log(member.id.value);
  console.log(member.quads);
})

eventstream.on('metadata', (metadata) => {
  console.log(metadata)
})

eventstream.on('end', () => {
  console.log("No more data!");

Furthermore, we also log the metadata. In the custom code, we’ve received an array of Quads. With the LDES client now, we see that it contains the url of the entry point of the engine and an object that contains treeMetadata with collections, nodes and relations.

This object is generated by the function extractMetadata @treecg/tree-metadata-extraction package.

TREE metadata extractor

The TREE metadata extractor does exactly what its name stands for: extracting TREE hypermedia (and by extension LDES) metadata from RDF data.

It is very simple to use, the only thing we have to pass to its only method, extractMetadata, is an array of Quads.

The resulting output is an object which contains a Map of collection, node and relation.

Let’s use this tool to read all the members from the non-fragmented example LDES.

npm i @treecg/tree-metadata-extraction

const extractMetadata = require('@treecg/tree-metadata-extraction').extractMetadata;
const metadata =  await extractMetadata(ldesStore.getQuads());

const collection = metadata.collections.get("http://www.example.org/EventStream");
for (const member of collection.member) {
    const memberID = member['@id'];
    console.log(ldesStore.getQuads(memberID));
}

The extractor creates a Map for all collections, nodes and relations it can find in the input quads. In our input, there are no relations and nodes and only one collection. So we can get this collection out of the metadata to retrieve all the member identifiers. And finally, we use the member identifiers to fetch the member triples from the RDF data.

Data publishers are too often asked to pick between two unsatisfying options. The data dump looks harmless: publish a full export and let consumers do their thing. In practice it creates a replication hell: multiple uncoordinated copies drift out of sync; consumers juggle deltas and snapshots; provenance cannot be traced; history gets lost in overwrites; and every “fresh” download quietly rebuilds the same indexes in a hundred places.

Take for example the address registry in Flanders for which data dumps are available. Probably every municipality in Flanders takes a copy of this file for use cases such as autocompleting the street names for the forms they use everywhere. When developers make such integrations, synchronization is an afterthought: this dataset doesn’t change that often anyway, right? Think again: there are minor changes every day, with new addresses coming into play and old ones becoming “historized”. When in 2016, 2019 and 2025, cities in Belgium decided to fuse, a lot of street names also needed to be renamed to avoid duplicate names. Instead of this base registry being updated from the source, services started making the changes they needed manually in their local copies, leading to a replication hell.

On the other side sits the querying API. It promises precision — ask only for what you need — while quietly enrolling the publisher in maintenance hell. New use case? New endpoint. New query language that became popular? Again yet another API to be provided. The provider becomes an involuntary platform operator, while consumers still only have a limited processing possibility of the dataset as they can only query the API in the ways the data provider managed to set up. Each endpoint that has been set up comes with its own maintenance cost. After a while, when priorities shift or budgets shrink, it will have become impossible to turn off any of the existing APIs as there may still be an application that relies on it. The budget that once was used for innovation and creating a better public service, is now being used for maintaining legacy APIs.

Take again the example of the address registry in Flanders for which, next to data dumps, also a plethora of API products are available. Specific functionalities that were once brought online need to remain maintained: there may always be that one service that is still relying on this API. Certainly for address registries, multiple functionalities of the dataset are expected, such as: finding the geolocation of an (or multiple) address(es), a geospatial interface to visualize the data, a historic view of what addresses existed in the past until today, an autocompletion interface for streetnames, municipalities and addresses, a SPARQL, GQL and GraphQL API for graph-based access, a specific service that allows calculating what addresses will be impacted by a road closure, etc. It doesn’t matter how many APIs you have: it will never be sufficient — there’s always going to be that other person who needs a functionality that does not yet exist.

There are, however, other paths. Let’s start from the idea of taking full copies, i.e. “dumps”, but let’s change the intent and the name. Let’s call it a stream. This way, it sets the expectation that developers of consumption pipelines will code for history and future: they will re-interpret what happened and, with exactly the same code, stay in sync with what will happen. This is the mindset behind Linked Data Event Streams (LDES). The ambition level is to introduce semantic interoperability through Linked Data, and combine this idea with how developers interact with streams. LDES publishes the dataset as an append-only sequence of immutable members with stable identifiers, so any party can replicate once and then follow updates. Hence the straightforward name: the combination of having this ambition towards interoperability, and the ambition to always keep every copy up to date, becomes Linked Data + Event Streams: LDES in short.

This shift unlocks governance opportunities. With an authoritative event source online, the publisher can decide which higher-level interfaces to keep maintaining, and which to let the ecosystem carry. A SPARQL endpoint, an OGC API, or a GraphQL service may be useful today and optional tomorrow. If a GraphQL API stops aligning with the publisher’s priorities, the publisher can bring it offline, while the consumer that still needs it can spin up their own GraphQL server that replicates and synchronises from the event source, preserving functionality without forcing the publisher to keep every interface alive forever. If you maintain a base registry or any dataset that changes over time, start by publishing the LDES at the event source. Everything else can — and should — derive from there.

For the technical details, see the LDES specification at https://w3id.org/ldes/specification. Various implementations of clients and servers are available. In order to interact with the community, visit https://ldes.tech.

The talk at ENDORSE 2021

At ENDORSE 2021 there was a talk in which I explain the contents of this blog post:

P.S.

This post was planned to be published in 2021, but remained in draft until 2025. It was only in 2025 that there was time to finalize it, keeping the initial planned publication date.