Operational Data Store

For Západoslovenská distribučná we built an Operational Data Store (ODS) — a centralized data layer that consolidates operational data from several internal systems and provides other applications with a current, unified and reliable view of it. We built the entire solution exclusively on open source technologies, with no dependency on any specific vendor’s licenses.

The brief

Before ODS was deployed, the operational data of ZSDIS was scattered across several internal systems and each application obtained it on its own. The result was inconsistent and often outdated views of the same data — there was no single reliable source of truth. At the same time, every new solution had to integrate the individual source systems anew, which needlessly prolonged development.

An architecture built on open source

At the heart of the system is a PostgreSQL database complemented by an extensive set of ETL processes that, during processing, obtain data from the internal systems of ZSDIS. The source systems include, among others:

• SAP systems,
• the consolidated metering centre,
• the GIS system,
• the balancing system,
• and a number of other sources.

The choice of open source technologies gives the customer full control over the solution and no vendor lock-in.

ETL processes that respect dependencies

The most complex part of the delivery are the ETL processes, which transform the input data into target structures so that they are easy to use for downstream systems. The data is interlinked — the output of one ETL process often feeds into the next.

To handle this interdependence, we designed a hierarchical system that respects the dependencies between data and schedules the individual processes optimally. As a result, the input data reaches the ODS in the shortest possible time.

Reactive real-time processing

The entire processing is reactive. The processes do not wait for predetermined points in time but start in real time as a reaction to incoming input data. The data is therefore processed immediately after it becomes available.

Making data available

We make the processed data available to the internal systems of ZSDIS according to their needs through:

• a set of microservices,
• Apache Kafka,
• RabbitMQ.

Customer benefits

• A single source of truth — applications draw from one consistent and reliable data layer instead of scattered silos.
• Real-time data — the reactive ETL system processes data immediately, so consumers have up-to-date information available in real time.
• Relief for source systems — applications query the ODS instead of directly querying production systems (SAP and others), which they therefore do not burden.
• Faster development of new solutions — new applications, such as the sanctions screening, build on a ready-made data layer instead of re-integrating each system separately.
• No vendor lock-in — the ODS is built exclusively on open source technologies, giving the customer full control over the solution.