Skip to content
Back to work

Digital Platforms · Logistics

Real-Time Data Streaming Platform

For a logistics operator, an event streaming platform that puts vehicle, depot and order events in front of dispatch while they can still be acted on.

Built on
Kafka, Python, AWS, Docker
Industry
Logistics
Project type
Digital Platforms
Primary service
Technology Solutions
Scope
Platform architecture, streaming infrastructure, schema governance, observability.

Repository not shown as per company policy

01Challenge

Batch reporting is fine for accounts and useless for dispatch.

  • News arrived a day late

    Exceptions surfaced in the next morning’s report, long after the customer had noticed.

  • Point-to-point integrations

    Every new consumer meant another direct connection into the systems producing the data.

  • Recovery was manual

    When a job failed, rebuilding the missing period meant stitching data together by hand.

02Approach

Separate the backbone from the consumers.

  • Events, not extracts

    Source systems publish what happened as it happens, rather than being queried on a schedule.

  • Schema governance

    Schemas are versioned and checked at ingestion, so producers and consumers can change independently.

  • Infrastructure as code

    Every environment is defined in code, so staging behaves like production.

03Outcome

Publish once, let every team read at its own pace.

Dispatch needed to know about a slipping delivery while it could still be rescued. Overnight batch reporting told them the next morning.

  • An event backbone

    Vehicles, depots and the order system publish events once; dispatch, reporting and alerting each consume them independently.

  • Contracts at the boundary

    Every event is validated against a versioned schema on arrival, so a malformed message cannot break the consumers downstream.

  • Replay as a feature

    Retained history means recovery and backfill are a replay, not a reconstruction from logs.

What changed

  • Dispatch sees delivery exceptions as they happen, not in the next morning’s report.
  • New consumers subscribe to the event stream without changes to the systems that produce events.
  • Recovery and backfill are a replay from retained history rather than a manual reconstruction.

How it works

One stream, several consumers.

retained historyVehicle telemetryLocation, statusDepot systemsScans, loadsOrder systemBookings, changesStream processingRolling operationalaggregatesLive dispatch viewExceptions as they happenAlertingLate and at-risk deliveriesWarehouse sinkReporting and history, with replayfrom retained topicsEvent backboneOne stream, several consumersKafka topicsRetained historySchema validationVersioned contracts, checked at the boundary

One run, step by step.

01 / 05

Inside the system

What it does.

  • Event schemas versioned and validated at ingestion
  • Stream processing for rolling operational aggregates
  • Replay from retained history for recovery and backfill
  • Live dispatch view alongside a warehouse sink
  • Infrastructure defined as code across environments

Built with

What it runs on.

Streaming
KafkaSchema validationStream processing
Services
PythonDocker
Cloud
AWSInfrastructure as code
Operations
Monitoring and alertingReplay and backfill
Services appliedTechnology SolutionsData & AnalyticsDevOps & MLOps

Dispatch hears about a problem while it can still be fixed.

More work

Related projects.

View all work
AI & AutomationHealthcare

Autonomous AI Log Monitoring & Observability Platform

For a healthcare media and clinician engagement platform, a five-agent system that reads a production error, writes the fix and opens a reviewed pull request, with an engineer still deciding what ships.

Data & AnalyticsMedia & Publishing

Data & BI Modernization

For an established independent book publisher, a decade of accumulated ETL, warehouse and dashboard tools replaced with one governed cloud platform, without breaking a single number the business depended on.

AI & AutomationHealthcare

Drug Competitor Identification

A brand-intelligence tool that asks a language model who a drug competes with, then checks the answer against regulatory reference data before anyone is asked to trust it.

Discuss a similar project.

If something here is close to what you need, tell us about your situation and we will walk you through how we would approach it.

Discuss a Similar Projectcontact@genclover.com

Intelligence → Innovation → Automation → Growth