Open Data Platform

An open, developer-first data platform that combines orchestration, lakehouse processing, governance, BI, and observability in one stack.

Explore Docs Quick Start

Core Capabilities

Medallion Architecture

Ingest and transform data through Bronze, Silver, and Gold layers with both Spark-compatible and Postgres/dbt transformation paths.

Full Observability

Integrated Prometheus metrics, Grafana dashboards, Loki logs, and Tempo traces with OpenTelemetry instrumentation.

Deploy Anywhere

Run locally with Docker Compose, scale with Kubernetes (kind), or deploy to Azure AKS and Scaleway with Terraform.

Architecture

Three planes — Operator, Control, and Data — connected through a React launchpad that links all platform surfaces.

flowchart LR
  subgraph userSurface["User Surface"]
    browser["Browser"] --> portal["Frontend Portal :3000"]
  end
  subgraph orchestration["Orchestration"]
    airflowUI["Airflow UI :8080"] --> airflowSched["Airflow Scheduler"]
    airflowSched --> pipelines["Pipelines"]
  end
  subgraph storageData["Storage and Data"]
    minio["MinIO S3 :9000"]
    lakehouse["Lakehouse Bronze/Silver/Gold"]
    warehousePg["Warehouse Postgres"]
  end
  subgraph analytics["Analytics"]
    superset["Superset :8088"]
  end
  subgraph governance["Governance"]
    datahubUI["DataHub UI :9002"]
    datahubGMS["DataHub GMS"]
  end
  subgraph observability["Observability"]
    prometheus["Prometheus :9090"]
    grafana["Grafana :3001"]
    loki["Loki :3100"]
    tempo["Tempo :3200"]
  end
  portal --> airflowUI
  portal --> datahubUI
  portal --> superset
  portal --> grafana
  airflowSched --> minio
  pipelines --> lakehouse
  lakehouse --> warehousePg
  warehousePg --> superset
  airflowSched --> datahubGMS
  pipelines --> prometheus
  grafana --> prometheus
  grafana --> loki
  grafana --> tempo

Technology Stack

Built on battle-tested open source technologies.

Quick Start

Get the full platform running locally in three commands.

terminal

# Clone and configure
$ cp .env.template .env

# Bootstrap the full stack
$ ./scripts/platform/bootstrap_all.sh --auto-fill-env

# Open the platform launchpad
$ open http://localhost:3000

See the full Development Guide for prerequisites and advanced setup.

Documentation

Everything you need to understand, deploy, and extend the platform.

Open Data Platform

Core Capabilities

Medallion Architecture

Full Observability

Deploy Anywhere

Architecture

Technology Stack

Quick Start

Documentation

Architecture

Data Model

Deployment

Development

Ingestion Guide

Governance

CI/CD

Testing