High-Speed Analytics Lakehouse & ETL Pipeline
Construct a columnar data warehouse ingestion pipeline with automated schema migrations.
Problem Statement
Operational databases degrade when handling large analytical aggregations. You will design a change-data-capture (CDC) pipeline that transforms relational writes into columnar ClickHouse tables for sub-second analytical queries.
What You'll Build
An automated data ingestion engine that syncs transactional records into an analytical lakehouse and provides high-speed aggregation endpoints for executive business dashboards.
Technology Stack
Skills You'll Master
Requirements & Prerequisites
- Comfort with complex SQL queries (Window functions, CTEs).
- Docker for running PostgreSQL and ClickHouse containers.
Guided Milestones (3)
Step-by-Step ArchitectureTransactional Seed & CDC Ingestion
Generate 1,000,000 synthetic transaction records and stream changes via CDC.
- Data generator script
- CDC pipeline worker
ClickHouse Columnar Schema Design
Design MergeTree tables with partition keys optimized for temporal analysis.
Analytics API & Dashboard Service
Expose sub-50ms aggregation endpoints for time-series charts.
Project Submission & Verification
Submit your public GitHub repository and live deployment link for automated test checks.
Add to ASCI Portfolio
Demonstrates practical data platform engineering and columnar storage optimization.