data-engineering-infrastructure

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

Code examples showing flow deployment to various types of infrastructure

Deploy a Prefect flow to serverless AWS Lambda function

Get started with Prefect by scheduling your Prefect flows with GitHub Actions

Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS

Data engineering project using Azure Data Factory and Apache Airflow

This is a repo designed to show the workflow of a data contract, with pre-commit hooks and GitHub Actions on how to have the contract power a data platform

Standardizes fragmented multi-strategy analytics outputs into a schema-contracted, partitioned Parquet lake with SLO-style quality checks, partition-pruned DuckDB queries, and an incident runbook.

Blog space on data engineering, machine learning, platform engineering.

Python package that simplifies the creation of AWS infrastructure for simulating real-time data streaming and batch processing, ideal for integrating into machine learning projects.

This project implements a real-time event streaming pipeline for a music streaming service, inspired by Spotify Wrapped and Billboard charts. The pipeline is powered by Apache Airflow, Apache Kafka, dbt, Docker, GCP, Spark-Streaming, and Terraform.

Local-first multimodal data engineering platform for dataset processing, curation, transformation, retrieval, RAG, and reproducible AI workflows.