← Courses Catalogue

Data Engineering Zoomcamp

▸ Docker Terraform

▸ Docker Sql

MIntroduction to Docker MVirtual Environments and Data Pipelines MDockerizing the Pipeline MRunning PostgreSQL with Docker MNY Taxi Dataset and Data Ingestion MCreating the Data Ingestion Script MpgAdmin - Database Management Tool MDockerizing the Ingestion Script MDocker Compose MSQL Refresher MCleanup

▸ Images

▸ Pipeline

{ }Docker Compose

▸ Docker Helper Scripts

{ }Docker Ingest { }Docker Pgadmin { }Docker Postgres

{ }Dockerfile { }Ingest Data { }Pyproject

▸ Images

▸ Terraform

MTerraform Overview MRefresh token/session, and verify authentication

▸ Terraform

▸ Terraform Basic

▸ Terraform With Variable AWS

{ }Main { }Variables

▸ Terraform With Variables

{ }Main { }Variables

▸ Workflow Orchestration

{ }Docker Compose

▸ Flows

{ }Hello World { }Python { }Getting Started Data Pipeline { }Postgres Taxi { }Postgres Taxi Scheduled { }Gcp Kv { }Gcp Setup { }Gcp Taxi { }Gcp Taxi Scheduled { }Chat Without Rag { }Chat With Rag

▸ Images

▸ Data Warehouse

{ }Big Query Hw { }Big Query Ml { }Big Query MExtract Model

▸ Extras

{ }Gitignore { }Pyproject { }Web To Gcs With Progress Bar { }Web To Gcs

▸ Images

▸ Analytics Engineering

▸ Class Notes

MDE Zoomcamp 4.1.1 — Analytics Engineering Basics MDE Zoomcamp 4.1.2 — What is dbt? MDE Zoomcamp 4.2.1 — dbt Core vs dbt Cloud MDE Zoomcamp 4.3.1 — dbt Project Structure MDE Zoomcamp 4.3.2 — dbt Sources MDE Zoomcamp 4.4.1 — dbt Models MDE Zoomcamp 4.4.2 — dbt Seeds and Macros MDE Zoomcamp 4.5.1 — Documentation MDE Zoomcamp 4.5.2 — dbt Tests MDE Zoomcamp 4.5.3 — dbt Packages MDE Zoomcamp 4.6.1 — dbt Commands

▸ Images

▸ Refreshers

▸ Setup

MCloud Setup Guide MTroubleshooting DuckDB Out of Memory Errors MLocal Setup Guide

▸ Taxi Rides Ny

{ }Gitignore { }Dbt Project

▸ Macros

{ }Get Trip Duration Minutes { }Get Vendor Data { }Macros Properties { }Safe Cast

▸ Models

▸ Intermediate

{ }Int Trips Unioned { }Int Trips { }Schema

▸ Marts

{ }Dim Vendors { }Dim Zones { }Fct Trips

▸ Reporting

{ }Fct Monthly Zone Revenue { }Schema

▸ Staging

{ }Schema { }Sources { }Stg Green Tripdata { }Stg Yellow Tripdata

{ }Package Lock { }Packages

▸ Seeds

▦Payment Type Lookup { }Seeds Properties ▦Taxi Zone Lookup

▸ Data Platforms

▸ Images

▸ Notes

M5.1 - Introduction to Bruin M5.2 - Getting Started with Bruin M5.3 - Building an End-to-End Pipeline with NYC Taxi Data M5.4 - Using Bruin MCP with AI Agents M5.5 - Deploying to Bruin Cloud M5.6 - Core Concepts: Projects M5.6 - Core Concepts: Pipelines M5.6 - Core Concepts: Assets M5.6 - Core Concepts: Variables M5.6 - Core Concepts: Commands

▸ Batch

▸ Code

NTest NPyspark NTaxi Schema { }Spark Sql Big Query NSpark Sql { }Spark Sql NGroupby Join NRdds NSpark Gcs Mfor newer versions of spark use that: { }Download Data NHomework

▸ Images

▸ Setup

▸ Config

MHadoop Yarn MLinux MMacos MWindows

▸ Streaming

▸ Extras

▸ Ksqldb

▸ Pyflink

{ }Gitignore { }Docker Compose MHomework { }LICENSE { }Makefile { }Requirements

▸ Src

▸ Job

{ }Aggregation Job { }Start Job { }Taxi Job

▸ Producers

{ }Load Taxi Data { }Producer

▸ Python

▸ Avro Example

{ }Consumer { }Producer { }Ride Record Key { }Ride Record { }Settings

▸ Docker

{ }Docker Compose

▸ Kafka

{ }Docker Compose

▸ Spark

{ }Build { }Docker Compose

▸ Json Example

{ }Consumer { }Producer { }Ride { }Settings

▸ Redpanda Example

{ }Consumer { }Docker Compose { }Producer { }Ride { }Settings

{ }Requirements

▸ Resources

▸ Schemas

▸ Streams Example

▸ Faust

{ }Branch Price { }Producer Taxi Json { }Stream Count Vendor Trips { }Stream { }Taxi Rides { }Windowing

▸ Pyspark

{ }Consumer { }Producer { }Settings { }Spark Submit NStreaming Notebook { }Streaming

▸ Redpanda

{ }Consumer { }Docker Compose { }Producer { }Settings { }Spark Submit NStreaming Notebook { }Streaming

▸ Theory

▸ Java

▸ Kafka Examples

{ }Gitignore { }Build

▸ Gradle

▸ Wrapper

▸ Src

▸ Main

▸ Avro

▸ Java

▸ Org

▸ Example

{ }AvroProducer

▸ Customserdes

{ }CustomSerdes

▸ Data

{ }PickupLocation { }Ride { }VendorInfo

{ }JsonConsumer { }JsonKStream { }JsonKStreamJoins { }JsonKStreamWindow { }JsonProducer { }JsonProducerPickupLocation { }Secrets { }Topics

▸ Resources

▸ Test

▸ Java

▸ Org

▸ Example

▸ Helper

{ }DataGeneratorHelper

{ }JsonKStreamJoinsTest { }JsonKStreamTest

▸ Workshop

{ }Docker Compose { }Flink Config

▸ Live

{ }Gitignore { }Docker Compose { }Flink Config { }Main

▸ Notebooks

NConsumer Db { }Models NProducer

{ }Pyproject.flink { }Pyproject

▸ Src

▸ Job

{ }Aggregation Job { }Pass Through Job

▸ Producers

{ }Models { }Producer Realtime

{ }Makefile { }Pyproject.flink { }Pyproject

▸ Src

▸ Consumers

{ }Consumer Postgres { }Consumer

▸ Job

{ }Aggregation Job Demo { }Aggregation Job { }Pass Through Job

▸ Producers

{ }Producer Realtime { }Producer

{ }Gitignore MAfter Sign Up MAwesome Data Engineering MCertificates

▸ Cohorts

▸ 2022

▸ Week 1 Basics N Setup

▸ Week 2 Data Ingestion

▸ Airflow

MSetup Official MSetup Nofrills

▸ Dags

{ }Data Ingestion Gcs Dag

▸ Dags Local

{ }Data Ingestion Local { }Ingest Script

{ }Docker Compose 2.3.4 { }Docker Compose Nofrills { }Docker Compose { }Dockerfile

▸ Docs

▸ Extras

{ }Data Ingestion Gcs Dag Ex2 { }Web To Gcs

{ }Requirements

▸ Scripts

▸ Homework

MHomework { }Solution

▸ Transfer Service

▸ Week 3 Data Warehouse

▸ Airflow

MSetup Official MSetup Nofrills

▸ Dags

{ }Gcs To Bq Dag

{ }Docker Compose Nofrills { }Docker Compose

▸ Docs

▸ Scripts

▸ Week 5 Batch Processing

▸ Week 6 Stream Processing

▸ 2023

MLeaderboard MProject

▸ Week 1 Docker Sql

MPrepare Postgres

▸ Week 1 Terraform

▸ Week 2 Workflow Orchestration

▸ Week 3 Data Warehouse

▸ Week 4 Analytics Engineering

▸ Week 5 Batch Processing

▸ Week 6 Stream Processing

MHomework { }Producer Confluent { }Settings { }Spark Submit { }Streaming Confluent

▸ Workshops

▸ 2024

▸ Docker Terraform

MPrepare Postgres MSolutions

▸ Workflow Orchestration

▸ Images

▸ Data Warehouse

▸ Analytics Engineering

▸ Batch

▸ Streaming

{ }Docker Compose MHomework

MLeaderboard MProject

▸ Workshops

▸ Dlt Resources

MIntro NHomework Solution NHomework Starter NWorkshop

MData ingestion with dlt

▸ Images

▸ 2025

▸ Docker Terraform

MModule 1 Homework: Docker & SQL

▸ Workflow Orchestration

▸ Flows

{ }Getting Started Data Pipeline { }Postgres Taxi Scheduled { }Postgres Taxi { }Postgres Dbt { }Gcp Kv { }Gcp Setup { }Gcp Taxi Scheduled { }Gcp Taxi { }Gcp Dbt

▸ Images

▸ Data Warehouse

NDLT Upload To GCP MHomework { }Load Yellow Taxi Data

▸ Analytics Engineering

▸ Batch

MModule 5 Homework

▸ Streaming

▸ Homework

▸ Workshops

▸ Dlt

MData ingestion with dlt MWorkshop "Data Ingestion with dlt": Homework

▸ Images

▸ Img

{ }Dynamic Load Dlt

▸ 2026

▸ Docker Terraform

MModule 1 Homework: Docker & SQL

▸ Workflow Orchestration

▸ Data Warehouse

NDLT Upload To GCP MModule 3 Homework: Data Warehousing & BigQuery { }Load Yellow Taxi Data

▸ Analytics Engineering

MModule 4 Homework: Analytics Engineering with dbt

▸ Data Platforms

MModule 5 Homework: Data Platforms with Bruin

▸ Batch

MModule 6 Homework

▸ Streaming

▸ Workshops

▸ Dlt

{ }Analysis MHomework: Build Your Own dlt Pipeline NDlt Pipeline Overview

▸ Images

{ }Open Library Pipeline { }Pyproject

MFrom APIs to Warehouses: AI-Assisted Data Ingestion with dlt

▸ Images

▸ Architecture

▸ Aws

MLearning in public

▸ Projects

▸ Scripts

{ }Generate Thumbnails { }Pyproject

MWorkshop Best Practices

Built with LearnaOS

← Courses Catalogue

section

Avro Example

Ride Record Key