# Start Data Engineering

> A newsletter with tutorials, data design patterns, open-source tools, and techniques used by data-driven companies to help you become a better data engineer.

## Pages

- [Writing memory efficient data pipelines in Python](https://www.startdataengineering.com/post/writing-memory-efficient-dps-in-python/index.llms.md)
- [How to Prevent Missing Data With Referential Integrity Checks](https://www.startdataengineering.com/post/why-referential-integrity-matters/index.llms.md)
- [What is an Open Table Format? & Why to use one?](https://www.startdataengineering.com/post/what_why_table_format/index.llms.md)
- [What is a Data Warehouse?](https://www.startdataengineering.com/post/what-is-a-data-warehouse/index.llms.md)
- [How to decide on a data project for your portfolio](https://www.startdataengineering.com/post/what-data-project-to-build/index.llms.md)
- [How to become a valuable data engineer](https://www.startdataengineering.com/post/valuable-de-guide/index.llms.md)
- [What are Common Table Expressions(CTEs) and when to use them?](https://www.startdataengineering.com/post/using-common-table-expression-in-redshift/index.llms.md)
- [Uplevel your dbt workflow with these tools and techniques](https://www.startdataengineering.com/post/uplevel-dbt-workflow/index.llms.md)
- [What are the types of data quality checks?](https://www.startdataengineering.com/post/types-of-dq-checks/index.llms.md)
- [How to test PySpark code with pytest](https://www.startdataengineering.com/post/test-pyspark/index.llms.md)
- [How to Use Spark SQL Merge Into - Step-by-Step Tutorial](https://www.startdataengineering.com/post/spark-sql-merge-into/index.llms.md)
- [What do Snowflake, Databricks, Redshift, BigQuery actually do?](https://www.startdataengineering.com/post/sf-v-dbx/index.llms.md)
- [What is a self-serve data platform & how to build one](https://www.startdataengineering.com/post/self-serve-data-platform/index.llms.md)
- [Review: Building a Real Time Data Warehouse](https://www.startdataengineering.com/post/review-building-a-real-time-data-warehouse/index.llms.md)
- [How to turn a 1000-line messy SQL into a modular, & easy-to-maintain data pipeline?](https://www.startdataengineering.com/post/quick-scalable-business-value-pipeline/index.llms.md)
- [Should Data Pipelines in Python be Function based or Object-Oriented (OOP)?](https://www.startdataengineering.com/post/python-fp-v-oop/index.llms.md)
- [How to Write Integration Tests for Python Data Pipelines](https://www.startdataengineering.com/post/python-datapipeline-integration-test/index.llms.md)
- [Free 10-Minute Polars Tutorial for Data Engineers](https://www.startdataengineering.com/post/polars-tutorial-data-engineer/index.llms.md)
- [What are the Key Parts of Data Engineering?](https://www.startdataengineering.com/post/parts-of-dataengineering/index.llms.md)
- [5 Steps to land a high paying data engineering job](https://www.startdataengineering.com/post/n-steps-high-pay-de-job/index.llms.md)
- [25 SQL tips to level up your data engineering skills](https://www.startdataengineering.com/post/n-sql-tips-de/index.llms.md)
- [6 Responsibilities of a Data Engineer](https://www.startdataengineering.com/post/n-job-reponsibilities-of-a-data-engineer/index.llms.md)
- [Setting up a local development environment for python data projects using Docker](https://www.startdataengineering.com/post/local-dev/index.llms.md)
- [How to set up CI/CD for data infrastructure](https://www.startdataengineering.com/post/infrastructure-cicd-data-engineering/index.llms.md)
- [How to improve at SQL as a data engineer](https://www.startdataengineering.com/post/improve-sql-skills-de/index.llms.md)
- [IaC (Infrastructure-as-Code) fundamentals for data engineers](https://www.startdataengineering.com/post/iac-for-data-engineering-terraform/index.llms.md)
- [How to unit test sql transforms in dbt](https://www.startdataengineering.com/post/how-to-test-sql-using-dbt/index.llms.md)
- [Scheduling a SQL script, using Apache Airflow, with an example](https://www.startdataengineering.com/post/how-to-schedule-a-sql-script-using-apache-airflow-with-an-example/index.llms.md)
- [How to Manage Upstream Schema Changes in Data Driven Fast Moving Company](https://www.startdataengineering.com/post/how-to-manage-upstream-schema-changes-in-data-driven-fast-moving-company/index.llms.md)
- [How to Get Really Good at Advanced SQL for Data Engineering](https://www.startdataengineering.com/post/how-to-get-better-at-advanced-sql/index.llms.md)
- [How to Extract Data from APIs for Data Pipelines using Python](https://www.startdataengineering.com/post/how-to-extract-data-from-api-for-data-pipelines/index.llms.md)
- [How to Backfill a SQL query using Apache Airflow](https://www.startdataengineering.com/post/how-to-backfill-sql-query-using-apache-airflow/index.llms.md)
- [Free Python Standard Library How-to Cheatsheet for Data Engineers](https://www.startdataengineering.com/post/free-python-library-usage-cheatsheet-for-des/index.llms.md)
- [Six Data Modeling Techniques For Building Production-Ready Tables Fast](https://www.startdataengineering.com/post/fast-consistent-data-model/index.llms.md)
- [Whats the difference between ETL & ELT?](https://www.startdataengineering.com/post/elt-vs-etl/index.llms.md)
- [Data Pipeline Design Patterns - #1. Data flow patterns](https://www.startdataengineering.com/post/design-patterns/index.llms.md)
- [How to quickly deliver data to business users? #1. Adv Data types & Schema evolution](https://www.startdataengineering.com/post/deliver-data-quickly-with-schema-evolution-and-adv-data-types/index.llms.md)
- [Data Engineering Interview Preparation Series #2: System Design](https://www.startdataengineering.com/post/de_interview_sd/index.llms.md)
- [Data Engineering Best Practices - #2. Metadata & Logging](https://www.startdataengineering.com/post/de_best_practices_log/index.llms.md)
- [How to build a data project with step-by-step instructions](https://www.startdataengineering.com/post/de-proj-step-by-step/index.llms.md)
- [3 Data Storage Techniques Every Data Engineer Should Know](https://www.startdataengineering.com/post/data-storage-pattern-rule-of-thumb/index.llms.md)
- [Data Pipeline Design for Fact & Dimension Tables](https://www.startdataengineering.com/post/data-pipeline-design-facts-dimensions/index.llms.md)
- [How to Ingest Data: 2 Essential Patterns](https://www.startdataengineering.com/post/data-loading-patterns/index.llms.md)
- [6 Data Engineering Skills To Progress in the Age of AI](https://www.startdataengineering.com/post/data-engineering-skills-in-age-of-ai/index.llms.md)
- [Build Data Engineering Projects, with Free Template](https://www.startdataengineering.com/post/data-engineering-projects-with-free-template/index.llms.md)
- [Data Engineering Project: Stream Edition](https://www.startdataengineering.com/post/data-engineering-project-for-beginners-stream-edition/index.llms.md)
- [End-to-end data engineering project - batch edition](https://www.startdataengineering.com/post/data-engineering-project-e2e/index.llms.md)
- [How to Transition from Data Analyst to Data Engineer](https://www.startdataengineering.com/post/data-analyst-to-data-engineer/index.llms.md)
- [How to create an SCD2 Table using MERGE INTO with Spark & Iceberg](https://www.startdataengineering.com/post/create-scd2-table-with-merge-into-with-spark-iceberg/index.llms.md)
- [Data Pipeline Design Patterns - #2. Coding patterns in Python](https://www.startdataengineering.com/post/code-patterns/index.llms.md)
- [Automating data testing with CI pipelines, using Github Actions](https://www.startdataengineering.com/post/ci-data-test/index.llms.md)
- [Change Data Capture, with Debezium](https://www.startdataengineering.com/post/change-data-capture-using-debezium-kafka-and-pg/index.llms.md)
- [Designing a "low-effort" ELT system, using stitch and dbt](https://www.startdataengineering.com/post/build-a-simple-data-engineering-platform/index.llms.md)
- [Apache Superset Tutorial](https://www.startdataengineering.com/post/apache-superset-tutorial/index.llms.md)
- [Hands-on Apache Airflow 3.0 Tutorial](https://www.startdataengineering.com/post/airflow-tutorial/index.llms.md)
- [Advanced SQL is knowing how to model the data & get there effectively](https://www.startdataengineering.com/post/advanced-sql/index.llms.md)
- [6 Key Concepts, to Master Window Functions](https://www.startdataengineering.com/post/6-concepts-to-clearly-understand-window-functions/index.llms.md)
- [3 Key Points to Help You Partition Late Arriving Events](https://www.startdataengineering.com/post/3-key-points-to-help-you-partition-late-arriving-events/index.llms.md)
- [10 Skills to Ace Your Data Engineering Interviews](https://www.startdataengineering.com/post/10-Skills-to-ace-your-data-engineering-interview/index.llms.md)
- [Start Data Engineering](https://www.startdataengineering.com/index.llms.md)
- [Go from building portfolio projects to designing production-ready data products](https://www.startdataengineering.com/courses/data-engineering-design-patterns-course/index.llms.md)
- [Courses & Books](https://www.startdataengineering.com/courses.llms.md)
- [How to Create Python Data Pipelines by Defining Architecture and Generating Code with LLMs](https://www.startdataengineering.com/post/architect-data-pipelines/index.llms.md)
- [About](https://www.startdataengineering.com/about.llms.md)
- [Just Python, SQL, & Airflow will not land you a data engineering job
](https://www.startdataengineering.com/courses/data-engineering-course/index.llms.md)
- [Subscribe](https://www.startdataengineering.com/newsletter.llms.md)
- [10 Key skills, to help you become a data engineer](https://www.startdataengineering.com/post/10-key-skills-data-engineer/index.llms.md)
- [4 Data Engineering Concepts To Land A High-Paying Data Engineering Job](https://www.startdataengineering.com/post/4-concepts-high-paying-data-eng-job/index.llms.md)
- [What is a staging area?](https://www.startdataengineering.com/post/What-and-Why-staging/index.llms.md)
- [Advantages of Using dbt(Data Build Tool)](https://www.startdataengineering.com/post/advantages-of-using-dbt-data-build-tool/index.llms.md)
- [Apache Airflow Review: the good, the bad](https://www.startdataengineering.com/post/apache-airflow-review-the-good-the-bad/index.llms.md)
- [A proven approach to land a Data Engineering job](https://www.startdataengineering.com/post/approach-to-land-a-DE-job/index.llms.md)
- [How to do Change Data Capture (CDC), using Singer](https://www.startdataengineering.com/post/cdc-using-singer/index.llms.md)
- [How to choose the right tools for your data pipeline](https://www.startdataengineering.com/post/choose-tools-dp/index.llms.md)
- [How to set up a dbt data-ops workflow, using dbt cloud and Snowflake](https://www.startdataengineering.com/post/cicd-dbt/index.llms.md)
- [Building Cost Efficient Data Pipelines with Python & DuckDB](https://www.startdataengineering.com/post/cost-effective-pipelines/index.llms.md)
- [CTEs(Common Table Expression) or Temporary Tables for Spark SQL](https://www.startdataengineering.com/post/cte-or-temp-table/index.llms.md)
- [ Enable stakeholder data access with Text-to-SQL RAGs](https://www.startdataengineering.com/post/data-democratize-llm/index.llms.md)
- [Data Engineering Project for Beginners - Batch edition](https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/index.llms.md)
- [Designing a Data Project to Impress Hiring Managers](https://www.startdataengineering.com/post/data-engineering-project-to-impress-hiring-managers/index.llms.md)
- [Data Engineering Projects](https://www.startdataengineering.com/post/data-engineering-projects/index.llms.md)
- [What is the difference between a data lake and a data warehouse?](https://www.startdataengineering.com/post/data-lake-warehouse-diff/index.llms.md)
- [Using Joins and Group Bys the right way for data warehousing](https://www.startdataengineering.com/post/data-modeling-join-groupby/index.llms.md)
- [How to Implement Data Quality Checks in Python Without Third-Party Tools](https://www.startdataengineering.com/post/data-quality-with-python/index.llms.md)
- [dbt (Data Build Tool) Tutorial](https://www.startdataengineering.com/post/dbt-data-build-tool-tutorial/index.llms.md)
- [Data Engineering Best Practices - #1. Data flow & Code](https://www.startdataengineering.com/post/de_best_practices/index.llms.md)
- [Data Engineering Interview Preparation Series #1: Data Structures and Algorithms](https://www.startdataengineering.com/post/de_interview_dsa/index.llms.md)
- [Data Engineering Interview Preparation Series #3: SQL](https://www.startdataengineering.com/post/de_interview_sql/index.llms.md)
- [Demonstrate Python Expertise by Building Libraries: From Architecture to Published Package](https://www.startdataengineering.com/post/demonstrate-python-expertise-building-libraries/index.llms.md)
- [Docker Fundamentals for Data Engineers](https://www.startdataengineering.com/post/docker-for-de/index.llms.md)
- [Ensuring Data Quality, With Great Expectations](https://www.startdataengineering.com/post/ensuring-data-quality-with-great-expectations/index.llms.md)
- [How to Identify and Fix Small Files Problem with Spark & Iceberg](https://www.startdataengineering.com/post/fix-small-files-spark-iceberg/index.llms.md)
- [How to add tests to your data pipelines](https://www.startdataengineering.com/post/how-to-add-tests-to-your-data-pipeline/index.llms.md)
- [Understand & Deliver on Your Data Engineering Task](https://www.startdataengineering.com/post/how-to-deliver-on-your-de-tasks/index.llms.md)
- [How to gather requirements to re-engineer a legacy data pipeline](https://www.startdataengineering.com/post/how-to-gather-requirements-reengineering-data-pipeline/index.llms.md)
- [How to Join a fact and a type 2 dimension (SCD2) table](https://www.startdataengineering.com/post/how-to-join-fact-scd2-tables/index.llms.md)
- [3 Key techniques, to optimize your Apache Spark code](https://www.startdataengineering.com/post/how-to-optimize-your-spark-jobs/index.llms.md)
- [How to submit Spark jobs to EMR cluster from Airflow](https://www.startdataengineering.com/post/how-to-submit-spark-jobs-to-emr-cluster-from-airflow/index.llms.md)
- [How to Validate Datatypes in Python](https://www.startdataengineering.com/post/how-to-validate-datatypes-in-python/index.llms.md)
- [How to implement data quality checks with greatexpectations](https://www.startdataengineering.com/post/implement_data_quality_with_great_expectations/index.llms.md)
- [3 Design Decisions for Maintainable Incremental Data Pipelines](https://www.startdataengineering.com/post/incremental-load-strategy/index.llms.md)
- [How to Quickly Learn Any Data Engineering Tool](https://www.startdataengineering.com/post/learn-data-engineering-tool/index.llms.md)
- [How to ensure consistent metrics in your warehouse](https://www.startdataengineering.com/post/metrics_sot/index.llms.md)
- [How to gather requirements for your data project](https://www.startdataengineering.com/post/n-questions-data-pipeline-req/index.llms.md)
- [6 Steps to Avoid Messy Data in Your Warehouse](https://www.startdataengineering.com/post/n-steps-avoid-messy-dw/index.llms.md)
- [How to reduce your Snowflake cost](https://www.startdataengineering.com/post/optimize-snowflake-cost/index.llms.md)
- [4 Key Patterns to Load Data Into A Data Warehouse](https://www.startdataengineering.com/post/patterns-to-load-data-into-data-warehouse/index.llms.md)
- [How to Pull Data from an API, Using AWS Lambda](https://www.startdataengineering.com/post/pull-data-from-API-using-lambda-s3/index.llms.md)
- [Python Essentials for Data Engineers](https://www.startdataengineering.com/post/python-for-de/index.llms.md)
- [Python Notebooks in Production: How marimo Solves Jupyter's Biggest Problems for Software Engineers](https://www.startdataengineering.com/post/python-notebook-best-practices-for-data-engineering/index.llms.md)
- [How to reference a seed from a different dbt project?](https://www.startdataengineering.com/post/ref-seed-from-diff-dbt-project/index.llms.md)
- [How to Scale Your Data Pipelines](https://www.startdataengineering.com/post/scale-data-pipelines/index.llms.md)
- [Setting up end-to-end tests for cloud data pipelines](https://www.startdataengineering.com/post/setting-up-e2e-tests/index.llms.md)
- [How to quickly set up a local Spark development environment?](https://www.startdataengineering.com/post/spark-local-setup/index.llms.md)
- [SQL or Python for Data Transformations?](https://www.startdataengineering.com/post/sql-v-python/index.llms.md)
- [How to trigger a spark job from AWS Lambda](https://www.startdataengineering.com/post/trigger-emr-spark-job-from-lambda/index.llms.md)
- [How to update millions of records in MySQL?](https://www.startdataengineering.com/post/update-mysql-in-batch/index.llms.md)
- [How to use nested data types effectively in SQL](https://www.startdataengineering.com/post/use-structs-sql/index.llms.md)
- [Use Given/When/Then Specs to Make AI Generate Production-Ready Pipelines, Not Spaghetti Code](https://www.startdataengineering.com/post/using-llms-for-data-engineering/index.llms.md)
- [Visual Studio Code (VSCode) extensions for data engineers](https://www.startdataengineering.com/post/vscode-extensions-for-data-engineers/index.llms.md)
- [What Does It Mean for a Column to Be Indexed](https://www.startdataengineering.com/post/what-does-it-mean-for-a-column-to-be-indexed/index.llms.md)
- [What, why, when to use Apache Kafka, with an example](https://www.startdataengineering.com/post/what-why-and-how-apache-kafka/index.llms.md)
- [How to make data pipelines idempotent](https://www.startdataengineering.com/post/why-how-idempotent-data-pipeline/index.llms.md)
- [Why use Apache Airflow (or any orchestrator)?](https://www.startdataengineering.com/post/why-to-use-orchestrators/index.llms.md)