How To Learn Data Engineering In 2026/2027/2028/2029

DE Roadmaps Are Getting Longer Every Year

Trying to upskill as a data engineer? You most likely have come across one of the many data engineering roadmaps that list a long set of tools.

If you are:

Wondering how to convince recruiters and non-technical hiring managers to interview you, when you don’t “know” a tool

New to the career and overwhelmed by the proliferation of tools

Worried that LLMs will take away all data jobs

This post is for you.

Upskilling in DE is not about knowing every tool/platform. Upskilling is understanding what each tool provides, and its trade-offs.

By the end of this post, you will have an approach that you can use to quickly pick up any tool or framework.

Tools and frameworks are opinionated approaches to implementing fundamentals and best practices.

Let’s define skills to learn with examples:

Fundamental concepts: Represent the building blocks of data pipelines. They include data storage, data movement, distributed data processing, Metadata, Lineage, Observability, Scheduling, Orchestration, and Coding (Python & SQL).
Best Practices: Represent design patterns to build easy-to-maintain pipelines.

They include data modeling, multi-hop architecture, Idempotency, full refresh, incremental processing, DQ WAP, lambda architecture, partitioning/clustering/sorting, etc. The data engineer needs to understand the whys and trade-offs of these best practices.
Tools: Software that enables you to implement fundamental concepts. E.g., Spark, Airflow, Cron, Iceberg, etc.

Tools can enrich the fundamental concepts, improving devex. e.g., an Iceberg catalog enables time travel, schema evolution, and ACID compliance for data IO, and dbt provides end-to-end lineage for your pipelines.
Frameworks: Standardize best practices designed for industry use. e.g., Medallion/dbt project structure is a framework based on how companies build data products.
Platforms: SAAS to handle your data infrastructure. E.g Snowflake, Managed Clickhouse, Starrocks. Some companies focus on managing OSS, some manage closed systems (e.g. Snowflake), some are a mix of both (e.g. dbt cloud, databricks)

With knowledge of the fundamentals and best practices, you can quickly pick up any tool/framework/platform.

Note

Pay close attention changes/upgrades to fundamentals & best practices. These are typically industry-changing.

E.g., Table formats (e.g., Iceberg) provide ACID compliance for data storage.

To quickly learn a tool/framework/platform, do the following:

Identify the fundamentals and best practices it enables. Most tools only focus on a subset of fundamentals/best practices.
Identify its tradeoffs by determining which fundamentals & best practices are hard to implement. E.g., dbt-cli restricts users to SQL only (limited Python support).

Identify the additional work you need to do to manage the use of this tool.
Read their docs and understand the tool-specific way of doing things. E.g., dbt project structure.
Identify plugins or extensions to the tool. E.g., dbt packages, Airflow plugins

With this knowledge, you will be able to make a reasonable judgment of a specific tool.

Note It’s hard to understand the nuances of a tool or the various ways it can fail, without truly using it.

I find searching for “tools cons, reddit” or using LLMs to list a tool’s flaws helpful.

Use the approach we saw at Quickly Learn a Tool to evaluate Apache Iceberg.

When would you use it? What are the tradeoffs you will be making by choosing to use it?

Let me know in the comments below.