LLMs have gotten good at code generation. If you are wondering
Is it worth spending time learning Python?
What skills matter as AI gets better at syntax and repetitive tasks?
About being left behind if you are reading code
Then this post is for you. LLMs can generate a lot of code, but to understand it, you need to know how to code.
TL;DR: Yes, you need to know how to code.
To be an effective data engineer, you need to know when to modify LLM-generated code.
You Can’t Be a Data Engineer Without Knowing How to Code
Let’s define what “knowing how to code” means. It’s not knowing a command’s syntax, but knowing it exists.
Knowing the features of a language (or a library) enables you to think of effective ways to build software.
Let’s look at a simple example, where knowing that a feature exists leads to maintainable code.
In the LLM-generated code above, the file OUT_FILE is not removed.
Knowing how to code means knowing what you can do with a language (or library) and its trade-offs.
Systems Design Is Not Possible Without Knowing How to Code
You often hear, “systems design is more important”. Usually from experienced engineers who have the benefit of pre-AI coding experience.
But if you are new, how do you design a system without knowing what its parts (aka code) should look like?
The other common argument is that LLMs are like assembly language and takes care of the details for you.
But they are not like assembly language. They are a non-deterministic code-generation system.
You need to know how to code, so that you can read/modify LLM-generated code to ensure it does what it’s supposed to do.
You Are Accountable for the Code You Push to Prod
You need to read LLM-generated code to verify that it is doing what it’s supposed to do.
Reading code is understanding intent. Writing code is representing intent. Understanding is usually harder than representing.
To read code, you need to know how to code.
If you create a system with an LLM and it breaks (which all software does), how will you fix it?
You can ask the LLM to create a fix; now you are in a death spiral: how will you know if it is correct?
How will you know that any test case it generates does what you intend?
Another problem with LLMs is their verbosity. More code is not better; better code is better
Better Code = Code that does only what it's supposed to do within a set of constraints, doesn't cause bugs (realistically: keeps bugs to a minimum), and is understandable.
The main takeaway is that LLMs are probabilistic systems and cannot be held accountable.
Skill Issue
Another common argument is “skill issue”. This is usually framed as “you are not using X skill, adversarial agents, evals, etc.”
While some of these systems can improve the code generation. You should judiciously incorporate them.
You still need to understand the code, since you are accountable for its outcome.
Learn by Doing
In this section, we’ll cover strategies you can use to learn new tools/systems/coding techniques, etc.
Learning to Code Is Not a One-and-Done Deal
People tend to think about learning to code as a big milestone. And spend a significant amount of time (days, months, and sometimes even years) debating whether to learn a new language.
A better way to think about learning to code is to narrowly focus on your use case.
Applying this to learning Python for Data Engineering, you can
- Learn Python basics
- Learn data pipeline patterns when using Python
- Building ETL projects
- Building systems (DQ, Lineage, Observability, etc.) for your ETL projects or using a new tool (Databricks, Snowflake, etc.) for your projects
Here’s how to decide what tools to learn.
Build a base and then specialize as needed. You don’t need to learn everything about Python, just enough to be able to do data engineering.
Over time, you should keep learning new techniques (both going deep and wide).
For example, knowing the standard Python libraries and Pyspark API.
Learning Technique: Slow Is Smooth, Smooth Is Fast
The most effective way I’ve found is the following
Read docs/books -> understand what can be done (no need to memorize syntax) -> build a simple project manually for the first time
Once you get a feel for the tools/systems, start using LLMs; you will be able to spot issues quickly.
Spend a few hours learning a tool/library vs spending months/years fixing issues over time.
Push Yourself, Go Beyond Standard ETL Projects
With LLMs, you can push yourself much harder than before.
If you are used to building a data portfolio, ask yourself: what can be better? How can I make this project work better with LLMs?
Do this ==> Build a Python Data Quality library
Conclusion
To recap, we saw
Learning to code is critical, now more than ever.
Learn the fundamentals, and learn new tools as necessary.
I’d love to hear your counterarguments. Let me know in the comments below.