AI (& the discourse around it) is causing anxiety among data engineers. If you are anxious about
The data field not existing in a few years and worried about employability
Agents seemingly being able to do pipeline, architecture, etc
No jobs for most people in 5 years
Then this post is for you. While LLMs can generate code & plausible architecture, an engineer still has to be accountable for the output.
Solve problems for people, and you’ll always be in demand.
Control what you can and build useful systems. People will always need problem solvers.
This post assumes that you are already adding business value and have the data skills; if not, read those first.
Build Reliable Systems
The best way to be valuable is to learn new things & build software to solve pain.
As data engineers, we own the software we build. We need to know why and how our software works.
We can’t offload system design to LLMs.
But we can use LLMs to review your design.
LLMs sometimes generate novel solutions and sometimes garbage solutions.
If LLMs recommend a novel solution, we learn something new; if it recommends a garbage solution, we don’t have to use it.
When we build software, we are building business understanding1.
Don’t delegate critical thinking to an LLM2.
If your foundations are shaky, LLMs won’t be effective.
LLMs are extremely helpful when you have the deterministic systems and code design nailed down, as shown below.
As of writing this, coding is not solved.
When you build a new system, reflect on how the software could’ve been better and how the process could’ve been faster. Rinse & repeat.
Next, the question arises: What to build?
What to Build
Start by building systems to fix your problems.
Build Systems to Make Your Life Easy
Audit your work: where do you spend most of your time? Which of those can you streamline or fully automate?
Is your dev pipeline (JIRA -> PR -> Review -> Deploy) slow? How can you speed it up with LLMs?
You can use LLMs to
- Create a first-pass PR
- Debug issues from a stack trace
All of these require an LLM API key and can be done with a simple GitHub Actions workflow.
For example, the code below shows a simple PR review bot.
name: Code Review
...
jobs:
review:
runs-on: ubuntu-latest
steps:
...
- uses: anthropics/claude-code-action@v1
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
github_token: ${{ secrets.GITHUB_TOKEN }}
prompt: |
Review PR #${{ github.event.pull_request.number }}.
First read CODING_STANDARDS.md in the repo root; these are the
standards to review against. Then run `gh pr diff` to see the
change, and read the surrounding code for anything you need to
understand it. Post one review comment with `gh pr comment`:
violations of the coding standards, with the file, the rule broken
and a suggested fix; then bugs and risky behavior; then anything
that would trip up the next person to read it.
If it looks fine, say so in a sentence.
claude_args: >-
--allowedTools "Read,Glob,Grep,Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr comment:*)"- 1
- Use specific coding standards
- 2
-
Specify
githubcommands to use - 3
- Specify allowed tools
Enable data engineers to take on more challenging problems and move faster.
Build Systems to Make Your Stakeholders’ Lives Easy
Next, think about how your stakeholder operates.
For example, if they ask about column definitions, data sources, or quality checks that were run on a data asset? Use LLMs + GitHub Workflow to answer them directly via Slack.
Note: In the above, we use GitHub workflow, but it can be any LLM hook that your company has
The key idea is that the LLM is able to access the repository of information, in this case, your codebase.
Add contextual information to your repo. You can do this with
Keeping all the information in one place (your repository) and giving the LLM full read access will produce better answers for stakeholders.
The main takeaway is to identify where/when stakeholders are blocked and build systems to unblock them.
Identify Bottlenecks at Your Company and Fix Them
Next, map out how your business works, from interactions with external clients, vendors, etc. Who interacts with whom, etc.
Identify bottlenecks: can they be sped up, and how?
You can automate parts involving Excel workbooks and manual reviews.
If you understand how the flow works, you can build systems to speed it up.
Don’t try to replace people; enable them by building systems that make their lives easier.
Don’t Overwhelm People
Most importantly, since code/content generation is cheap, do not overwhelm people.
No one can review a 10,000-line PR.
Be thoughtful about how the output of the system you build will be used.
Ask yourself:
Does this actually help people, or is it just another thing they have to do?
Conclusion
To recap, we saw
- How to leverage LLMs to build reliable systems
- What types of systems to build
- That we shouldn’t overwhelm users with slop
Build problem-solving software; be responsible for its output. You will always be in demand.