How to Get Promoted to a Senior Data Engineer?

In the data industry (& SWE), the term “value” is thrown around a lot: “adding value”, “business value”, etc.

Data engineers are expected to add business value, but what exactly is this “value”? Is it just a salesy term, or is there a way to identify this value?

If you

Are a DE stuck at junior or mid level and unable to get promoted

Do not know how to identify projects that add value

Are being told you are thinking tactically, not strategically

Then this post is for you.

There is no universal definition of value.

However, using the ideas below, you can identify what your leadership values. And position yourself as an invaluable problem solver (aka adding business value).

Note

The techniques in this article are meant for someone in junior/mid levels to get promoted to a senior role.

Prerequisite: Read this article on the baseline skills you need, before using the techniques below.

Business value is whatever leadership cares about

Even if you understand your business and can propose a valuable project.

Unless it’s part of the company-wide strategy, it will be perceived as a one-off project.

Note

Leadership = A person who manages multiple teams. When we say leadership, we mean at least one of them.

Positioning = How people view you & your work

Pay attention to what leadership values and needs (typically said during presentations, future plans, Q&As, etc.), then build projects toward that end.

This approach positions you as someone who solves “real” business problems.

There may be bigger problems, but unless leadership is excited about them, it’s better to solve a perceived problem than to change people’s perceptions.

Your project’s output should do at least one of the following

Directly impact a key metric
Be necessary (not good to have) for new initiatives
Resolve a common pain (e.g., data refresh is too slow, etc.)

Help stakeholders with problems that leadership is aware of. Position yourself as someone who enables teams to do their job.

Alternatively, you can build tools, develop processes, or build systems to streamline data pipeline management. This will bring you more exposure among engineers.

Make sure that leadership is aware of this problem before you solve it. You can also bring it to engineering leader’s attention by following the WTF notebook method.

The more people who want the output, the better your recognition will be.

The technicality of the project matters less than how many people would benefit, so aim for projects whose outcomes impact people (& that helps them get promoted).

Build solutions for your stakeholders, not yourself

Engineers tend to build projects they’d use, but that’s only good if stakeholders share the same skill set and priorities.

You can quickly identify our stakeholder skill set by looking at

The current tools they use (SQL, YAML, BI tools, vendor vs. open source)
History of failures; for example, was your company burned by vendor price increases and had to spend a lot of time migrating off that tool?

Don’t put stakeholders through the pain of learning a new system. Build for your stakeholders using patterns and tools that they are familiar with.

If you are introducing new tools/processes, consider the risks and do a cost-benefit analysis.

For example, if you are building a system based on OSS, what are the risks? Ops, debugging, management, etc. Is the cost (management) worth the benefit (not having to pay a vendor)?

Many data systems are high risk. Always do this analysis to show leadership that you’ve thought about risks (critical as you move up the career ladder).

Takeaway

Build projects using existing tools and familiar systems, unless there is a very good reason not to.

Ask your team, manager, and stakeholders for feedback before building. Identify interested people; they will be your evangelists.

Showcase your work, associate your name with positive outcomes

Assuming we have a clear problem and a solution stakeholders prefer, the next step is to run a demo tour.

Note

Demo tour = Presenting the results of your project at multiple meetings

Create slides to

Define the business problem
Show the current state
Show the solution (keep this high level and save the detailed architecture for documentation)
Show the outcome ($ increase in revenue by using this dataset, time saved, etc)
What’s next (frame this project as part of a larger pattern, even if it ends up going nowhere)

Here we want to associate your name with the outcome, so make sure to present this in every appropriate meeting.

Always frame the project as a system rather than a one-off dataset.

For example, if you build a sales dataset, mention that it covers part of the sales pipeline and that future work can use it and build on it.

It shows leadership that you are thinking strategically (long-term direction), not just tactically (doing tasks), which is critical for promotion.

Conclusion

To recap, we saw

Use this framework to identify potential projects and showcase your skills.

This will help with promotions & link you to strong business outcomes, which is critical for future references.

How do you ensure you add business value? Let me know in the comments below.