Build Data Engineering Projects, with Free Template

· 6 min read
Setting up data infra is one of the most complex parts of starting a data engineering project. Overwhelmed trying to set up data infrastructure with code? Or using dev ops practices such as CI/CD for data pipelines? In that case, this post will help! This post will cover the critical concepts of setting up data infrastructure, development workflow, and sample data projects that follow this pattern. We will also use a data project template that runs Airflow, Postgres, DuckDB & Quarto to demonstrate how each concept works.
Read More

Python Essentials for Data Engineers

· 10 min read
You know Python is essential for a data engineer. Does anyone know how much one should learn to become a data engineer? When you're in an interview with a hiring manager, how can you effectively demonstrate your Python proficiency? Imagine knowing exactly how to build resilient and stable data pipelines (using any language). Knowing the foundational ideas for data processing will ensure you can quickly adapt to the ever-changing tools landscape. In this post, we will review the concepts you need to know to use Python effectively for data engineering. Each concept has an associated workbook for practicing these concepts. You can try them out directly in your browser with GitHub Codespaces.
Read More

Building Cost Efficient Data Pipelines with Python & DuckDB

· 12 min read
Imagine working for a company that processes a few GBs of data every day but spends hours configuring/debugging large-scale data processing systems! Whoever set up the data infrastructure copied it from some blog/talk by big tech. Now, the responsibility of managing the data team's expenses has fallen on your shoulders. You're under pressure to scrutinize every system expense, no matter how small, in an effort to save some money for the organization. It can be frustrating when data vendors charge you a lot and will gladly charge you more if you are not careful with usage. Imagine if your data processing costs were dirt cheap! Imagine being able to replicate and debug issues quickly on your laptop! In this post, we will discuss how to use the latest advancements in data processing systems and cheap hardware to enable cheap data processing. We will use DuckDB and Python to demonstrate how to process data quickly while improving developer ergonomics.
Read More

Enable stakeholder data access with Text-to-SQL RAGs

· 13 min read
You want to democratize your company's data to a larger part of your organization. However, trying to teach SQL to nontechnical stakeholders has not gone well. Stakeholders will always choose the easiest way to get what they want: by writing bad queries or opening an ad-hoc request for a data engineer to handle. You hope stakeholders will recognize the power of SQL, but it can be disappointing and frustrating to know that most people do not care about learning SQL but only about getting what they need, fast! The result is that the data team is overloaded with ad-hoc data requests or has to deal with bad queries that can bring a warehouse to its knees. Imagine a scenario where stakeholders can independently analyze data in our warehouse without needing a new dashboard for each request. This would free up the data team for more focused, deep work while empowering stakeholders to become proficient in data analysis. It is asking a lot of stakeholders to take the time to learn SQL when they have multiple other priorities. One way to get stakeholders to get good at SQL is to enable them to see the SQL query they will need to run to get the data they need. In this post, we will build an RAG to convert stakeholder questions to SQL queries, which the stakeholder will run. Repeatedly seeing the SQL queries necessary to get the needed data, the stakeholders can modify them according to their needs and eventually write SQL themselves! By the end of this post, you will learn how to build a simple LLM-powered text-to-SQL query engine for your data.
Read More

How to reduce your Snowflake cost

· 14 min read
Have you worked on Snowflake SQL written without concern for maintainability or performance? Most data projects are built without consideration of warehouse costs! You may be a new data engineer brought in to optimize Snowflake usage or suddenly thrust into a cost-reduction project. While the Snowflake contracts are signed by management without your consultation, the cost reduction initiative will fall on you, the data engineer! On top of that, you will be held responsible for skyrocketing costs in addition to the feature delivery work that you are expected to deliver! It can be quite stressful and frustrating to deal with costs in a short amount of time! A high warehouse bill can cost jobs and maybe a key metric for your career aspirations. What if your Snowflake warehouse can be a profit center? Imagine Snowflake cost management on auto-pilot. In addition to money saved, you can improve your team's morale and use the savings as a key point for your promotion! In this post, we will discuss four strategies for reducing your Snowflake costs. By the end of the posts, you will have a plan that includes quick wins, long-term planning, and monitoring costs so that they don't go haywire.
Read More

How to test PySpark code with pytest

· 8 min read
Working on a large codebase without any tests can be nerve-wracking. One wrong line of code or an in-conspicuous library update can bring down your whole production pipeline! Data pipelines start simple, so engineers skip tests, but the complexity increases rapidly after a while, and the lack of tests can grind down your feature delivery speed. It can be especially tricky to start testing if you are working on a large legacy codebase with few to no tests. In long-running data pipelines, bad code can take hours to be identified and fixed, causing stakeholder frustration! No testing => Stress, Worry, Extra work from an apparent bug that made it to production! What if you could confidently push changes to production without worrying about slowing down your pipeline? Quickly delivering features and empowering your team to be fast can uplevel your career as a data engineer. Tests => Peace of mind, fast feature delivery, and happy stakeholders. While testing will not catch all potential issues, it can avoid a significant number of production issues. In this post, we go over the types of tests and how to test PySpark data pipelines with pytest. We will create tests (unit, integration, and end-to-end) for a simple data pipeline that demonstrates key concepts like fixtures and mocking. By the end of this post, you will be able to identify pieces of your data pipeline to add tests.
Read More

Docker Fundamentals for Data Engineers

· 7 min read
Docker can be overwhelming to start with. Most data projects use Docker to set up the data infrastructure locally (and often in production as well). Setting up data tools locally without Docker is (usually)a nightmare! The official Docker documentation, while extremely instructive, does not provide a simple guide covering the basics for setting up data infrastructure. With a good understanding of data components and their interactions combined with some networking knowledge, you can easily set up a local data infrastructure with Docker. Knowing the core fundamentals of Docker will not only help you set up data infrastructure quickly but also empower you to think about networking, volumes, ports, etc., which are critical parts of most cloud data infrastructure. I wrote a post that covers the fundamental concepts one will need to set up complex data infra locally. By the end of the post, you will be able to use Docker to run any data tool (that is open source) locally on your laptop. In the post, we set up a Spark cluster, Postgres database, and minio (OSS cloud storage system) that can communicate with each other using Docker.
Read More

Data Engineering Best Practices - #2. Metadata & Logging

· 9 min read
Imagine this scenario: You are on call when suddenly an obscure alert pops up. It just says that your pipeline failed but has no other information. The pipelines you inherited (or didn't build) seem like impenetrable black boxes. When they break, it's a mystery—why did it happen? Where did it go wrong? The feeling is palpable: frustration and anxiety mount as you scramble to resolve the issue swiftly. It's a common struggle, especially for new team members who have yet to unravel the system's intricacies or data engineers who have to deal with pipelines built without observability. The root cause often lies in systems built without consideration for debugging and quick issue identification. The consequence? Lengthy downtimes, overburdened on-call engineers, and a slowdown in feature delivery. However, the ramifications extend beyond the technical realm, as incorrect data or failure to quickly fix a high-priority pipeline can erode stakeholder trust. Bugs are inevitable, but imagine a system that detects issues and provides the necessary information for an engineer to fix them quickly! A well-setup system that captures pertinent pipeline metadata and logs and exposes them in an easy-to-access UI will significantly reduce engineering time spent on fixing bugs. In the following post, you will learn what metadata is (in the context of data pipelines), how to log and monitor it, and how to design actionable alerts that simplify resolving bugs, even for someone new to the team. You will also set up an end-to-end logging system with Spark, Prometheus, and Grafana.
Read More