Data Engineering Project for Beginners - Batch edition
· 22 min read
Data engineering project for beginners, using AWS Redshift, Apache Spark in AWS EMR, Postgres and orchestrated by Apache Airflow.
Change Data Capture Using Debezium Kafka and Pg
· 17 min read
Change data capture tutorial using Debezium Kafka and Postgres. Change data capture is a software design pattern used to capture changes to data and take corresponding action based on that change. The change to data is usually one of read, update or delete. The corresponding action usually is supposed to occur in another system in response to the change that was made in the source system.
What Does It Mean for a Column to Be Indexed
· 10 min read
Tutorial on what happens when you create an index in a database. Trade offs to be aware of while creating an index and how to use it efficiently.
dbt(Data Build Tool) Tutorial
· 11 min read
DBT (data build tool) tutorial. Build a project simulating a real life ELT project using the data build tool.
Advantages of Using dbt(Data Build Tool)
· 3 min read
In this article we aim to go over the reasoning behind why someone might want to use dbt. If you are interested in running dbt check out this article dbt tutorial. Some common questions from Data Engineers about dbt are
it is not very clear to me why would I use
dbtinstead of running SQL queries on Airflow
Apache Airflow Review: the good, the bad
· 6 min read
When getting started with Apache Airflow, data engineers have questions similar to the two below
“What are people’s opinions of Airflow?”
Review: Building a Real Time Data Warehouse
· 4 min read
Many data engineers coming from traditional batch processing frameworks have questions about real time data processing systems, like
“What kind of data model did you implement, for real-time processing?”
3 Key Points to Help You Partition Late Arriving Events
· 5 min read
3 Techniques to process events arriving late in real time streaming systems
Scheduling a SQL script, using Apache Airflow, with an example
· 3 min read
One of the most common use cases for Apache Airflow is to run scheduled SQL scripts. Developers who start with Airflow often ask the following questions
“How to use airflow to orchestrate sql?”
10 Key skills, to help you become a data engineer
· 3 min read
Key skills required for a data engineer. If you are getting into data engineering, this post gives you a ordered list of topics you can start learning.