3 Key techniques, to optimize your Apache Spark code

Jun 19, 2020 · 19 min read

This post covers key techniques to optimize your Apache Spark code. You will know exactly what distributed data storage and distributed data processing systems are, how they operate and how to use them efficiently. Go beyond the basic syntax and learn 3 powerful strategies to drastically improve the performance of your Apache Spark project.

What, why, when to use Apache Kafka, with an example

Jun 11, 2020 · 10 min read

Apache Kafka is a distributed streaming platform. This post goes over the common scenarios when using Apache Kafka will be beneficial, how to use it and the basic concepts of Apache Kafka

A proven approach to land a Data Engineering job

Jun 2, 2020 · 8 min read

Proven approach to get usable experience and land a data engineering job

What Does It Mean for a Column to Be Indexed

May 2, 2020 · 10 min read

Tutorial on what happens when you create an index in a database. Trade offs to be aware of while creating an index and how to use it efficiently.

Advantages of Using dbt(Data Build Tool)

Apr 25, 2020 · 2 min read

In this article we aim to go over the reasoning behind why someone might want to use dbt. If you are interested in learning dbt checkout this article . Some common questions from Data Engineers about dbt are

it is not very clear to me why would I use dbt instead of running SQL queries on Airflow

Apache Airflow Review: the good, the bad

Apr 18, 2020 · 6 min read

When getting started with Apache Airflow, data engineers have questions similar to the two below

“What are people’s opinions of Airflow?”

Review: Building a Real Time Data Warehouse

Apr 11, 2020 · 4 min read

Many data engineers coming from traditional batch processing frameworks have questions about real time data processing systems, like

“What kind of data model did you implement, for real-time processing?”

3 Key Points to Help You Partition Late Arriving Events

Apr 5, 2020 · 5 min read

3 Techniques to process events arriving late in real time streaming systems

Scheduling a SQL script, using Apache Airflow, with an example

Mar 29, 2020 · 3 min read

One of the most common use cases for Apache Airflow is to run scheduled SQL scripts. Developers who start with Airflow often ask the following questions

“How to use airflow to orchestrate sql?”

10 Key skills, to help you become a data engineer

Mar 20, 2020 · 3 min read

Key skills required for a data engineer. If you are getting into data engineering, this post gives you a ordered list of topics you can start learning.