Java Juggernaut: The key to data engineering mastery — Data Engineering and Programming Skills When we think about data engineering, the first programming skills that usually come to mind are SQL and maybe Python. SQL is this well-known language for querying data, deeply ingrained in the world of data and pipelines. Python, on the other hand, has become quite powerful in data science and…

How to set up a personal proxy server for WhatsApp on AWS — Today, Jan 5th, 2023, WhatsApp announced launching proxy support for WhatsApp users all over the world (https://blog.whatsapp.com/connecting-to-whatsapp-by-proxy). This feature is helping people to get connected via this app, especially in areas where the app is filtered by governments or other authorities. To help people to use this feature, I personally…

How does a data engineer develop a data pipeline? — Developing a new data pipeline is a long process. It usually starts with the business justification, defining metrics, researching available data and tables, and finally, developing and launching the pipeline. In this article, I specifically focus on the development stage. Assume, at this step; you know the business requirements, worked…

A “peel-and-understand-the-layer” strategy for complex SQL queries — A single SQL query for an ad hoc analysis or reporting could vary between 2–40 lines (typically). But when it comes to data pipelines and scheduled queries, a single SQL query could be hundreds of lines (easily! believe me). A single SQL query with hundreds of lines, tens of CTEs…

Explaining the P-value in the most "ridiculous" way — P-value is not an easy and intuitive concept at first (especially if you read its definition on Statistics textbooks or Wiki pages). But, here, I try to explain it simply using a powerful word called "Ridiculous." You must ask why I use a word like ridiculous several times in the…

A few tips to improve the quality of your SQL queries and collaborate better — A data pipeline is stuck and needs someone's attention. Or a query needs to be refactored using a new table because one of the used tables is getting deprecated or unavailable. Or we need to add more to our SQL logic of a very old data pipeline. In all those…

Why is code review essential? And what are the best practices? — What is code review? We all write code, and we would love to commit our code to the main pool of codes. It is an exciting moment, and we cannot wait to see our latest masterpiece in the main repo. Let’s commit and merge. …

What you need to know before landing a data engineering role at large companies like FAANG — As a data engineer who works for a large FAANG company, I hear people frequently ask what skills are important to land a Data Engineering job at a well-known tech company? Many people think they need to be fluent in Spark or know everything about Hadoop systems to get a…

Or what are the Dimensional Design principles to make a good data warehouse? — For data scientists, machine learning models and visualization packages are essential tools. But more importantly, data scientists rely on data and data infrastructures to do their analytics and modeling. Without data and databases, all developed analytics tools and techniques are useless. Many data scientists get their data in raw formats…