Something was bugging me…
While working on my Coffee Vending Machine Business Analytics project, it really bugged me that it felt contrived and artificial.
The reason being, I found the dataset while browsing through Kaggle, downloaded it, imported it into PowerBI and went my merry way doing data exploration.
But I know it isn't that easy in the real world.
I wanted to learn about the challenge of getting the raw data right from the source and automatically feeding it into PowerBI, Tableau or any other sources.
Idea
I was already working with transaction data from the vending machine so I looked at other datasets in Kaggle. These were my thoughts with what I found:
- I could download a supply-chain data set, with different tables of transaction data of different warehouse. I can then do machine learning and optimization as an end goal. Pipeline-wise I just need to find a way to artificially re-stream the data somehow.
- I could just find a more-raw sales transaction data set, and 're-stream' the data somehow.
After those two ideas, I quickly realized that the common theme was finding a way to 're-stream' the data. So I tried reading about it.
Looking at job listings for ideas
It would be painful to learn about a tool only to find out that no employer in the job market is looking for it. So browsed through some data analyst/data engineering job descriptions to look at the current tools.
The top data engineering tools were: Spark, Airflow, Databricks, Hadoop, MongoDB, DBT, Kafka, Snowflake, Talend, DataStage, AWS Redshift, AWS Glue and more…
At this point I have no idea what most of these are, save for MongoDB and AWS.
So to speed up my research, I did something…
ChatGPT
Instead of spending a full day opening 20+ tabs of google search results, I asked ChatGPT for the current industry standards and how to I can simulate one for myself.
Some of the prompts I started with in one particular chat:
- "Loading the transformed data into the data warehouse (Amazon Redshift/Google BigQuery/Snowflake) using Talend/AWS Glue. Which is compatible with which?"
- "What other database-ETL-warehouse tech stacks are there?"
- "I want to focus on Kafka, Spark, Hadoop, AWS Glue, Redshift, Snowflake, Redshift, Bigquery, Data bricks. Give stacks involving these and appropraite datasets/industries to process."
and one of the answers:
Aftermath of going down the Data Engineering rabbit hole
To get a handle of the black-hole of information ChatGPT threw at me, I tried visualizing them with Miro. Here are how my charts progressed:
Don't worry… I learned and enlightened myself… ver. 2:
I decided that I will just try to focus on the Tableau delivery out of the five possible.
Going to the local library (thanks NLB)
I got acquainted with "The Fundamentals of Data Engineering" by Joe Reis & Matt Housley (O'Reilly).
It is a great book and covers every data engineering concept to get started and to get advanced. Highly recommended.
To be continued…
This is not easy…