JBrien's Tek Blog

Learning Data Engineering (Part 3) – Data Engineering Undercurrents

Thu, 19 Dec 2024 00:00:00 GMT

What separates an enterprise vs non-enterprise data engineering setup is whether the data engineer has accounted for the 'Data Engineering Undercurrents'.

The most obvious one is 'Security', but as Joe Reis and Matt Housley shares with us, there are 5 others.

Here I identify the undercurrents relevant to me (which is all categories) and see how I can implement them.

External reference image: view the original diagram at O'Reilly.

_{Reis, J., & Housley, M. (2022). Fundamentals of Data Engineering. O'Reilly Media.}

With a bit of time and effort, I expanded on each of the 6 points with the help of LLMs. For each sub-point, I then research the industry standard for implementing these concepts. This is what I came up with:

Laying it out like this has us get started on a checklist on the necessary bells and whistles that our data infrastructure has to have. It'll get filled up as the project progress.

~End

Taking a short break from Data Engineering: Learning Game Development

Thu, 19 Dec 2024 00:00:00 GMT

TLDR

Mushroom Text Game: https://github.com/jbgithub22/Mush/tree/main
Learned about HD-2D games
Started learning about making 2D games in Unreal Engine
Made Unreal Engine 2D Game: https://jbgithub22.itch.io/hello-world-test-game

Reminisce

The first game I ever made was in school: ASCII Mushroom text game...

^{Screenshot of the text game}

With one booboo, I skipped the last few classes which taught classes/oop and so made the entire game with just functions. It was fun nonetheless~

So now I want to try my hand at it again

I didn't want to overwhelm myself so I was looking to make 2D games with RPG Maker. Then I moved to looking at Unity.

I was looking at Unity for future-proofing. I wanted to learn just one thing that would give me access to developing many types of games, from mobile all the way up to potentially making VR games.

Then I learned that Unreal Engine can also do it, and with even higher fidelity, and with an even more attractive licensing agreement.

So I started research

Then I found this video by Thomas Brush titles "Can You Really Make A 2D Game In Unreal?" that linked to a video by Cobra Code:

https://www.youtube.com/watch?v=QVxK2dPJr4g&t=2s&ab_channel=CobraCode

And that's where I learned about HD-2D games, which is a trademark owned by Square Enix by the way.

https://www.youtube.com/watch?v=K6wW0pO08LE&t=0s&ab_channel=UnrealEngine

I gew up playing games like Maplestory, Ragnarok Online, Diablo 2, Red Alert and Starcraft. So learning that I could make these kinds of games with triple-A lighting, shadows and effects is very exciting.

It is easy to say that game development is hard. It is hard for a non-gamedev to understand how hard it is. So I created a list of thing that I needed to do:

Components (Large View)

Character Sprites
1. Swordsman
  1. Design
  2. Animation
2. Archer
  1. Design
  2. Animation
3. Mage
  1. Design
  2. Animation
4. Healer/Priest
  1. Design
  2. Animation
5. NPCs
  1. Vendors
    1. Design
    2. Animation
  2. Quest Givers
    1. Design
    2. Animation
6. Monsters/Antagonists
  1. Design
  2. Animation
Level Design
1. Only 1 town map where the demo/story is set
  1. Buildings/Locations
    1. Interiors
  2. Points of Interest
  3. History
Music
1. Picking out inspiration music to 'take inspiration from'
2. Draft
3. Refinement
Story
1. Quest
  1. Ideas
  2. Dialogue
  3. Rewards
  4. Stake/Pull factor

It is a lot. The more characters, NPCs, monsters, items, effects and gameplay you add, naturally the more work you need to do. And as someone who has worked with photoshop in an architectural studio, I shudder at the amount of art I need to draw.

At least so far the rate of increase in work is linear. BUT, the calculus of the amount of work you need to do changes to exponential the more intertwining gameplay mechanics and quests you add. YIKES.

Starting out

So I found an Unreal Engine 2D game tutorial and made this:

Download link: https://jbgithub22.itch.io/hello-world-test-game

Conclusion

It's fun!

Let's see how it goes~

Learning Data Engineering (Part 2) - Hooking up to a Blockchain API

Mon, 25 Nov 2024 00:00:00 GMT

Click here for Part 1.

Let's start with the questions that I had:

What data do I feed through the pipeline and what would I want to do with it at the end?

I already have some ideas of what kind of data I'd choose as source. Starting with questions I already asked ChatGPT previously...

I chose blockchain data.

Of all the available ones, I chose the Ethereum Blockchain specifically so that I can relate the data to NFT marketplace data, which most are traded with Ethereum.

Blockchain data is transaction data. Each block can be interpreted as a large JSON file with the transaction value, sender and receiver information. I also dabbled with NFTs before and know that there's a treasure trove of data there as well.

The data products that I can produce with these data can range from 'simple' Exploratory Data Analysis to more complex predictions for prices or ideal themes for NFT products.

To proceed...

I have these steps in mind...

Step 1: The Alchemy API for the Ethereum blockchain

ChatGPT already provided me with an idea for a full pipeline in the previous post (such as "Ethereum Blockchain > The Graph > Apache Flink > Google BigQuery > Plotly Dash"). This is too complex for me at this point so I opted to just try connecting to the blockchain API and output a CSV file of a few blocks.

Managed to do it after some fiddling, research and prompt engineering.

Link to the Jupyter Notebook

My measurable outcome: for this step is to get a csv file of the metadata. (Successful)

Step 2: The web3 API and MongoDB

The Alchemy API is generally not open-source. The Web3 API is and the web3.py library is more actively maintained than the alchemy.py library.

I wanted to enrich my learning, so I decided to include MongoDB in this prototype pipeline.

I downloaded Docker and followed some tutorials to get a MongoDB container running. And yes, that also means I also just learned how to use Docker with this step.

My measurable outcome: Send Ethereum Blockchain data to MongoDB. (Successful)

Step 3: Kafka

I now add Kafka to the pipeline. This was hard and deserves an entire post just for setting it up.

Gist of it:

Kafka and Docker networking, setting up internal and external listener addresses.
Setting up Kafka in 'Kraft' mode so it doesn't require another software called 'Zookeeper'.
Interfacing with MongoDB requires a plugin for Kafka
Resolving version conflicts between the plugin, kafka, and java installation (when building the docker image)
and probably more that I can't remember...

In the end I got it running.

Step 4: The OpenSEA API

After step 3 I realized that it's not really the blockchain metadata that I want. What I want is the NFT metadata.

It wasn't all a waste though as it meant that I already had the infrastructure setup for MongoDB and Kafka, and it works. So I went to OpenSEA to see what API they have for me to tap on.

I needed streaming data and found that I had to use the websocket API instead of the RESTful API. This means that I have to use javascript, and so I did.

Setting up the node.js project and this is what I finished with.

My measurable outcome: Send OpenSEA NFT transaction data to Kafka. (Successful)

Step 5: Spark

"Spark is like killing a fly with a bazooka" - Reddit User

This is even harder than Kafka to setup for a newbie. But yes, after hours of troubleshooting, prompt engineering, reddit/stackoverflow research, blog posts, hours of youtube tutorials... I got it working.

Gist of it:

Fixing version conflicts between the different distributions, licensing and plugins for Java, Scala and Spark.
Building the three different docker images (Jupyter, Spark Master and Spark Worker).
Setting up docker networking for the Spark Master and Workers.
Getting the write config for the spark session.
Handling json data and transforming the data types.

My measurable outcome: Extract streaming data from a Kafka topic, do minor transformation on the data, and publish to a different Kafka topic. (Successful)

Step 6: Understanding the state of Hadoop

It is while researching on Hadoop that I encountered the concept of a Data 'Lakehouse'.

Long story short, we only try to use Hadoop if we have our own servers AND are managing petabytes of data. There is also the open-source alternative which is MinIO.

So instead of simulating a Hadoop cluster locally, I opted to use Azure Storage and Databricks. I will dive into this in a separate post.

Gist of it:

Post Conclusion

Rather than steps, the above were really leaps and each step easily deserves its own post. I am actually farther ahead than all these already so I'm speeding the writing a bit.

We continue our dive into the data engineering abyss in the next post.

Moving from Github Pages to Cloudflare Pages

Sun, 24 Nov 2024 00:00:00 GMT

So, I decided to change my website host once again.

The reason is because I want to make the github repo for my website private, for security and copyright's sake, for when I do end up adding something that requires proper protection later.

(Edit: Making the Github repo of a static website doesn't hide the code and its contents, but it is useful in other ways.)

In the unpaid version of GitHub, you can't make a GitHub pages site for it if you make the repo private. You'd have to pay a subscription to be able to do so.

I never thought websites could 'move' around the internet the same way one would move house, so this was a journey, an interesting literal one.

The trip so far has been:

Moved from Hostinger to AWS EC2
Moved from AWS EC2 to Github Pages
Moved from Github Pages to Cloudflare Pages

We'll see what happens next once I move to the next design iteration of my website~

The tutorials I referred to:

What I think NFTs are for...

Mon, 07 Oct 2024 00:00:00 GMT

[Repost: This was first published in another blog on 28 Sep 2023.]

I see NFTs as they are more popularly known as not much different from collectible items that have managed to carve out their own niche in the collectibles marketplace.

People consider it to have value, whether it is for the novelty of the medium, sense of innovation, or that it could be used as some sort of commodity to be traded and thus make money from.

I see it really as no different from collector baseball caps; or collector shoes; or Starbucks mugs.

Baseball caps have a use; Shoes have a use; Mugs also have a use; and so, NFTs also have a use. But these collector items are not useful in the same sense that NFTers oversell collectible NFTs as.

The most useful form of NFTs right now are virtual tickets for events (I do hope they can find more uses for NFTs…)

The Collectible NFT was born out of an invention that had the potential to disrupt a system where rich, powerful, and influential people dominated. So getting involved, owning, or creating an NFT granted many agencies of ability to stand toe toe-to-toe with these figures, even though most are really just doing sales and marketing.

And so, I treat NFTs for what they are. A means to bring my work out to the public, while having some improved control over the intellectual ownership of it, and maybe make some pocket money out of it. Nothing more, nothing less. Cheers~

My First NFT

Sun, 06 Oct 2024 00:00:00 GMT

[Repost: This was first published in another blog on 28 Sep 2023.]

This was a ride…
To get an NFT minted required a lot of steps…:
1. Research a way to exchange Fiat currency for Ethereum,
2. Research which NFT marketplace to list your NFT on,
3. Research whether to list your NFT priced in Ethereum or Polygon,
4. Discover that you have to pay this thing called ‘gas’ and the price can fluctuate from $50 one moment to suddenly $200 an hour later…,
5. Still not finished…

6. Then once you’re ready to actually mint, realize that you can put ‘Traits’ and ‘Properties’ onto your NFT, so you research whether you need to add these things or not…
7. Then you realize that if you just mint the highest resolution version of your art, someone can just copy the image, so you realize that you can create ‘unlockable content’,
8. So you go through the hundreds of tutorials on youtube on how to host this ‘unlockable content’ online and discover ‘IPFS’ and upload into that…
9. Then once you’ve settled down and be ready to mint your NFT, be overwhelmed at how other NFT creators create tens of thousands of NFTs for a single collection at once, and here you are uploading a singly humble cleaned up scan of your hand drawing,
10. Once you’ve uploaded the file, inputted the description and properties, pasted the link of your unlockable content… finally… I got my first NFT.

So… 10 steps. Your mileage may very~

Also… link to this NFT collection of mine:
https://opensea.io/collection/jb-barcoma-sketchbook

Learning Data Engineering

Sat, 05 Oct 2024 00:00:00 GMT

Something was bugging me…

While working on my Coffee Vending Machine Business Analytics project, it really bugged me that it felt contrived and artificial.

The reason being, I found the dataset while browsing through Kaggle, downloaded it, imported it into PowerBI and went my merry way doing data exploration.

But I know it isn't that easy in the real world.

I wanted to learn about the challenge of getting the raw data right from the source and automatically feeding it into PowerBI, Tableau or any other sources.

Idea

I was already working with transaction data from the vending machine so I looked at other datasets in Kaggle. These were my thoughts with what I found:

I could download a supply-chain data set, with different tables of transaction data of different warehouse. I can then do machine learning and optimization as an end goal. Pipeline-wise I just need to find a way to artificially re-stream the data somehow.
I could just find a more-raw sales transaction data set, and 're-stream' the data somehow.
After those two ideas, I quickly realized that the common theme was finding a way to 're-stream' the data. So I tried reading about it.

Looking at job listings for ideas

It would be painful to learn about a tool only to find out that no employer in the job market is looking for it. So browsed through some data analyst/data engineering job descriptions to look at the current tools.

The top data engineering tools were: Spark, Airflow, Databricks, Hadoop, MongoDB, DBT, Kafka, Snowflake, Talend, DataStage, AWS Redshift, AWS Glue and more…

At this point I have no idea what most of these are, save for MongoDB and AWS.

So to speed up my research, I did something…

ChatGPT

Instead of spending a full day opening 20+ tabs of google search results, I asked ChatGPT for the current industry standards and how to I can simulate one for myself.

Some of the prompts I started with in one particular chat:

"Loading the transformed data into the data warehouse (Amazon Redshift/Google BigQuery/Snowflake) using Talend/AWS Glue. Which is compatible with which?"
"What other database-ETL-warehouse tech stacks are there?"
"I want to focus on Kafka, Spark, Hadoop, AWS Glue, Redshift, Snowflake, Redshift, Bigquery, Data bricks. Give stacks involving these and appropraite datasets/industries to process."

and one of the answers:

Aftermath of going down the Data Engineering rabbit hole

To get a handle of the black-hole of information ChatGPT threw at me, I tried visualizing them with Miro. Here are how my charts progressed:

Don't worry… I learned and enlightened myself… ver. 2:

I decided that I will just try to focus on the Tableau delivery out of the five possible.

Going to the local library (thanks NLB)

I got acquainted with "The Fundamentals of Data Engineering" by Joe Reis & Matt Housley (O'Reilly).

It is a great book and covers every data engineering concept to get started and to get advanced. Highly recommended.

To be continued…

This is not easy…

(Link to Part 2)

Solved: CI/CD to AWS EC2 with Github Actions

Sun, 10 Dec 2023 00:00:00 GMT

I've finally gotten CI/CD setup with AWS EC2 for the second time, after doing the same with Hostinger.

I followed this tutorial Hosting Your Resume on AWS EC2 with a CI/CD Setup Using GitHub Actions | AWS Project Demo and got it working.

I wasn't sure why it wasn't working before the first few weeks of my time with AWS, but got it to work now.

The exact steps I did to get this to work was:

Create another EC2 instance to experiment on.
Create a dummy website and github repository to push to the EC2 instance.
Followed the youtube tutorial, keyed in the github secrets, worked out the yml file and tested it. Worked on the first try, so tutorial instructions were good.
Next, applied the same methodology to the first EC2 instance that my active website is on.
After some tweaks to the linux command scripts in the yml file, I got it to work.

Next task is to docker-ize the website, or the ML apps that I've yet to deploy to the server.

Solved: Linking a Namecheap domain to an AWS EC2 Instance

Fri, 08 Dec 2023 00:00:00 GMT

The issue was that my website (jbrienweb.dev), being a .DEV website, HAS to have HTTPS, which means installing SSL certificates on my EC2 server.

To get that information, I contacted Namecheap customer support, where they informed me of the above requirement. Thanks Namecheap, response was quick (within a couple of hours).

So I followed the last few minutes of this tutorial: https://www.youtube.com/watch?v=8Uofkq718n8

Installed SSL certificates with Certbot; got more help from ChatGPT and used :

sudo apache2ctl configtest

To troubleshoot root directory issues and voila, finally got my website to work with a Namecheap domain name.

Moving from Hostinger to AWS EC2 with a Namecheap domain name

Thu, 16 Nov 2023 00:00:00 GMT

I started hosting my web development practice site on Hostinger, with a domain name bought from Namecheap, following a 'reddit tip' that I read to not buy a domain name and host the website with the same company.

Exploring this topic of web development and hosting even more, I discovered that I may want more funcitonality with my website.

I just learned what it means to be a 'Back-end' developer vs a cloud developer/engineer: Back-end still deals with the code that you write on your IDE, but more towards managing the flow of data between the server and interface, say the comments on a comment box being collated in a database when you click the 'submit button'.

A cloud developer deals with the monster called 'Amazon Web Service (AWS)', which is actually more of an umbrella term for a bunch of products and services, rather than a single product.

AWS to the cloud industry is what Autodesk is to the CAD industry: with Autodesk you have Revit, Fusion 360, Maya, 3DS Max, Navisworks, etc... basically product names which don't tell you what they can do for you, so you have to read up on them to find out what they do. I'm not going to cover what products AWS offers, because it is huge and confusing, but I will talk about the Elastic Compute Cloud (EC2).

Amazon Web Service Elastic Compute CLoud (AWS EC2)

This is basically a virtual machine, which in other words is a computer virtually created by Amazon in their data centres, the screen/terminal of which you can access from tyour web browser. When you create the 'instance' which is what each virtual computer is called, you can specify whether it should install a Linux system, a Windows system, and many others.

Following the typical youtube tutorial (there are many of them), and chatgpt answers, I went with Ubuntu with Apache2.

There are other services like Lightsail, S3, Bucket, but I felt that EC2 would allow me to expand, since it literally is a computer of its own, and I can install whatever software the linux OS allows me to install. This makes way for opportunity to do fancy over-engineered stuff with my practice website like Machine Learning apps, heavy animations, games, etc...

Hostinger

I made the mistake of subscribing for 4 years, paying up front about USD 180, which is 'relatively' cheap, but also 'relatively' expensive if you consider that you could be paying zero dollars, and I felt that I may outgrow it within a year. Thankfully Hostinger allows you to apply for refund within 30 days from making a purchase. I will still consider Hostinger for simpler, lighter static websites that I may need in the future, since a full blown linux machine is a bit overkill if I just want to host a blog, the complexity is just as such:

Setting up AWS EC2 for Github auto-deployment

Is a pain in the ***e.

With hostinger I just need to copy a bunch of lines to github and hostinger, allow webhook, and done.

First, you have to create the virtual machine 'instance', which have a lot of options. Choose operating system, setup the firewall (if you mess up, you won't know if it's the initial setup that is the problem, or a config file in the pc, or your domain name, etc).

Second, you have to install and configure the Apache2 server software. You have folder structure and config files that you have to manually create yourself.

Third, get your website files into the EC2 instance.

If you want to do it via github, you need to use this thing called 'Github Actions', which is a rabbit hole in itself. Then you have to play with SSH Key Pairs, which are basically a pair of codes that you have to key in to github and make sure is in your EC2 instance as well otherwise EC2 will not allow github to connect.

If you firewall settings is wrong, this will fail; if your Apache2 config is wrong, this will fail; if your Github/EC2 key pairs are wrong, this will fail; if yoru Github Actions setup is wrong, this will fail. My setup is still failing, so for now I just cloned the github repo manually from the EC2 instance itself, and moved it to the proper location.

Pointing the Namecheap domain name to the AWS EC2 server

There are a couple of ways to do this as well.

Copy the public IP address of the EC2 instance and go to the advance DNS setting of the namecheap domain, paste it in for a 'DNS Type A record', or:
Setup a 'hosting zone' in another AWS service/product called Route 53, which is basically just AWS' version of their own Hostinger or Namecheap daomin/hosting service. Then, this hosting zone will give you the nameserver address, which you add onto the namecheap domain settings.
Just create a new domain name with Route 53.

Three options: one involves one service, next only involves two services, another involves three. I don't know the pros and cons, but one thing I do know is that I still can't connect to my website via the domain name. But it's only been less than 48hrs, which according to dnslookup, the internet and chatgpt, is about the time needed for the DNS to 'propagate'. So I'll wait 2 days and check again. In the meantime, my website can still be viewed via the Public IP address of the EC2 instance, so it's what I'm sharing right now.

The end~

Web Hosting & Web Development

Sun, 12 Nov 2023 00:00:00 GMT

It all started with NetworkChuck, the 'Coffee.Hacking.Tech' youtuber.

I think it was me wanting to try out virtual machines since I've been curious about dual-booting operating systems, and isolating or compartmentalizing my pc, since from the age of 12 I've been torrenting all sorts of stuff.

Well, I've never had my PC infected despite that, even though I was only using Windows Defender (I got sick with the Norton Anti-Virus pop-ups, iykyk).

Ah yes sorry for the tangent, here's his video about creating your own website: You need a website RIGHT NOW!! (create a website for FREE)

Attempt #1: Wordpress

I mean it's good: free domain names, free hosting; but I wanted to do more things than just blog and build a portfolio, and even the website has some quirks...

For some styles, you are going to have to pay, like getting a black background colour for your website for example... So if I'm going to have to pay, might as well pay for a proper domain and website.

It was a good place to start things for free, and you get a taste of the kind of work flow and effort that you need to get a bunch of webpages together.

Attempt #1 product (1 out of 5 odd webpages I fumbled around with):
jbtest69.wordpress.com

Attempt #2: Github

I've been doing Angela Yu's Web Development Bootcamp on Udemy, sponsord by Singapore's National Library Board.

Attempt #2 product:
jbgithub22.github.io/jbrienwebdev/

Attempt #3: Hostinger

Recommended by NetworkChuck, so I went ahead with it.

Paid for a 4 year package, which is about USD 180... but then I read in reddit that having your domain name and hosting at the same company isn't as good as an idea as it seems to be.

Attempt #3 product:
cezarbarcoma.net

Attempt #4: Namecheap + Hostinger

So I went to Namecheap, and tried to register 'jbrienwebdev'. It suggested 'jbrienweb.dev' which I thought was cool, so I bought it. It was about $15/year.

I could have bought a domain in Hostinger for 0.99cents/year, but then it renews to $50++ the year after, so nah.

Attempt #4 product:
jbrienweb.dev

Attempt #5: Namecheap + Hostinger + Github

I haven't actually tried this, since I'm still going through Angela Yu's course. I'll update this some time later.

Attempt #6: Namecheap + Hostinger + Github + Wordpress

I wasn't intending to write about Attempt 5 and 6, but while watching Programming Is NOT Enough | Add these 7 skills…, I thought... It should be possible to use github to update the files on the host right? I guess that's basic... Then I though... what if I could combine everything that I've writtent about so far into Attempt #6, and lo and behold you can: WordPress GitHub: How to Incorporate and Use GitHub for Better Website Development

Hello World

Tue, 31 Oct 2023 00:00:00 GMT

Let's get the cliche out of the way.

I don't know how many draft wordpress websites I've done now at this point. A quick glance to a new tab shows 5 wordpress websites, including a test website that wordpress suggested me to name as 'jbtest69.wordpress.com', which is as nice and simple as it gets.

So what's going to be in this little nook of the internet:::::

Servers and Homelab

I just got to building my first DIY RAID 6 NAS using TrueNAS. My first PC build was a Intel Xeon E3-1245v2 CPU on an Asrock H77 Pro4/MVP motherboard, inside a Fractal Design case (probably the first gen). This was in 2012, built for my 'Architectural Career' as an 18 year old. This is the system that I'm using for the NAS.

After that I moved to a Threadripper 1950x, just a couple of months after release. On an Asrock X399 taichi with 4 PCIE slots, because you know, 'future-proofing'. 6 years later only a 1070 occupies one slot still. This I'll turn into a Proxmox server. I think there's opportunity to use this as an ML machine. Just waiting for a few more GPUs to land on my lap... and also actually get started training models...

The 3rd and current gen PC that I'm using is on an AMD 5950X. Main purpose is to run Minecraft. For games. I also play the highest fidelity and most graphically intense version of DICE's Battlefield, the indie game called Battlebit /s. At all other times I play Factorio.

But seriously though, I need to start using those cores.

Coding

As of writing, I am many things. A guy on career break; a guy not working; unemployed; traveller; and whatnot. But a person I met at a night out drinking advised me to present myself as a 'Part-Time Master of Data Science Student' and so that's what I shall be calling myself.

So I'll also be putting up some python code, maybe some R, arduino and rpi sketches, because I have a ton of unfinished IOT projects. So stay tuned~

Robotics

A remnant of my uni days. I actually think it has potential. I'm talking about my 2-wheeled table cleaning robot. This is a medium for so many things. Coding, Computer Vision, Mech & Elec Engineering, ML and so on. I've got to set up my 3D printer first.

Closing

I'm going Bali day after tomorrow this post is published. So we'll see how things go after that. Ciao~