What separates an enterprise vs non-enterprise data engineering setup is whether the data engineer has accounted for the 'Data Engineering Undercurrents'.
The most obvious one is 'Security', but as Joe Reis and Matt Housley shares with us, there are 5 others.
Here I identify the undercurrents relevant to me (which is all categories) and see how I can implement them.
External reference image: view the original diagram at O'Reilly.
Reis, J., & Housley, M. (2022). Fundamentals of Data Engineering. O'Reilly Media.
With a bit of time and effort, I expanded on each of the 6 points with the help of LLMs. For each sub-point, I then research the industry standard for implementing these concepts. This is what I came up with:
Laying it out like this has us get started on a checklist on the necessary bells and whistles that our data infrastructure has to have. It'll get filled up as the project progress.
~End