Preferences

I recently moved to data engineering role where everything uses GCP services (think BigQuery, DataProc, Cloud Storage, ...) and wondered is all that was really necessary?

What would be the simple yet robust infra for data eng? Not thought a lot about it for now, so I am curious if some of you have would have any insights.


The same thing that happened to devops from 2017-2024 (see: https://logical.li/blog/devops/) is happening with dataops. Hype train and jargon based decisions are taking place.

In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".

Decisions are sticky so take context into account.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal