Preferences

The tension comes from our addiction to FAANG-style software engineering.

You can't be a Google engineer without thinking about scale-out. However, Google et al have a different kind of engineering than most other companies: they have tons of requirements of you that make engineering hard, but they also have tons of tools and libraries that make scale easy. In these environments, it makes sense to make even the most trivial systems horizontally scalable.

Conversely, if you do not have Google's or Amazon's distributed system components, and you don't have access to expertise with those tools, scale-out is likely the hardest problem to solve. GCP, AWS and others know this, so they charge you a lot to solve the scale-out problem for you (using their internal tools).

This is the source of the tension. FAANG-style engineering is what pads your resume (and establishes you as a "smart engineer" in the eyes of people who want to work at FAANGs), and simpler systems get things done until you absolutely need to scale out.


> FAANG-style engineering is what pads your resume (and establishes you as a "smart engineer" in the eyes of people who want to work at FAANGs)

This isn't just engineers doing this though. I worked at startups where there was pressure to build really complex systems with lots of staff. This goes from engineer who is promoted for building complex "hard" systems( manager directly asked me "whats hard about" simple system you built). Managers are promoted for hiring more people. Directors are promoted for "growing their org" and so on. Its a collective collusion to create more "bullshit jobs" . VCs demand that you build these systems with tons of staff to supposedly outcompete FAANGs.

What kind of tools they have to make scale easy?
I used to work at G, so my view is based on that. The three key infrastructure services that handle this for you are: Colossus (scale-out distributed filesystem), Chubby (lock service), and Spanner/Bigtable (scale-out databases). All three of these handle concurrency for you, and as long as you are okay accepting their concurrency models, you basically get the hard parts of a distributed system done for you. Of these three, only Spanner and Bigtable have similar competitors that are publicly available.

On top of those services, Google also has distributed systems that handle higher-level things like authentication, and they have libraries that help you do things like load balancing and load shedding. To top it off, the monitoring available is top-notch, and there are lots of administration tools that allow you to make sure that your service and all of your customers are well-behaved.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal