40
points
smueller1234
Joined 1,094 karma
- smueller1234 parentSlight problem with that if you would like to live in a functioning, thriving democracy: democracy in the sense of "one person, one vote" requires or at least greatly benefits from a broadly educated population. It's not sufficient, but very likely necessary.
- You're right -- the theoretical particle physicists at my faculty were using Mathematica very heavily when I was still in academia and maintained a dedicated compute cluster for it.
They really did not appreciate the debugging experience, but maybe that's improved in 15 years. :)
- I realize you're making a general point about space/IO ratios and the below is orthogonal, no contradiction.
It's actually a lot less user-facing per disk IO capacity that you will be able to "sell" in a large distributed storage system. There's constant maintenance churn to keep data available: - local hardware failure - planned larger scale maintenance - transient, unplanned larger scale failures (etc)
In general, you can fall back to using reconstruction from the erasure codes for serving during degradation. But that's a) enormously expensive in IO and CPU and b) you carry higher availability and/or durability risk because you lost redundancy.
Additionally, it may make sense to rebalance where data lives for optimal read throughput (and other performance reasons).
So in practice, there's constant rebalancing going on in a sophisticated distributed storage system that takes a good chunk of your HDD IOPS.
This + garbage collection also makes tape really unattractive for all but very static archives.
- I think Chips and Cheese is more like a fine replacement for realworldtech.com sans the toxic and highly educational and entertaining forums. Anandtech was much more accessible to the general tech public, but also more commercial and thus hit and miss on the content (no judgement intended, gotta eat).
- Google's internal systems have been written against the Colossus semantics for many, many years and thus benefit from it's upsides (performance, cost efficiency, reliability, strong isolation for a multi tenant system, ability to scale byte and IO usage fairly independently, tremendously good abstraction against and automation of underlying physical maintenance, etc) while not really having too much of an issue with any of the conscious trade-offs (like no random writes).
On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)
One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.
(I work on Google storage part time, I am not a Colossus developer.)
- Concur, Colossus is one of the examples where Google built what almost feels like magic technology. I work on Google Storage (among other things), and I've wished for a Cloud offering that exposes Colossus for years.
I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before.
One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware?
Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well).
Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.
- 4% of revenue is terrifying for large corporations.
- https://www.s3ns.io/en
This is Google + Thales doing the 3rd party operator model, with the operator being a subsidiary of Thales and not Google.
(NB: I work for Google in the EU.)
- Was about to say that we managed upwards of 4k servers worth of MySQL databases (as in 4k baremetal servers worth, not 4k small VMs each having a small MySQL) using "Orchestrator" at Booking.com ten years ago. I checked if it was still kicking before writing this and found that the GitHub repo was archived last year. The next Google hit I find is this article from three days ago:
https://www.percona.com/blog/orchestrator-for-managing-mysql...
Please do some additional research into the state of maintenance of that piece of tech before jumping on it. But it certainly did a lot of powerful things for us back in the day. The automatic promotion of followers was key to our deployment.
- They make it easier, but just at a source code level. They're not a real (and certainly not full) abstraction. An example that'll be making it obvious: if you replace the underlying type with a floating point type, the semantics would change dramatically, fully visible to the user code.
With larger types that otherwise have similar semantics, you can still have breakage. A straightforward one would be padding in structs. Another one is that a lot of use cases convert pointers to integers and back, so if you change the underlying representation, that's guaranteed to break. Whether that's a good or not is another question, but it's certainly not uncommon.
(Edit: sibling comments make the same point much more succinctly: ABI compatibility!)
- It's actually quite likely something else (unless it's just an excuse to reap short term savings): in a company large enough, deciding to shift staffing from one area to another is hard. As an executive with thousands of staff, you can tell your management team to each cough up a certain number or (somewhat) suitably qualified people. But again, if large enough, incentives diverge, so you don't necessarily end up with the top talent you thought you needed for your big new thing.
An "easy" solution is to do a layoff, then open roles elsewhere, allowing for selection.
It's commonly practiced across the large companies in the industry.
(Not speaking for my employer)
- Multiple types of TPUs.
(I work for Google, but the above is public information.)
- Former Perl language contributor here. A sibling comment to this already pointed out that you must use strict mode with Perl to retain your well being.
The two languages certainly both have their terrible warts. I think in the implicit conversion gotchas, JS is actually markedly worse. Perl has polymorphic values, but somewhat typed (for its basic types) operators (eg "eq" for strings, "==" for integers). JS has both implicit value type conversions and overloaded operators. That leads to an unholy level of indeterministic mess.
- Many moons ago, I made a case for making strict mode the default in Perl. We settled on the current backwards compatibility compromise, which is that breaking changes are hidden behind a minimum version toggle:
Eg. putting "use v5.14.0;" or similar on top of your file (or compilation unit/scope) will indeed turn on strict mode for you, along with adding a number of features as well.
At the time, also auto-toggling warnings was considered unacceptable because technically, using the warnings pragma anywhere had some edge case action at a distance. This has been remedied in some later release after I wasn't involved in the language development anymore, and from some more recent version, warnings are also part of the standard import.
I imagine you (TheDauthi) already know that, though.
- It's not an idle use case by the way. mhx wrote and maintains a library that provides important backwards compatibility for native (typically C based) extensions for Perl across decades of language releases.
- See my response to a sibling of the comment you're responding to. The library had shocking code quality issues. It's unlikely that they're all peachy now.
The other side of this is that while Pieter's writing was marketing genius, it was also woefully understating the complexity of any practical use case. The way I tried to summarize that to folks who were keen to try zeromq then was that they should start at the back of the book with the most complex example, and that's by far the simplest setup that they could hope to end up with once they start thinking about putting something into production. And everything leading up to that - a book no less - was exclusively educational/toy use cases.
- Zeromq will have changed a lot since then, but some time in the 2010s, I prototyped a system using it (which was going to be a major production system in a large tech company) and had weird unexpected blocking issues with it. To debug, I sat down to read a bunch of the zeromq code, just to realize that it was using assert() to handle wire protocol errors (unrelated to the blocking bug).
I've never dropped a piece of software as quickly as that.
- "Before I built a wall I’d ask to know What I was walling in or walling out, And to whom I was like to give offense."
Only poem I can cite by heart a quarter of a century after spending time with it in school.
- I haven't ever used a 3d printer. But your comment made me realize that if PrusaSlicer is based slic3r, it's actually also using software that I wrote many, many years ago.
That's another side of open source: if you don't rely on it to make a living (though it did help in getting my first job as a developer!), there's that pure joy in seeing your software get picked up and used by others. This little discovery made my day.
- Apologies for nitpicking, but n^m doesn't mean m loops (ie n^1000 dient mean 1000 passes): that would be mn (1000n in your example). I think your intuition argument kind of breaks down there.
- It's easy enough to repeat, but I'd be leaking sensitive info that way. Sorry :/
- Sorry for a second response to the same comment. While on the topic of entertainment, consider the opposite of putting erasure codes on many weird devices!
Years ago, some colleagues and I did the napkin math over lunch to estimate what a single hard drive would look like that could store all of Google's data. This was for a humorous internal talk. The linear speeds of the outer sectors were enough to leave the solar system (not just higher than Earth's escape velocity). Good laughs were had, I think.
- I would say for sure LTO tape drives.
- You're right and it did get used for that. Unfortunately that's nowhere large enough a use case to support the entire product/business/r&d.
- Oh hilarious! The business logic for getting optimal routing for faxes was... quirky. Picking the right source country/line/carrier to get the best quality line while getting the lowest cost and balancing capacity. In the old days all held in a complex queue in a MySQL db (sigh). I recall getting paged out of bed a number of times as an escalation to get it all working again after we hit yet another lock contention issue in MySQL.
Good times!
- Booking is the largest and thus a juicy target for both attackers and articles. Having worked there, I know that competitors had the same problems though.
- Nice catch! That was indeed something we found about a decade ago when I worked there. I don't recall what the security folks tried to do about it.
Generally, though, more and more hotels shifted from faxes to API integrations via channel managers. But that's a whole extra step in sophistication.
- And even though I've not worked there in many years, I can tell you with extreme certainty that booking doesn't do that. They're also (hopefully obviously!) PCI compliant.
- IIRC when I worked at booking until 2017, they had already stopped the practice to share customer data via email for privacy+security reasons years ago.