25
points
robertkoss
Joined 169 karma
- robertkoss parentI will never understand how people honestly think that there is a such a thing as a central DB. Do you really think that Gov Agencies from all over the world deploy Gotham just connected to the internet without controlling inflow / outflow of data? I would bet money that 99% of critical systems are not even connected to the internet but air-gapped because, believe it or not, people at those agencies are not that stupid.
- Yeah, but Foundry is so ahead, not seeing DataSphere competing there honestly. The only reason is, you already are on SAP and don't want a second system.
Also the engineering / product culture @Palantir is diametrically opposed to what exists at SAP, so I favour Palantir.
- I think Foundry is insanely impressive tbh. If you set it up correctly, its insanely powerful
- Chat, what do I see here?
- The problem that this article has is essentially this:
> Thiel by contrast is profiting from the use of AI weapons targeting systems used in the Ukraine war and the genocide in Gaza.
Thiel is IMO not doing this for profit. He is deeply ideological, which should be more worrisome.
- That was a great read!
- Germany here. Seems to be global then.
- yup, it's an ad in disguise.
- You were talking about data engineering. If you do not write tests as a data engineer what are you doing then? Just hoping that you don't fuck up editing a 1000 > line SQL script?
If you use Athena you still have to worry about shuffling and joining, it is just hidden.. It is Trino / Presto under the hood and if you click explain you can see the execution plan, which is essentially the same as looking into the SparkUI.
Who cares about JVM versions nowadays? No one is hosting Spark themselves.
Literally every tool now supports DataFrame AND SQL APIs and to me there is no reason to pick up SQL if you are familiar with a little bit of Python
- That is a false dichotomy. You can use SQL tools but still have to choose the instance type.
Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.
- OG polars announcement: https://www.hackerneue.com/item?id=23768227
- Love it!
Still don't get why one of the biggest player in the space, Databricks is overinvesting in Spark. For startups, Polars or DuckDB are completely sufficient. Other companies like Palantir already support bring your own compute.
- great to hear, vite.
- I would love to have a React Copilot that has access to the console, network logs, the actual html elements, computed styling etc. + my code.
This would be such a game-changer. I am also convinced that monorepos will become the de facto standard, since it is way easier for LLMs to navigate / implement features across the whole stack.
- Its the same if you sell B2B software and have to offer SSO to your customers. Every auth provider like Auth0, Clerk, WorkOS etc. increases their prices tremendously if you require SSO...
- its pretty obvious you have neither used foundry nor gotham.
- I dont get it. Foundrys documentation is completely public, you can even sign up for the dev tier and try it out. It is not secretive at all. If there is one word to describe their products it would be ontology and literally no one has mentioned that.
And even on Gotham there is countless footage etc. on Gaia, Dossier, Meta Constellation etc.
They had to disclose this during the IDO, why are journalists just scratching the surface when discussing Palantir.
It is obviously a tech company that has a clever business model, deploying their engineers and PMs into the board room of Fortune500s and solving their problems.
Not trying to defend Palantir, but the journalistic work is just poor.
- 1 point
- 4 points
- 2 points
- I used to be a big fan of the platform because back in 2020 / 2021 it really was the only reasonable choice compared to AWS / Azure / Snowflake for building data platforms.
Today it suffers from feature creep and too many pivots & acquisitions. That they are insanely bad at naming features doesn't help either.
- Don't know why, but to me there is little that I find more interesting than axiomes in math. ZFC / Peano always fascinated me, especially in the light of Gödels incomplete theorem.
- As someone who has no touchpoints with lower languages at all, can you explain to me why those files are called c01, c02 etc.?
- 6 points
- Love it! Competition for Databricks is always appreciated and I think having a competitor that is not running on the JVM is amazing. Working with polars feels always insanely lightweight compared to Spark. If you would provide Workflows / Scheduling out of the box, I would migrate my Spark jobs today :)
- This article is just shameless advertising for Estuary Flow, a company that the author is working for. "Operational Maturity", as if Iceberg, Delta or Hudi are not mature. These are battle-tested frameworks that have been in production for years. The "small files problem" is not really a problem because every framework supports some way of compacting smaller files. Just run a nightly job that compacts the small files and you're good 2 go.