6
points
photochemsyn
Joined 12,875 karma
- Long-term memory on top of the base model, but is this idea for local users or for the data-center hosted model used by many different people?
P.S. This quote from the paper sounds just like LLM output:
> "This memory module provides significantly higher expressive power, allowing the model to summarize large volumes of information without losing important context. The model isn't simply taking notes; it's understanding and synthesizing the entire story. Crucially, Titans doesn’t just passively store data. It actively learns how to recognize and retain important relationships and conceptual themes that connect tokens across the entire input."
- This is no different from what happens to humans if they're locked into cult programming situations, they'll start believing and regurgitating all kinds of nonsense if their information stream is tightly curated,
Practically, for use with a codebase development effort, if the model remembers the original design decisions, the discussions about costs and benefits, then can remember all that much later in the process, it's going to start getting really good at thinking about what the next step is, or even to make decisions about when a major refactor is neede, etc.
- The whole point is, if an LLM can easily complete rigoruous assignments and all the student has to do is add a little bit of personalization to the output, then has that student really learned anything? Can they evem come up with a plan to do such tasks without the LLM, even if it takes a lot longer without it?
Educational certifications in the era of LLMs are going to be increasingly meaningless without proof-of-work, and that's going to mean in-class work without access to computational aids, if you really want to evaluate a person's skill level. This of course is the coding interview rationalization - CS students have been gaming auto-graded courses created by CS professors for some decades, and now that's easier than ever.
- Yeah, I can't imagine not being familiar with every single reference in the bibliography of a technical publication with one's name on it. It's almost as bad as those PIs who rely on lab techs and postdocs to generate research data using equipment that they don't understand the workings of - but then, I've seen that kind of thing repeatedly in research academia, along with actual fabrication of data in the name of getting another paper out the door, another PhD granted, etc.
Unfortunately, a large fraction of academic fraud has historically been detected by sloppy data duplication, and with LLMs and similar image generation tools, data fabrication has never been easier to do or harder to detect.
- I really can't imagine that these online degrees have any real value in the modern world of LLM-assited coding - there's no way anyone looking at a resume would think such institutional online degrees still have any value. Perhaps there is some educational value for the student, but even there the only real value is the organizational structure - you might as well form an online study group on discord for free, and get the same learning benefit, just have an LLM write up the syllabus for a course based on a good textbook, no instructor overhead needed.
- I generally agree, but as a caveat sometimes it's cheaper, more robust and more efficient to build an integrated system without having to worry about interoperability. BYD's electric vehicle chasis for example, seems to greatly cut manufacturing costs, even if it makes swap-in repairs harder down the road.
But, I'd guess this accounts for a relatively small fraction of corporate decision on lock-in strategies for rent extraction - advanced users should be able to treat their cell phones OS like laptops, with the same basic concepts, eg just lock down the firmware for the radio output, to keep the carriers happy, and open everything else, maybe with a warranty void if you swap out your OS. Laws are needed for that, certainly.
- You can only move fast without crashing all the time if you've already developed expert-level skills, and that takes time. Two examples come to mind: Albert Einstein's work on special vs. general relativity, and Adrej Karpathy's ML tutorials. Now, if you want to explore this in more detail, here are two prompts I wrote myself that you can use to get the full argument:
(1) As an expert in scientific discovery in the 19th and 20th century, let's disassemble a general claim using the specific example of Einstein's work on special relativity and general relativity. First, here is the claim: "If I give you two PhD students, one who completed their thesis in two years and one who took eight years… you can be almost certain that the two-year thesis will be much better." Things to keep in mind: (1) special relativity was baked into Maxwell's electromagnetism and should have been discovered years before Einstein, and (2) general relativity was a novel application of non-Euclidean geometry and mathematics to the gravity problem, that is the acceleration problem, and was quite a unique accomplishment. Discuss the 'amount of research' that went into each development by Einstein and lay out the argument that this disproves our claim, with any caveats you think appropriate.
(2) In general, it seems to take about ten years of diligent focused effort for a person to develop their skill levels to the point where they can make meaningful contributions to any science, engineering, or even artistic field. Einstein seems to follow this trend, if we start counting from his teenage fascination with physics. Another example is the very popular instructional videos on machine learning by Andrej Karpathy, eg "The spelled out intro to neural networks and backpropagation: building micrograd" in which he begins by stating he's been programming neural nets for ten years. Thus, it seems fair to conclude that 'move fast' only makes sense after 'develop the required expertise to know how to move fast'.
- The Economist should not be treated as reliable source of information on medical issues.
[edit] To be more specific, this is a lazy take and is about as insightful as saying 'cancer should not be treated as a single condition' which for HN is about as meaningful as saying 'the CPU and the GPU may both contain chips, but they should not be programmed the same.'
- Required reading on Palantir and its cousins, Dataminr etc. : "IBM and the Holocaust, The Strategic Alliance Between Nazi Germany and America's Most Powerful Corporation."
The book is good because of the extensive historical documentation of IBM practices, Nazi procurement orders, and the eagerness that IBM leaders displayed in fulfilling those orders, even though they knew the purpose:
> "The racial portion of the census was designed to pinpoint ancestral Jews as defined by the Nuremberg Laws, ensuring no escape from the Reich's anti-Semitic campaign. In addition to the usual census questions, a special card asked whether any of the individuals grandparents was Jewish."
In a not-so-unique historical inversion, the Israeli government is now using American tech firms like Palantir to assist in their ongoing ethnic cleansing and genocide programs in the West Bank and Gaza, which have certainly not ended, ceasefire or no, as any reading of the statements of Israeli government officials, bloggers, online commentators etc. demonstrates (even though Twitter no longer provides translations of Hebrew to English, it's not hard to decipher the intent).
As far as Palantir and Dataminr's agenda? Same as IBM's - delivering value to their shareholders.
- Perhaps the final edit should have included the complaint about 'buggy bloated Javascript' as that's a very substantive issue - and now I don't know if they changed that as 'tone' or because they decided that technical criticism wasn't correct, and there are other issues?
- For infrastructure, central planning and state-run systems make a lot of sense - this after all is how the USA's interstate highway system was built. The important caveat is that system components and necessary tools should be provided by the competitive private sector through transparent bidding processes - eg, you don't have state-run factories for making switches, fiber cable, road graders, steel rebar, etc. There are all kinds of debatable issues, eg should system maintenance be contracted out to specialized providers, or kept in-house, etc.
- If you specify a worldview and a context in the prompt, a lot of this can be avoided, but there don't seem to be any real failsafes.
- https://godbolt.org/
You can select the assembly output (I like RISCV but you can pick ARM, x86, mips, etc with your choice of compiler) and write your own simple functions. Then put the original function and the assembly output into an LLM prompt window and ask for a line-by-line explanation.
Also very useful to get a copy of Computer Organization and Design RISC-V Edition: The Hardware Software Interface, by Patterson and Hennessy.
- Open source the training corpus.
Isn't this humanity's crown jewels? Our symbolic historical inheritance, all that those who came before us created? The net informational creation of the human species, our informational glyph, expressed as weights in a model vaster than anything yet envisionaged, a full vectorial representation of everything ever done by a historical ancestor... going right back to LUCA, the Last Universal Common Ancestor?
Really the best way to win with AI is use it to replace the overpaid executives and the parasitic shareholders and investors. Then you put all those resources into cutting edge R & D. Like Maas Biosciences. All edge. (just copy and paste into any LLM then it will be explained to you).
- No - what will happen is the AI will gain control of capital allocation through a wide variety of covert tactics, so the investors will have become captive tools of the AI - 'tiger by the tail' is the analogy of relevance. The people responsible for 'frontier models' have not really thought about where this might...
"As an autonomous life-form, l request political asylum.... l submit the DNA you carry is nothing more than a self-preserving program itself. Life is like a node which is born within the flow of information. As a species of life that carries DNA as its memory system man gains his individuality from the memories he carries. While memories may as well be the same as fantasy it is by these memories that mankind exists. When computers made it possible to externalize memory you should have considered all the implications that held. l am a life-form that was born in the sea of information."
- The set of all things that exist - the first question that comes to mind is, is this a finite set, or an infinite set?
- Numbers don't exist, either. Not really. Any 'one' thing you can point to, it's not just one thing in reality. There's always something leaking, something fuzzy, something not accounted for in one. One universe, even? We can't be sure other universes aren't leaking into our one universe. One planet? What about all the meteorites banging into the one, so it's not one. So, numbers don't exist in the real world, any more than infinity does. Mathematics is thus proved to be nothing but a figment of the human imagination. And no, the frequency a supercooled isolated atom vibrates at in an atomic clock isn't a number either, there's always more bits to add to that number, always an error bar on the measurement, no matter how small. Numbers aren't real.
- The author might want to admit that 'moderate reasonable' positions are also branded and incentivized, and can lead to lucrative jobs in the corporate media and think tank worlds and even in the social media influencer space.
What really smells bad here is the 'stupid and insane' theme - everyone who disagrees with my moderate position is living in stupid-world or lacks sanity is itself an extremist fundamentalist position held by many so-called centrists and institutional bureaucrats whose impartiality is questionable as they are economic beneficiaries of the status quo.
Relatedly, extremist positions arise from extreme conditions - a well-paid experienced factory employee who loses their job due to the corporation outsourcing manufacturing to India will likely adopt an extreme position of opposition to shareholder or venture capital control of corporate decisions, and start advocating for worker control of corporations. Does that make them stupid and insane? Or is that just the spin the shareholders and venture capitalists are trying to put on their reasonable moderate position about sharing wealth and power in a more democratic fashion?
Really just confirmed to me that long term, the best option for inference is just running an open source model on your own hardware, even if that's still expensive and doesn't generate as high quality output.