Preferences

It’s fascinating to think about what true GUI for LLM could be like.

It immediately makes me think a LLM that can generate a customized GUI for the topic at hand where you can interact with in a non-linear way.


karpathy
Fun demo of an early idea was posted by Oriol just yesterday :)

https://x.com/OriolVinyalsML/status/1935005985070084197

spamfilter247
My takeaway from the demo is less that "it's different each time", but more a "it can be different for different users and their styles of operating" - a poweruser can now see a different Settings UI than a basic user, and it can be generated realtime based on the persona context of the user.

Example use case (chosen specifically for tech): An IDE UI that starts basic, and exposes functionality over time as the human developer's skills grow.

superfrank
On one hand, I'm incredibly impressed by the technology behind that demo. On the other hand, I can't think of many things that would piss me off more than a non-deterministic operating system.

I like my tools to be predictable. Google search trying to predict that I want the image or shopping tag based on my query already drives me crazy. If my entire operating system did that, I'm pretty sure I'd throw my computer out a window.

iLoveOncall
> incredibly impressed by the technology behind that demo

An LLM generating some HTML?

superfrank
At a speed that feels completely seamless to navigate through. Yeah, I'm pretty impressed by that.
iLoveOncall
Read the code that is actually being generated. It's only the content of the page, which itself is loaded progressively.

It takes 2 seconds to generate an extremely basic 300 characters page of content. Again, what is impressive here?

It's not fast, it gives the illusion of being fast.

superfrank
I know what it's doing and I'm impressed. If you understand what it's doing and aren't impressed, that's cool too. I think we just see things differently and I doubt either of us will convince the other one to change their mind on this
hackernewds
it's impressive but it seems like a crappier UX? that none of the patterns can really be memorized
asterisk_
I feel like one quickly hits a similar partial observability problem as with e.g. light sensors. How often do you wave around annoyed because the light turned off.

To get _truly_ self driving UIs you need to read the mind of your users. It's some heavy tailed distribution all the way down. Interesting research problem on its own.

We already have adaptive UIs (profiles in VSC anyone? Vim, Emacs?) they're mostly under-utilized because takes time to setup + most people are not better at designing their own workflow relative to the sane default.

aprilthird2021
This is crazy cool, even if not necessarily the best use case for this idea
throwaway314155
I would bet good money that many of the functions they chose not to drill down into (such as settings -> volume) do nothing at all or cause an error.

It's a fronted generator. It's fast. That's cool. But is being pitched as a functioning OS generator and I can't help but think it isn't given the failure rates for those sorts of tasks. Further, the success rates for HTML generation probably _are_ good enough for a Holmes-esque (perhaps too harsh) rugpull (again, too harsh) demo.

A cool glimpse into what the future might look like in any case.

superconduct123
That looks both cool and infuriating
suddenlybananas
Having different documents come up every time you go into the documents directory seems hellishly terrible.
falcor84
It's a brand of terribleness I've somewhat gotten used to, opening Google Drive every time, when it takes me to the "Suggested" tab. I can't recall a single time when it had the document I care about anywhere close to the top.

There's still nothing that beats the UX of Norton Commander.

sensanaty
[flagged]
"Please don't fulminate."

"Don't be curmudgeonly. Thoughtful criticism is fine, but please don't be rigidly or generically negative."

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."

https://news.ycombinator.com/newsguidelines.html

danielbln
Maybe we can collect all of this salt and operate a Thorium reactor with it, this in turn can then power AI.
sensanaty
We'll need to boil a few more lakes before we get to that stage I'm afraid, who needs water when you can have your AI hallucinate some for you after all?
TeMPOraL
Who needs water when all these hot takes come from sources so dense, they're about to collapse into black holes.
cjcenizal
My friend Eric Pelz started a company called Malleable to do this very thing: https://www.linkedin.com/posts/epelz_every-piece-of-software...
whatarethembits
I'm curious where this ends up going.

Personally I think its a mistake; at least at "team" level. One of the most valuable things about a software or framework dictating how things are done is to give a group of people a common language to communicate with and enforce rules. This is why we generally prefer to use a well documented framework, rather than letting a "rockstar engineer" roll their own. Only they will understand its edge cases and ways of thinking, everyone else will pay a price to adapt to that, dragging everyone's productivity down.

Secondly, most people don't know what they want or how they want to work with a specific piece of software. Its simply not important enough, in the hierarchy of other things they care about, to form opinions about how a specific piece of software ought to work. What they want, is the easiest and fastest way to get something done and move on. It takes insight, research and testing to figure out what that is in a specific domain. This is what "product people" are supposed to figure out; not farm it out to individual users.

swader999
You bake those rules into the folders in a Claude.md file and it becomes it's guide when building or changing anything. Ubiquitous language and all that Jazz.
jonny_eh
An ever-shifting UI sounds unlearnable, and therefore unusable.
It wouldn't be unlearnable if it fits the way the user is already thinking.
guappa
AI is not mind reading.
Behavioral patterns are not unpredictable. Who knows how far an LLM could get by pattern-matching what a user is doing and generating a UI to make it easier. Since the user could immediately say whether they liked it or not, this could turn into a rapid and creative feedback loop.
kevinventullo
So, if the user likes UI’s that don’t change, the LLM will figure out that it should do nothing?

One problem LLM’s don’t fix is the misalignment between app developers’ incentives and users’ incentives. Since the developer controls the LLM, I imagine that a “smart” shifting UI would quickly devolve into automated dark patterns.

A user who doesn't want such changes shouldn't be subjected to them in the first place, so there should be nothing for an LLM to figure out.

I'm with you on disliking dark patterns but it seems to me a separate issue.

NitpickLawyer
A sufficiently advanced prediction engine is indistinguishable from mind reading :D
OtherShrezzing
A mixed ever-shifting UI can be excellent though. So you've got some tools which consistently interact with UI components, but the UI itself is altered frequently.

Take for example world-building video games like Cities Skylines / Sim City or procedural sandboxes like Minecraft. There are 20-30 consistent buttons (tools) in the game's UX, while the rest of the game is an unbounded ever-shifting UI.

skydhash
The rest of the game is very deterministic where its state is controlled by the buttons. The slight variation is caused by the simulation engine and follows consistent patterns (you can’t have building on fire if there’s no building yet).
Tools like v0 are a primitive example of what the above is talking about. The UI maintains familiar conventions, but is laid out dynamically based on surrounding context. I'm sure there are still weird edge cases, but for the most part people have no trouble figuring out how to use the output of such tools already.
Like Spotify ugh
dpkirchner
Like a HyperCard application?
necrodome
We (https://vibes.diy/) are betting on this
diggan
Border-line off-topic, but since you're flagrantly self-promoting, might as well add some more rule breakage to it.

You know websites/apps who let you enter text/details and then not displaying sign in/up screen until you submit it, so you feel like "Oh but I already filled it out, might as well sign up"?

They really suck, big time! It's disingenuous, misleading and wastes people's time. I had no interest in using your thing for real, but thought I'd try it out, potentially leave some feedback, but this bait-and-switch just made the whole thing feel sour and I'll probably try to actively avoid this and anything else I feel is related to it.

necrodome
Thanks for the benefit of the doubt. I typed that in a hurry, and it didn’t come out the way I intended.

We had the idea that there’s a class of apps [1] that could really benefit from our tooling - mainly Fireproof, our local-first database, along with embedded LLM calling and image generation support. The app itself is open source, and the hosted version is free.

Initially, there was no login or signup - you could just generate an app right away. We knew that came with risks, but we wanted to explore what a truly frictionless experience could look like. Unfortunately, it didn’t take long for our LLM keys to start getting scraped, so the next best step was to implement rate limiting in the hosted version.

[1] https://tools.simonwillison.net/

diggan
My complaint isn't about that you need to protect it with a login/signup, but where in the process you put that login/signup.

Put it before letting people enter text, rather than once they've entered text and pressed the button, and people won't feel mislead anymore.

jchrisa
The generation is running while you login, so this appreciable decreases wait time from idea to app, because by the time you click through the login, your app is ready. (Vibes DIY CEO here.)

If login takes 30 seconds, and app gen 90, we think this is better for users (but clearly not everyone agrees.) Thanks for the feedback!

stoisesky
This talk https://www.youtube.com/watch?v=MbWgRuM-7X8 explores the idea of generative / malleable personal user interfaces where LLMs can serve as the gateway to program how we want our UI to be rendered.
nbbaier
I love this concept and would love to know where to look for people working on this type of thing!
semi-extrinsic
Humans are shit at interacting with systems in a non-linear way. Just look at Jupyter notebooks and the absolute mess that arises when you execute code blocks in arbitrary order.
bicepjai
What is the mess you are referring with regards to Jupyter notebooks ?
semi-extrinsic
If you run cells out of order, you get weird results. Thus you have efforts like marimo which replace jupyter with something that reruns all dependent cells.

This item has no comments currently.