4
points
Does anyone know if there are there any plans for browsers to natively integrate LLMs, LLM APIs, or LLM models like Llama for local use by web applications?
I feel there's a large opportunity here for a more privacy-friendly, on-device solution that doesn't send the user's data to OpenAI.
Is RAM the current main limitation?
https://developer.chrome.com/docs/ai/built-in
I hope Apple will follow suit with some of their small models (https://huggingface.co/apple/OpenELM).
And then maybe even Firefox will join them...
(V)RAM+processing power+storage(I mean what kind of average user wants to clog half their hard drive for a subpar model that output 1 token a second?)
IMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts