> the 12-billion-parameter model is about 24GB in size
Probably not on the device itself but I would love that use case as well. At least going to my own server. I’d want to protect notes in particular, which is why I don’t do any cloud backup on my RM2. But some self hosted, AI assisted OCR workflows could be really nice.
if you have a 3090, you could self host
Like writing on an ePaper tablet, exporting the PDF and feed this into this model to extract todos from notes for example.
Or what would be the SotA for this application?