>The second incorrect method to save a matrix of embeddings to disk is to save it as a Python pickle object [...] But it comes with two major caveats: pickled files are a massive security risk as they can execute arbitrary code, and the pickled file may not be guaranteed to be able to be opened on other machines or Python versions. It’s 2025, just stop pickling if you can.
Security: absolutely.
Portability: who cares? Frameworks move so quickly that unless you carry your whole dependency graph between machines you will not get bit compatible results with even minor version changes. It's a dirty secret that no one seems to want to fix or care about.
In short: everything is so fucked that pickle + conda is more than good enough for whatever project you want to serve to >10,000 users.
Security: absolutely.
Portability: who cares? Frameworks move so quickly that unless you carry your whole dependency graph between machines you will not get bit compatible results with even minor version changes. It's a dirty secret that no one seems to want to fix or care about.
In short: everything is so fucked that pickle + conda is more than good enough for whatever project you want to serve to >10,000 users.