kipukun parent
To the second footnote: you could utilize Polar's lazyframe API to do that cosine similarity in a streaming fashion for large files.
That would get around memory limitations but I still think that would be slow.
You'd be surprised. As long as your query is using Polars natives and not a UDF (which drops it down to Python), you may get good results.
A (simple) benchmark would be great to figure out where the practical limits of such an approach are. Runtime is expected to grow with O(n*2) which will get painful at some point.