Yeah when I implemented a RAG myself I wondered why people were storing the text separately, it doesn't make any sense to me!

It's not that "vector databases are the wrong abstraction", it's that "vector data is not an abstraction at all". It's just a data type with some operators, you are responsible for architecting that tool into your system in a coherent way.