Very true, I just rolled out annotations for Kavita (a self-hosted book/comic server) and epub doesn't have the ability to store it in the file (although Kavita has a no-modification policy).

Although for cases like Kavita, storing in the file would be problematic if multiple users want their own annotations without concerns of data leaking.