Sounds like a lot of fun, and a huge time sink.
Tldr: use OpenWRT to create a static IP reservation for a dedicated torrent instance, then use source-based routing to send traffic over a VPN running on the router.
Sounds pretty awesome. I've not dabbled with PBR on home equipment. We did use it in prior employment to force all PCI traffic over a dedicated physical network between datacenters.
Within openwrt, I’ve installed WireGuard and OpenVPN for accessing different vpns.
Finally, again in openwrt, I installed the policy based routing package[1]. This package makes it super easy to route all traffic from the transmission container to the VPN network.
There are probably many ways you could do this, but this setup is great for me!
1: https://docs.openwrt.melmac.net/pbr/