This is something I've been thinking too (though it seems clear to me it's only one factor out of many). I've heard at least two anecdotes from trans women I follow on social media that led me to this theory. One was prescribed estrogen doses too infrequently, and the other had lost access to medication, and they both started feeling like total crap when they went off estrogen, without any associated perceptible changes to their bodies.