Exactly, this is perfectly put and something I've hit several times myself. I have learned to do the exact opposite: sprint to structure the project just enough to begin to tackle the hairy stuff as soon as possible, so that any changes required for the hairy stuff are being made without a ton of other code and tests to update along the way.
Recent example: the architecture that worked well to generalize validations & mutations for individual records of different types had to change drastically to support relational cascades between multiple records of different types, but it was only a day of refactoring to do it early, instead of a month if it had been left to later.
I see this a lot with large scale refactorings, where people usually start with the easiest case and work their way up in the complexity ladder, and by the time they reach the hairy stuff it's basically time for another refactor to account for what's been missed. I started to look at any proof of concept or refactoring proposal that doesn't take into account the complicated bits, the corner cases, with a lot of suspicion.