It's rather odd to use words like "should" when you're advocating for disrespecting other people's wishes. There are sometimes reasons not to cooperate, but it seems like a good default.
The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.
DDoS remains illegal regardless of robots.txt.