immibis parent
The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.