I begrudgingly agree. We tried to stand up a spark cluster and there are just loads of configuration and little niggles that just get in the way, like making sure everything on your cluster has the libraries they need.

Putting it on databricks most of those problems went away and we could just get in with writing the code.

Could we have figured out how to get spark working in a bespoke configuration? almost certainly. By our mid September deadline? Probably not