Yes, the two are orthogonal concepts. Text did not disappear just because we invented photography. Bayesian data analysis is for inverse problems, such as using data to learn about the properties of the system/model that could have generated the data, and neural networks are for forward problems such as using data to generate more data or make predictions.

You can use BDA for forward problems too, via posterior predictive samples. The benefit over neural networks for this task is that with BDA you get dependable uncertainty quantification about your predictions. The disadvantage is that the modalities are somewhat limited to simple structured data.

You can also use neural networks for inverse problems, such as for example with Neural Posterior Estimation. This approach shows promise since it can tackle more complex problems than the standard BDA approach of Markov Chain Monte Carlo and with much faster results, but the accuracy and dependability are still quite lacking.

General quantitative thinking, and a sense of statistics, are still valuable. If you don't learn them from Bayes specifically, you should learn them somehow. The "square root of n rule" is still a stern master. And we're still not past having to think about whether our results make sense.

[0] The rule of thumb that signal-to-noise improves with the square root of the number of measurements. Also, as my dad put it: "The more bad data we average together, the closer we get to the wrong answer."

Why would it not be? You can use big data and neural nets to fit Bayesian models (variational inference).

Foundation models can be seen as approximate amortized posterior inference machines where the posterior is conditioning on the pre-training data. However, the uncertainty is usually ignored, and there may be ways to improve the state of the art if we were better Bayesians.

Yes, because Bayes' rule is fundamental if you're reasoning probabalistically. Bayesian methods produce better results with quantified uncertainty, we just don't have efficient methods to compute them for deep models yet.

Even in this era, there are some problems for which data is extremely limited. Those IMO tend to be the problems in which Bayesian techniques shine the most.

Most definitely. Many problems do not have giant datasets and it also depends on what is your task.