It all depends on what you are designing. Something DSP focused like a MJPEG encoder can be entirely simulated as it can be abstracted away from anything external. Most you would need would be a simple model of your external DRAM controller for a framebuffer. And this would definitely be both the fastest and easiest way to develop it.

On the other hand, I did a job where I had to design a secure enclave accessible as a typical LBA-accessible SD card through a standard SD reader. I designed a pcb specifically to let me sniff the protocol both with a real card and my own IP. Of course, I started with implementing everything in the SD spec to the letter. It still required a crazy week of in situ debugging with dozens of card readers and cards to see why both ends violated the spec but worked anyway. In the end, I was able to design the flash translation layer fully in simulation but the SD link and phy layer were developed almost fully with the signaltap debug and compile loop.