They're not just moving packets around, they're producing streams of data that pass through dozens of stages and end up being spit out into a DAC to produce radio signals (and vise-versa, signals into an ADC and then through many demodulation stages).

All the modulation, demodulation, framing, scrambling, forward error correction coding/decoding, etc. has to all happen continuously at the same time.

There are some open source software defined radios that can do that for one or two stations on a CPU at low data rates, but it's basically impossible with current CPUs for anything like the number of stations (phones) that are handled in one FPGA with decent data rates, latency etc.

You'd probably need a server rack's worth of servers and hundreds of times the power consumption to do what's happening in the one chip.

FPGAs are way superior running a lot of parallel pipelined logic. They can have SERDES that can communicate at gigabits per second. You can build logic that reacts in nanoseconds to external I/O with zero jitter.

Because whilst a CPU can emulate what you want, it can't do it efficiently.

Why do people use GPUs instead of CPUs?

For some workloads, an FPGA is orders of magnitude faster than a CPU.