Yes! The tiny number of particles are moving really fast, but there are very few of them. We are talking about vacuum that is less than 10^-17 torr. A thermos is about 10^-4 torr. The LHC only gets down to 10^-10 torr. At those pressures you can lower the temperature of a kilometer cube by 10 thousand kelvin by raising the temperature of a cubic centimeter of water by 1 kelvin. There is very little thermal mass in such a vacuum which is why temperature can swing to such wild levels.
This is also why spacecraft have to reject heat purely using radiation. Typically you heat up a panel with a lot of surface area using a heat pump and dump the energy into space as infrared. Some cooling paints on roofing do this at night which is kind of neat.
Suits are insulating for a reason. You want to prevent heating on the sun side and prevent too much cooling on the space side. Your body is essentially encapsulated in a giant thermos.
Cooling is achieved using a recirculating cold water system that is good for a few hours of body heat. Water is initially cooled by the primary life support system of the spacecraft before an EVA. Pretty much it starts off pretty cold and slowly over time comes up to your body heat. Recent designs use evaporative cooling to re-cool the water.
Life support systems are so cool.
Temperature is just the heat of particles moving. In the extreme case of a handful of N2 molecules moving at 1% the speed of light, it has a temperature of something like 9 billion Kelvin. But it's not going to heat you up if it hits you.
I know it is much hotter, but that's way way hotter and they only find it at a "wall" way farther out.
This is more the temperature of the solar wind, dwarfing the steady state temperature you'd reach from the photonic solar radiation at any distance. The Sun's blackbody varies from like 5000K to 7000K, you won't see objects heated in the solar system heated higher than that even with full reflectors covering the field of view of the rear with more sun and being near the surface of the sun, other than a tiny amount higher from stellar wind, tidal friction, or nuclear radiation from the object's own material I don't think.