With this being metal, small defects tend to grow over time- called metal fatigue. This all depends on the stress in the material which depends on a lot of things: fluid flows and pressures, temperatures (and temperature differences and rates of change), vibrations (structural or induced by fluid flow), etc etc.
Mechanical engineering at this level is very complex! So many of these things boil down to probability distributions of each process involved. I don't know how "close to the edge" the design is, but this is the kind of thing you have to do when optimizing for weight.
Very much so! I have seen hardware being accepted and rejected on the basis of those limits. There were also cases where medium sized defects were reworked and rectified after careful assessments and reviews.
> With this being metal, small defects tend to grow over time- called metal fatigue
The tank and feedline welds don't cycle too often. But that also makes it very critical in reusable rockets. It may fly fine ten times and then show up unexpectedly.
> I don't know how "close to the edge" the design is
That's the fun part! As you suspect, they have ridiculously low structural margins in order to optimize for mass. What it means is that many physical phenomena that you don't usually worry too much about (when margins are splendid), suddenly turn into critical issues. Then you're off to doing doing materials research and other scientific studies, instead of doing just design and engineering (rocket engineering suddenly becomes rocket science. literally!). I've seen cases where the engineers were forced to study the algorithms used by finite element analysis software used for structural simulations. It can get that 'close to the edge'!
I don't know if they use manual welding or robotic welding. But robotic welding is well established and is justified for the volumes of work that SpaceX carries out. What is more difficult is to avoid vertical weld seams on its cylindrical segments. I'm yet to encounter a roll forged cylinder that big, especially with stainless steel. (Disclaimer: I have no direct experience with industrial metalworking)
> If any failed weld can lead to a catastrophe like this, how would you guarantee the quality of each weld without going into nuclear power plant construction level of costs?
That is done using Non-destructive testing (NDTs). The usual methods are high-energy X-ray imaging, ultrasound testing (UT) and dye penetration testing (DP). These methods are usually reliable in catching such faults - even for machinery that's in use. For example, turbine blades in a jet engine.
Updated: As the other commenter pointed out, robotic welding doesn't ensure elimination of defects. The robotic process is more consistent and therefore reduces the defects. But uncontrollable random variations can still cause defects that could fail later. The only way to eliminate them (almost) entirely is to identify them with NDT and rectify them as long as the defects are within a certain tolerance limit.