Presumably this was all well understood from history/experience before they applied Agile approaches to space launch. What changed that made this a plausible way to proceed? Some fundamental breakthrough or just Musk's deep pockets?
I don't think anything changed fundamentally. Honestly, the design methodology for any engineering project would evolve organically to be similar to NASA's methodology as its complexity grows. Despite how it looks, Falcon 9 is actually a very conservative design that doesn't depart much from this philosophy. They did everything possible to keep the complexity low and reliability high. Merlin is a simple open cycle engine utilizing pintle injectors. The propellant combination was well known. The vehicle materials and structures were similar to traditional designs. Pneumatic systems were favored in place of pyro systems to improve reusability. Even the vertical landing technology (VTOL) was demonstrated in the 1990s (although Falcon is the first one to land vertically after reentry). The designers were industry veterans utilizing NASA's help. And they took a long time to perfect the basic technologies and had already nailed the basic platform design by the time they started experimenting rapidly with high production rates and landing. Switching to agile mode wasn't a big problem at that stage.
Starship on the other hand, was very unconventional from the word go. Stainless steel structures, methane as fuel, full-flow staged combustion cycle engines with near-limit chamber pressures, cartwheel separation maneuver (which they ultimately abandoned in favor of hot staging, with a lot of ongoing issues), use of sea-level engines in space (on the second stage), the belly-down reentry and the final belly-flop maneuver by the starship, etc. There are more than a dozen things in there that nobody has tried before. It just seems like they have too many things on their plate. Musk's deep pockets did play a big role here. But I'm going to avoid speculating on it.
> IIUC, the goal of SpaceX's "Agile" approach to lower the costs of putting payloads into orbit via launch vehicle reuse by amortizing the R&D and equipment cost over multiple profitable launches.
An issue that's deeply intertwined with reusability is reliability. We put absolute emphasis on reliability with expendable launchers. But if you think about it, this is even more paramount in the case of reusable launchers. Hardware tends to degrade overtime, increasing the chance of triggering any design or manufacturing flaws. Expendable launchers need to contend with this only once and for a short time. Reusable launchers have to deal with it over several flights, several structural and thermal cycles (causing fatigue cracks) and a large accumulated operating time. Periodic maintenance can eliminate some flaws - but not design flaws. Launchers also tend to be very sensitive to design flaws, due to the very low engineering margins available. So, there is a bigger incentive here to get it right in the first try itself. (For example, consider the fact there are fewer entities with the capability to build the common turbofan engine than to build a rocket engine.)
Another point is that the agile methodology banks on one particular behavior of software. They rarely degrade over time (unless you manage to leak resources wildly). This makes it amenable to rapid trials, failures and corrections. The cost of experimentation is also minimal, since you don't lose hardware in the process. But in hardware projects, many flaws show up only after an extended period of testing. The trials and corrections are also costly and strenuous.
> They are searching for the cheapest route to achieve their goals.
You may have realized by now what I'm trying to convey. I'm not sure if the agile method is the cheapest route in space tech.