I think what you are saying is true for CLI-only development using Swift. It is possible, but LLMs often get the commands wrong or don't realize how to accomplish something. There have been a number of times when claude/codex has told me I have to edit a plist manually in XCode before progress can continue.