Unix licensing costs almost definitely.
A/UX was forked from UNIX System V Release 2.2, with "additional features from System V Releases 3 and 4 and BSD versions 4.2 and 4.3". It also had a TCP/IP stack, since before MacTCP (if I recall correctly).
I don't have any direct knowledge of Apple's licensing issues with AT&T, but I worked for SCO in the early 90s, back when it was a UNIX OEM, not a litigation zombie.
AT&T licensing fees for UNIX back in the late 80s/early 90s were fiendish; that's the whole reason for the ascendancy of Linux today. (Linus couldn't afford his own Intel UNIX shrinkwrap, so he began rolling his own kernel.) I heard a ballpark figure that SCO paid out $200 in royalties every time they sold a copy of Open Desktop 3.0, which included royalties on SVR3.2, a third-party TCP/IP stack, X11 (which was free), and OSF Motif (which wasn't) and IXI's X Desktop (the desktop and utility apps build with Motif/X11). This drove the price of a copy of ODT 3 up to the $1000-2000 mark depending on trim. SCO also paid royalties to Microsoft every time we sold the developer tools, because SCO had inherited the MS C compiler by way of their Xenix license in the late 80s, and SCO SVR3.2 was built using MS C.
The AU/X "additional" features from SVR 3 and SVR 4 was probably the result of a similar process to SCO's cloning of SVR4 features in the early 90s. SCO's UNIX license only covered SVR3.2. AT&T jacked their license prices for SVR 4 so much that going to true SRV 4 would have drastically raised the price of Open Desktop. It worked out cheaper to assign a couple of hundred developers to work for 3-4 years at cloning SVR 4.3 (needed to hit the standards compliance checkboxes required by customers), while keeping the AT&T SVR3.2 copyright declarations in the header files (and on the box).
My guess would be that Apple hit the same wall that SCO hit after 1995; AT&T licenses were too expensive and cloning features and bolting them on top of an ancient code base was also becoming expensive. (As I remember it, SCO bought a license for SVR 4.3 in the end and switched to that, some time in the mid-to-late 90s, too late to dodge the bullet of Linux gobbling up the Intel/Posix world.)
It took a long time for Mac developers to really buy into Cocoa, many of them saw it as a less-capable version of existing APIs rather than the future of Mac development. That viewpoint wasn’t entirely without reason, at first, but the longer you spent with Cocoa, the more clear it became that Carbon was a second-class citizen.
My other thought is that Apple didn’t just want tech, but the experience to develop it. That’s why acquisitions were at the top of the list. Apple has certainly bought a few other companies just to access their engineering talent.
1. Switch to using the Carbon API, and create a "carbonized" application that works on both Mac OS 9 and Mac OS X (same executable for both platforms, basically a special executable format and some restrictions on how you used various APIs)
2. Once you don't care about Mac OS 9 support, migrate to Project Builder (or at least GCC)
3. Eventually, migrate to Cocoa
The path was set up so that you could stop at any point along the way and keep shipping your application. In my personal experience, steps #1 and #3 could be difficult and labor-intensive, but step #2 was usually easy.
My understanding is that Quark got stuck and took too long to complete step #1, long enough that their customers didn't want to keep Mac OS 9 around just so they could use QuarkXPress and switched to InDesign.
Step #2 mostly involved fixing the parts of your code which used weird, non-standard C extensions that GCC did not support. Things like:
(long)*ptr = x;
That's hard to side with, when preemptive multitasking and protected memory wasn't a thing in Classic Mac OS. Those were some big sticking points when trying to Get Shit Done on a Mac, especially in the late nineties.
I agree Cocoa was amazing tech, but I don't see the sluggish migration from Carbon being because people didn't buy into it. I think it's more that Carbon worked, and there's just other things to put the expensive engineer's time to good us on.
My experience with the Mac developer community at the time was that someone new would come to the mailing lists and ask whether they should use Cocoa or Carbon for developing. These are people who don't have any legacy code and don't have a bunch of skills they've built up, working on greenfield projects.
A common response was along the lines of, "Carbon is a Serious API for developing Serious Applications, and Cocoa is a kids tool for making toys."
They probably looked at projects like FreeBSD at the time (1997-2000 era) and decided that too much time would be required to re-implement what they wanted to do, vs buying NeXT.
Only the NSA was exempt from this requirement, and they did buy a lot of NeXT's equipment.
NeXT was never about UNIX, the compatibility that it had was just to ease the porting into the platform, not to get away from it.
The Objective-C APIs were what mattered.
Likewise when it was reborn as OS X.
I attended an Apple session at CERN during the early OS X days, with the same idea being transmitted.