1
point
zbowling
Joined 2,355 karma
Engineer at Meta. Formerly Modular AI (modular.com), Google (Fuchsia), and Apportable (YC W2011, acquired by Google).
- zbowling parentI built a MCP server that solves this actually. It works like a tool calling proxy that calls child servers but instead of serving them up as direct tool calls, it exposes them as typescript defintions, asks your LLM to write code to invoke them all together, and then executes that typescript in a restricted VM to do tool calling indirectly. If you have tools that pass data between each other or need some kind of parsing or manipulation of output, like the tool call returns json, it's trivial to transform it. https://github.com/zbowling/mcpcodeserver
- I wrote a better version of this idea: https://github.com/zbowling/mcpcodeserver
It works as an MCP proxy of sorts that converts all the child MCP tools into typescript annotations, asks your LLM to generate typescript, then executes those tool calls in a restricted VM to do the tool calls that way. It allows parellel process, passing data between tools without coming back to the LLM for a full loop, etc. The agents are pretty good at debugging issues they create too and trying again.
- I specifically built this as an MCP server. It works like an MCP server that proxies to other MCP servers and converts the tool defintions in to typescript anotations and asks your llm to generate typescript that runs in a restricted VM to make tools calls that way. It's based on the apple white paper on this topic from last year. https://github.com/zbowling/mcpcodeserver
- I hacked together a new MCP server this weekend that can significantly cut down the overhead with direct tool calling with LLMs inside different agents, especially when making multiple tool calls in a more complex workflow. Inspired by the recent blog post by Cloudflare for their CodeMod MCP server and the original Apple white paper, I hacked together a new MCP server that is a lot better than the Cloudflare server in several ways. One of them being not relying on their backends to isolate the execution of the tool calling but also just generally better support around all the features in MCP and also significantly better interface generation and LLM tool hinting to save on context window tokens. This implementation can also scale to a lot more child servers more cleanly.
Most LLMs are naturally better at code generation than they are at tool calling with code understanding being more foundational to their knowledge and tool calling being pound into models in later stages during fine tuning. It can also burn an excessive number of tokens passing data between tools via LLMs in these agent orchestrators. But if you move the tool calling to be done by code rather than directly by the LLMs in the agents and have the LLMs generate that code, it can produce significantly better results for complex cases and reduce overhead with passing data between tool calls.
This implementation works as an MCP server proxy basically. As an MCP server, it is also an MCP client to your child servers. In the middle it hosts a node VM to execute code generated by the LLM to make tool calls indirectly. By introspecting the child MCP servers and converting their tool call interfaces to small condensed typescript API declarations, your LLM can generate code that invokes these tools in the provided node VM instead of invoking directly and do the complex processing of the response handling and errors in code instead of directly. This can be really powerful with when doing multiple tool calls in parallel or with logic around processing. And since it's a node VM, it has access to standard node models and built in standard libraries there.
One issue is if your tool calls are actually simple, like doing a basic web search or a single tool call, this can a bit more unnecessary overhead. But the more complex the prompt, the more this approach can significantly improve the quality of the output and lower your inference billing costs.
- 1 point
- synology doens't even compete with synology anymore because all the new hardware requires locked in synology drives now.
It's creating a void that is getting filled with Ugreen, Minisforum, beelink, Aoostor for invoative platforms from China and classic competitors like Qnap, Asustor, Teramaster, etc for innovation for the small to mid-tier needs. 45drives in the larger spaces for folks wanting to manage things more on their own but have enterprise scale needs. Dell and HP have always competed on the high-end enterprise space and also becoming a better option, even though synology is so easy as an appliance.
- Check out Pixi! Pixi is an alternative to the common conda and pypi frontends and has better system for hardware feature detection and get the best version of Torch for your hardware that is compatible across your packages (except for AMD at the moment). It can pull in the condaforge or pypi builds of pytorch and help you manage things automagically across platforms. https://pixi.sh/latest/python/pytorch/
It doesn't solve how you package your wheels specifically, that problem is still pushed on your downstream users because of boneheaded packaging decisions by PyTorch themselves but as the consumer, Pixi soften's blow. The condaforge builds of PyTorch also are a bit more sane.
- Try pixi! Pixi is a much more sane way for building with conda + pypi packages in a single tool that makes this so much easier for torch development, regardless if you get the condaforge or pypi builds of pytorch. https://pixi.sh/latest/
- I worked on Mono a lot back in the early 2000s (back in the SVN days before it moved to Git, even). This move makes a lot of sense. Things evolved a lot over the years. Mono's legacy goals, which are to be a portable CLR (.NET) runtime for platforms that Microsoft didn't care about, don't make much sense today.
Mono made a lot of sense for running places where full .NET didn't, like in full AOT environments like on the iPhone where you can't JIT, or for random architectures that don't matter anymore but once did for Linux (Alpha, Itanium, PPC, MIPs, etc.). When Microsoft bought Xamarin (which itself was born out of the ashes of the Novell shutdown of the Mono effort) and started the DotNET Core efforts to make .NET more portable itself and less a system-provided framework and merge in a lot of the stuff Mono did a single more focused project made more sense.
Mono was still left out there to support the edge cases where DotNET Core didn't make sense, which was mostly things like being a backend for Wine stuff in some cases, some GNOME Desktop stuff (via GTK#, which is pretty dead now), and older niche use cases (second life and Unity still embed mono as a runtime for their systems). The project was limping, though, and sharing a standard library but different runtimes after much merging. Mono's runtime was always a little more portable (C instead of C++) and more accessible to experiment with, but we need that less and less, but it's still perfect for Wine. So, having it live on in Wine makes sense. It's a natural fit.
- As someone who used GitHub for years (2009-2014), got acquired into a big tech company for 8 years, so I didn't get to use it much, and then got to go back to a startup that used it again, it feels like a totally different product. It's a starkly different offering from that perspective.
GitHub Actions are a major part of that. Codespaces, a solution for Mac CI, that you automate so much, store secrets, drive not CI but CD straight from GitHub now, etc. It's evolved a ton. Then you have the other things they are doing as company with copilot and vscode (and formerly atom).
Sure, code search is only moderately better, and maybe some of the core git features aren't getting as much razzle-dazzle in the UI, but it's evolved a ton.
- Nah. Your objections are rooted in the very limited definition of what an OS is and what a user application model is that fits it. There is no reason why each process/app can't be sandboxed. In fact, it should be for security (we did it in fuchsia). It's actually the way things work with apps from the app store on MacOS in a lot of ways where you can't escape your jail except through what is explicitly entitled.
General purpose just means that it's generally useful for a wide range of applications. Android is that. Hell, you can find Android running small appliance server infrastructure and powering IoT devices. Even in 2024, iOS/iPadOS is general purpose at this point, and they have VERY different application models from legacy app models you find on Windows and Linux. You wouldn't not call NixOS a general-purpose OS, and it's like flatpak for literally every process down to things like bash, grep, and vim.
Snaps are fine. Aversion is only from how it was introduced to people in Ubuntu, but conceptionally, is great. cgroups wrapping user processes to box them is not only good for security but also for stability and solving dependency versioning issues. It's brilliant. It's similar to what we did in Fuchsia at Google (we took it to another level because we had no legacy to deal with).
And sure, maybe I have bad judgment on some things. I contributed a ton to GNOME in the early 2000s both code and ideas that were horrible in hindsight, but I'm not still stuck in an outmoded mental model for thinking about my user environment.
- Guess you have never used https://snapcraft.io/, https://flatpak.org/, or https://appimage.org/ because all of these do exactly that. Snap apps are straight linux containers and the others root like chroot jails I believe still.
Or even go take a look at Nix/NixOS and how they pull it off in another way. They have hermetic isolation down to a science.
Or heck, just look at what Android does, running each app under its own uid/gid, sandboxing 3rd party code, and keeping each app from reading and writing outside their little jails. Can't pollute a user directory or even write to /tmp if your user can't even enumerate it.
Hell, even built a whole sandboxing capability-based security model inside of Fuchsia at Google, which I worked on for 5+ years.
I've been building OSs for 20+ years, between Fuchsia and Android at Google and mobile/embedded products at Texas Instruments, so I hope I know what I'm talking about.
- Nah. XDG base directory was supposed to solve a ton of things 18 years ago, except none of them are hard-hitting user problems most folks really care about most of the time unless you are really anal about hidden folders in your home directory polluting things up (most people aren't), so it fails with partial support from apathy from some package devs on Linux to implement it.
Honestly, we should probably abandon the hope that we will ever get full universal adoption. It stinks even worse with the half-adoption state we are in now, with some apps writing to those directories and other apps not. If it wasn't for arch Linux folks trying to drive apps to adopt it more, I don't think it would be. Hell, default Debian/Ubuntu still doesn't set XDG_* directory ENVs by default (some flavors do), and some apps ignore the prescribed "if not set, use this default" nonsense in the spec and do what they have always done because it doesn't break compatibility for users.
Part of the spec stinks, too, like the spit between cache/config/data config where the spec has no written rules specified on when anything is expected to be cleaned up and when or what can or should be backed up by the user.
Let's move to containerizing most apps, putting everything in little jails so we don't have to deal with where they want to write to. They can have their own consistent and expected behaviors on their own island. Snapcraft, flatpack, or any of a bunch of other solutions are already available to solve this problem. Don't have to worry about what some app left behind or wrote to your system when it's all contained.
- I don't think XDG base dir will ever be fully and universally adopted. It's been years (21 years to be exact). If it wasn't for Arch Linux getting hostile to programs that don't follow (like locking permissions of the root user directory), it probably wouldn't have grown as much as it has recently to what it is now because, honestly, the spec is not actively solving real problems people have. Most of the adoption is happening because people are getting cranky about their $HOME directory layout enough that they are annoying enough to bully maintainers into supporting it, but 99% of users don't give a damn if the program follows XDG base dir or not. It's just making the things arguably "cleaner" according to freedesktop from a spec that hasn't gained wide adoption over 20 years.
I, however, don't like the loosy goosy config/data/cache split like they have it and the implied intent because not everything falls into those buckets and then you don't know the behavior of some system thing (is Dameon going to wipe my cache? are users going to copy the config folder around? what gets backed up exactly?) There is no standard for how the folders are really treated in the spec. Intentionally vague but possibly vague in ways that will break things if people make assumptions.
It's like the /usr split from the root. or /usr/local from the root. Or /opt. Left overs from a bygone era that seem to stick around. Trying to create yet another.
I reluctantly support it in our product, but only when those ENVs are explicitly set, and I 100% ignore XDG's behavior and what it says should be the default on every linux system when it's not set because it's not what other UNIX variants like MacOS expect. Just enough to keep the Arch Linux zealots from flooding our bug tracker each week, literally demanding we adopt it when they account for less than 1% of users. It's hard enough with docs to let users know where to find their files from our product on their system because users don't know what XDG is all the time and XDG for most is not the established convention they expect already. Frankly, there is too much legacy with trying to move everyone over to this spec, especially when it isn't driving some real compelling value for anyone except for folks who want a pristine user directory.
Honestly, we should rethink it entirely and move to a container-based solution where our desktop apps can't escape their little jails (almost like snap apps). Maybe learn from NIX a bit. Admit defeat that will ever get every linux user app in the world to care and adopt XDG base dir (or even user dir). It's hard enough we are supporting Mac and Windows ports that don't follow this behavior. Get away from the UNIX permission model of running the programs under the user and run each program under it's own uid. It's now android works.
- It's true that Linux has a stable-ish syscall table. What is funny is that this caused a whole series of Samsung Android phones to reboot randomly with some apps because Samsung added a syscall at the same position someone else did in upstream linux and folks staticly linking their own libc to avoid boionc libc were rebooting phones when calling certain functions because the Samsung syscall causing kernel panics when called wrong. Goes back to it being a bad idea to subvert your system libc. Now, distro vendors do give out multiple versions of a libc that all work with your kernel. This generally works. When we had to fix ABI issues this happened a few times. But I wouldn't trust building our libc and assuming that libc is portable to any linux machine to copy it to.
- And go is wrong for doing that, at least on Linux. It bypasses optimizations in the vDSO in some cases. On Fuchsia, we made direct syscalls not through the vDSO illegal and it was funny the hacks to go that required. The system ABI of Linux really isn't the syscall interface, its the system libc. That's because the C ABI (and the behaviors of the triple it was compiled for) and its isms for that platform are the linga franca of that system. Going around that to call syscalls directly, at least for the 90% of useful syscalls on the system that are wrapped by libc, is asinine and creates odd bugs, makes crash reporters heuristical unwinders, debuggers, etc all more painful to write. It also prevents the system vendor from implementing user mode optimizations that avoid mode and context switches when necessary. We tried to solve these issues in Fuchsia, but for Linux, Darwin, and hell, even Windows, if you are making direct syscalls and it's not for something really special and bespoke, you are just flat-out wrong.