Preferences

I'll go one further -

Forcing LLMs to output JSON is just silly. A lot of time and effort is being spent forcing models to output a format that is picky and that LLMs quite frankly just don't seem to like very much. A text based DSL with more restrictions on it would've been a far better choice.

Years ago I was able to trivially teach GPT 3.5 to reliably output an English like DSL with just a few in prompt examples. Meanwhile even today the latest models still have notes that they may occasionally ignore some parts of JSON schemas sent down.

Square peg, round hole, please stop hammering.


felixfbecker
MCP doesn't force models to output JSON, quite the opposite. Tool call results in MCP are text, images, audio — the things models naturally output. The whole point of MCP is to make APIs digestable to LLMs
fennecfoxy
I think perhaps they're more referring to the tool descriptions...not sure why they said output
I'm not sure about that. Newer models are able to output structured outputs perfectly and infact, if you combine it with changesets, you can have insanely useful applications since changesets also provide type-checking.

For example, in Elixir, we have this library: https://hexdocs.pm/instructor/

It's massively useful for any structured output related work.

fellatio
Any model can provide perfect JSON according to a schema if you discard non-conforming logits.

I imagine that validation as you go could slow things down though.

svachalek
The technical term is constrained decoding. OpenAI has had this for almost a year now. They say it requires generating some artifacts to do efficiently, which slows down the first response but can be cached.
ethbr1
Expect this is a problem pattern that will be seen a lot with LLMs.

Do I look at whether the data format is easily output by my target LLM?

Or do I just validate clamp/discard non-conforming output?

Always using the latter seems pretty inefficient.

This item has no comments currently.