Preferences

One user is able to fit more code into LLM context window since tabs take one token and spaces take 4.

https://old.reddit.com/r/ChatGPTCoding/comments/15s62yl/save...


zahlman
Did these systems really not learn to recognize a run of four spaces as a single token, despite all the Python source code fed to them?
cap11235
For one tokenizer, a couple years ago.

This item has no comments currently.