Preferences

Maybe that particular proof wasn't in its training data, and presenting it in the input added it to the model's corpus for your session?

These models seem to work similarly with code. If you ask them to recreate something that appears in hundreds of GitHub repositories, no problem, a little bit of rubber-ducky debugging and you'll have a working solution. If you ask it for something novel, without many publicly-available implementations, the quality of its output appears to drop off a cliff.


ChatGPT's code generation seems most useful for boilerplate code. I've seen it used to great effect for scaffolding projects that use stable and established frameworks. It'll easily handle generating as much stupid fluff as needed for you to start from—but it will barely get started on your domain-specific use cases, and won't be able to implement anything novel, as you said.

We still don't seem to have reached the point where it's "intelligent" enough to generate new information. Only repeat new permutations of existing information. Obviously, this is currently a fundamental property of LLMs (as someone who's implemented one), since they are trained to learn and repeat existing data (hell, even the model I've been playing with lately is benchmarked solely on "perplexity", how well it can repeat its training data).

> ChatGPT's code generation seems most useful for boilerplate code

that makes sense, it's going to be best at the kind of code you find out there on blogs and other sources. Boilerplate or code to demonstrate a specific concept or answer a question.

ChatGPT is going to be the best at replicating its training data, because the floor is made out of floor.

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal