1
point
olliepro
Joined 29 karma
Data-Scientist > CS/AI PhD
- Do you have a better way to measure LLMs? Measurement implies quantitative evaluation... which is the same as benchmarks.
- A more sound approach would have been to do a monte carlo simulation where you have 100 portfolios of each model and look at average performance.
- Ohio bill in motion to deny AI legal personhood: https://www.legislature.ohio.gov/legislation/136/hb469
- Descript has some nice video/screen recording tools to help beginners make this kind of thing look moderately professional. I've used it for various walkthroughs in my previous place of employment.
- Probably don’t need the name of the field for ChatGPT to get it.
- The causal masking means future tokens don’t affect previous tokens embeddings as they evolve throughout the model, but all tokens a processed in parallel… so, yes and no. See this previous HN post (https://www.hackerneue.com/item?id=45644328) about how bidirectional encoders are similar to diffusion’s non-linear way of generating text. Vision transformers use bidirectional encoding b/c of the non-causal nature of image pixels.
- > any automation that requires a human staff member to intervene to complete every run is not automation
Not strictly true. Barcode readers are used by humans and are definitely automation. The ironic part though is that the automation going on here is literally object classification, which humans are good at.
The play may be to collect data and make their system better.
- sentence embedding models are great for this type of thing.
- 1 point
An interesting problem since the creators of OLMO have mentioned that throughout training, they use 1/3 or their compute just doing evaluations.
Edit:
One nice thing about the “critic” approach is that the restaurant (or model provider) doesn’t have access to the benchmark to quasi-directly optimize against.