- What I would give for my org to just buy an oxide rack and run everything license free on there. Shame the only people with access to our CIO are salesmen…
- >when the research topic is preassigned
Well, when you have grant money for a project on X most supervisors will not let you do Y. Most students wouldn’t do research on Y if they weren’t going to be funded. I work in a lab where everyone else has to play ball by their grant proposals and you can sense a general lack of genuine curiosity. Which makes sense, they are not really different than a contract employee with a very specific deliverable that was designed before they even showed up. I can’t speak for other fields but especially in biomedical/compsci where your peers are making six figures working for graduate pay for 2 years (MSc) and then another 4-6 (PhD) doesn’t motivate you to engage outside of your exact degree requirements and your project. Add on that “curious” research doesn’t have a guaranteed path to publishing or to success and it suddenly becomes less appealing to gamble your future on such a thing. I would label my own research as “curious” in that I have support from professors at a few universities but on the whole we are facing challenges from academia at large. The only reason I can comfortably pursue something that has a genuine non-zero chance of failing into obscurity is that I am funded by an in-house university scholarship and I have a full time job.
- “Sports are the opiate of the middle class” as a reframe sucked all the fun out of sports for me. Overvalued to watch in person, too many ads to watch digitally. Not to mention having sports betting rammed down my throat. Professional sports is like any other medium from my childhood, monetized into the abyss and not as fun to interact with.
- The utility of LLM's in browsers escape me. What benefit is there? What 'tasks' does the average person use browsers for that the model can help with? I'm trying to think of what part of my daily routine would this benefit me and I'm drawing a blank...
- Similar situation. I work for a provincial government and make €61k, my scope is actually relatively large for how long I’ve been with my team but the actual problems are simple enough that some decent code means I have 0 downtime. As a result if I don’t bug anyone I typically get left alone to manage a bunch of products that run without issue. This week I literally have no meetings on my calendar, just a small project with a generous due date where I’m the solo developer.
I’m lucky in that before I got the job I was in talks to do a PhD but negotiated saying I’d only do it remote.
Now I do whatever is required to keep my day job happy and then spend the rest of my time working on my PhD. My plan was to go to FAANG after I got my degree but who knows… a comfy, unionized tech job that gives me ample time to do side projects is also not something I’d give up too easily.
I’d say do whatever is necessary to keep your job and then devote any extra hours 9-5 to some project. If I wasn’t doing my PhD I’d be making an app or a game probably, or maybe still moonlighting as a researcher. I think most office/tech jobs don’t require your full 40 hours and I can tell you I have a bunch of friends who have even less work responsibilities than me but they just use that spare time to play video games. Just do something productive 9-5 and you will outpace 99% of people is what I’ve found.
- >Can you choose the book you want, or does it have to be introductory Java.
2nd edition of Kleppmann comes out in a few months... if I flunk a DE interview think I can request it?
- > LLMs can't do math.
Ignoring conversations about 'reasoning', at a fundamental level LLMs do not 'do math' in the way that a calculator or a human does math. Sure we can train bigger and bigger models that give you the impression of this but there are proofs out there that with increased task complexity (in this case multi-digit multiplication) eventually the probability of incorrect predictions converges to 1 (https://arxiv.org/abs/2305.18654)
> And your 2nd and third point about planning and compounding errors remain challenges.. probably unsolvable with LLM approaches.
The same issue applies here, really with any complex multi-step problem.
> Again, mere months later the o series of models came out, and basically proved this point moot. Turns out RL + long context mitigate this fairly well. And a year later, we have all SotA models being able to "solve" problems 100k+ tokens deep.
If you go hands on in any decent size codebase with an agent session length and context size become noticeable issues. Again, mathematically error propagation eventually leads to a 100% chance of error. Yann isn't wrong here, we've just kicked the can a little further down the road. What happens at 200k+ tokens? 500k+ tokens? 1M tokens? The underlying issue of a stochastic system isn't addressed.
>While Yann is clearly brilliant, and has a deeper understanding of the roots of the filed than many of us mortals, I think he's been on a debbie downer trend lately
As he should be. Nothing he said was wrong at a fundamental level. The transformer architecture we have now cannot scale with task complexity. Which is fine, by nature it was not designed for such tasks. The problem is that people see these models work on a subset of small scope complex projects and make claims that go against the underlying architecture. If a model is 'solving' complex or planning tasks but then fails to do similar tasks at a higher complexity it's a sign that there is no underlying deterministic process. What is more likely: the model is genuinely 'planning' or 'solving' complex tasks, or that the model has been trained with enough planning and task related examples that it can make a high probability guess?
> So, yeah, I'd take everything any one singular person says with a huge grain of salt. No matter how brilliant said individual is.
If anything, a guy like Yann with a role such as his at a Mag7 company being realistic (bearish if you are a LLM evangelist) about what the transformer architecture can do is a relief. I'm more inclined to listen to him than a guy like Altman who touts LLMs as the future of humanity meanwhile is path to profitability is AI Tik-Tok, sex chatbots, and a third party way to purchase things from Walmart during a recession.
- I thank the stars every day that my direct manager is an actual engineer.
- People love gambling.
- The flurry of OpenAI deals make me think were reaching a tipping point maybe a lot sooner than I expected. This is a company that needs to show some viability of generative AI as a product and so far they've created AI-Slop TikTok (which the inference to run must be a huge negative for them) and a third party way to purchase things online. Consumer spending is so low that Costco isn't putting out really any Christmas merchandise this year and one of ChatGPT's avenue for profitability is going to be as an intermediary for consumer purchasing?
- I know a guy who does this kind of contract work for Python/C++ programming. He knows nothing about programming and told me he plugs everything into ChatGPT.
- I have a CS undergrad and a MSc in Computing, did either really prepare me for industry? Not really. But I made a lot of friends, drank a million beers, and overall had a good time. They look good on a resume too.
Even when I was in school I knew that CS as a field wasn't one where you could just get a degree and get a job (well outside of 2021 maybe). After one internship I realized there was a great divide between school and work, and this was at a work place with pretty low code quality.
'Self-taught' engineers is a bit of a funny term for me because while I technically have a CS degree, everything applicable I had to learn myself. Especially for a DE job where outside of some basic SQL and the general idea of a relational database a post-secondary degree doesn't exactly prepare you for such a role.
- I think this is my favorite part of the LLM hype train: the butterfly effect of dependence on an undependable stochastic system propagates errors up the chain until the whole system is worthless.
"I think it got 98% of the information correct..." how do you know how much is correct without doing the whole thing properly yourself?
The two options are:
- Do the whole thing yourself to validate
- Skim 40% of it, 'seems right to me', accept the slop and send it off to the next sucker to plug into his agent.
I think the funny part is that humans are not exempt from similar mistakes, but a human making those mistakes again and again would get fired. Meanwhile an agent that you accept to get only 98% of things right is meeting expectations.
I always like the phrase, "follow the money", in situations like this. Are OpenAI or Microsoft close to AGI? Who knows... Is there a monetary incentive to making you believe they are close to AGI? Absolutely. Take in this was the first bullet point in Microsoft's blog post.