- brotchie parentYou'd think the go-to workflow for releasing redacted PDFs would be to draw black rectangles and then rasterize to image-only PDFs :shrug:
- Did a similar back-of-the-napkin and got 5x $ / MW of orbital vs. terrestrial. This article's analysis is ~3.4x.
I do wonder, at what factor of orbital to terrestrial cost factor it becomes worthwhile.
The greater the terrestrial lead time, red tape, permitting, regulations on Earth, the higher the orbital-to-terrestrial factor that's acceptable.
A lights-out automated production line pumping out GPU satellites into a daily Starship launch feels "cleaner" from an end-to-end automation perspective vs years long land acquisition, planning and environment approvals, construction.
More expensive, for sure, but feels way more copy-paste the factory, "linearly scalable" than physical construction.
- +100000 to
A hybrid of Strong (the lifting app) and ChatGPT where the model has access to my workouts, can suggest improvements, and coach me. I mainly just want to be able to chat with the model knowing it has detailed context for each of my workouts (down to the time in between each set).
Strong really transformed my gym progression, I feel like its autopilot for the gym. BUT I have 4x routines I rotate through (I'll often switch it up based on equipment availability), but I'm sure an integrated AI coach could optimize.
- The question that really matters: is the net present value of each $1 investment in AI Capex > $1 (+ some spread for borrowing costs & risk).
We'll be inference token constrained indefinitely: i.e. inference tokens supply will never exceed demand, it's just that the $/token may not be able to pay back the capital investment.
- When reading "picture an apple with three blue dots on it", I have an abstract concept of an apple and three dots. There's really no geometry there, without follow on questions, or some priming in the question.
In my conscious experience I pretty much imagine {apple, dot, dot, dot}. I don't "see" blue, the dots are tagged with dot.color == blue.
When you ask about the arrangement of the dots, I'll THEN think about it, and then says "arranged in a triangle." But that's because you've probed with your question. Before you probed, there's no concept in my mind of any geometric arrangement.
If I hadn't been prompted to think / naturally thought about the color of the apple, and you asked me "what color is the apple." Only then would I say "green" or "red."
If you asked me to describe my office (for example) my brain can't really imagine it "holistically." I can think of the desk and then enumerate it's properties: white legs, wooden top, rug on ground. But, essentially, I'm running a geometric iterator over the scene, starting from some anchor object, jumping to nearby objects, and then enumerating their properties.
I have glimpses of what it's like to "see" in my minds eye. At night, in bed, just before sleep, if I concentrate really hard, I can sometimes see fleeting images. I liken it to looking at one of those eye puzzles where you have to relax your eyes to "see it." I almost have to focus on "seeing" without looking into the blackness of my closed eyes.
- 100%, other than selling RSUs to diversify, every single other investment I have is now buy and hold.
Even did a ~7 year career detour through quant finance "if I like Software Engineering and Mathematics so much, why don't I combine the two?"
Finally realized that the best use of my time was to just to work hard at a career I deeply enjoy (Software Engineering), working on products I actually care about (not valuing arcane derivatives products), and just invest the excess in diversified index funds (with some single stock selections here and there, thanks TSLA and NVDA).
Just as engineers can get "nerd sniped", I feel like "trading" is a somewhat malevolent strong attractor for a lot of folks. Folks do need general financial literacy, but an extra hour spent per day working harder to progress day job / long term career likely has a higher net present value than trading options or crypto.
- Look at the induced demand due to Claude code. I mean, they wildly underestimated average token usage by users. There's high willingness to pay. There's literally not enough inference infra available.
I was working on crypto during the NFT mania, and THAT felt like a bubble at the time. I'd spend my days writing smart contracts and related infra, but I was doing a genuine wallet transaction at most once a week, and that was on speculation, not work.
My adoption rate of AI has been rapid, not for toy tasks, but for meaningful complex work. Easily send 50 prompts per day to various AI tools, use LLM-driven auto-complete continuously, etc.
That's where AI is different from the dot com bubble (not enough folks materially transaction on the web at the time), or the crypto mania (speculation and not utility).
Could I use a smarter model today? Yes, I would love that and use the hell out of it. Could I use a model with 10x the tokens/second today? Yes, I would use it immediately and get substantial gains from a faster iteration cycle.
- First AI thing that’s made me feel a bit of derealization…
…and this is the worst the capabilities will ever be.
Watching the video created a glimmer of doubt that perhaps my current reality is a future version of myself, or some other consciousness, that’s living its life in an AI hallucinated environment.
- Similar experience, I had a Cyrix PR200 which really underperformed the equivalent Intel CPU.
Convinced my parent's to buy a new PC, they organized with a local computer store for me to go in and sit with the tech and actually build the PC. Almost identical specs in 1998: 400Mhz Pentium 2, Voodoo 2, no zip drive, but had a Soundblaster Live ($500 AUD for this at the time).
I distinctly remember the invoice being $5k AUD in 1998 dollars, which is $10k AUD in 2024 dollars. This was A LOT of money for my parents (~7% of their pretax annual income), and I'm eternally grateful.
I was in grade 8 at the time (middle school equivalent in USA) and it was the PC I learnt to code on (QBasic -> C -> C++), spent many hours installing Linux and re-compiling kernel drives (learning how to use the command line), used SoftICE to reverse engineer shareware keygen (learning x86 assembly), created Counterstrike wall hacks by writing MiniGL proxy dlls (learning OpenGL).
So glad there wasn't infinity pools of time wasting (YouTube, TikTok, etc) back then, and I was forced to occupy myself with productive learning.
/end reminiscing
- Have done both clinical Ketamine and Psilocybin therapy.
Ketamine was very interesting. Proper completely dissociative "K-hole" experience. I feel like it helped with Anxiety, but I can't pinpoint "why" from an introspective perspective.
Psilocybin on the other hand. Was a hero dose, and I'm a changed person afterwards.
Could feel the "layers" of my identity being stripped off, almost regression to a more child-like state. Very interesting experience. Had strong synesthesia: sounds would produce colors, colors would produce tastes, fun experience.
Near the peak of the experience I had these strong recurring auditory hallucination of my mothers says all these random words from my youth, these were accompanied by strong feeling of anxiety. After a lot of post-experience integration and reflection I realized that my mothers anxiety about the world was effectively "programmed" into my brain during my upbringing. e.g. Generationally transmitted anxiety.
Therapy always talks about childhood trauma, etc, but actually experiencing it was another level, and really helped me on my journey to being a less anxious person.
Before the Psilocybin experience, I suffered from existential depression: what's the point of living if the sun is going to explode in ~x billion years. Towards the peak of the experience everything was super chaotic, I felt like I was being transported into different realities (e.g. realities with different laws of physics, or different space time geometries). This was hugely anxiety inducing and would otherwise be called a "bad trip." I felt "lost" in this sea of all different realities.
As I was coming down from the peak and started to reintegrate, I had a strong distinct sense of "coming back" to our current reality. It felt like finding a safe tropical island in a sea of chaos: e.g. our currently reality is a safe space and point of stability in a sea of chaos and uninviting realities.
I was truly, deeply, grateful to be able to return to the familiar and it made me really really deeply appreciate myself and the blessing that our reality is to us.
Post the experience I also acquired the ability to observe my emotions from a third person perspective. e.g. rather than feeling "angry" I could tag the emotion "angry" and react accordingly, almost as if I gained ring 0 access to my brain when I previously only have ring 1 access.
All-in-all probably the most profound and healing experience of my life.
1. Deeply felt and understood my anxiety was generationally passed on from my mother's anxiety, 2. Eliminated my existential depression, giving me a deep appreciation for the beauty of our reality, 3. Gave me ring 0 access to my emotions making me a much more stable, calm person. - Open question for LLMs, does creativity and new ideas come from a process or is it a laddered emergent capability.
What I mean by this, is the process of coming up with novel ideas a single capability that has to be trained and reinforced.
Or is it a ladder of capabilities of increasing complexity in that a model that could figure of General Relativity from scratch would not be able to continue the process and perhaps come up with a viable “theory of everything.”
One thing I’ve wanted to do, I’m sure somebody has tried it, is build a dataset to RL a model to be more creative: Get a human expert in a field, have them ask a reasoning model some open questions, and then have the expert look at 20 outputs and rank them by creativity / insight. Have the expert iterate and see how much new “insight” they can mine from the model.
Do this across many fields, and then train a model on these rankings.
Perhaps creativity is a different way of moving in latent space which is “ablated” from existing models because they’re tuned to be “correct” rather than “creative.”
Also curious what techniques there are to sample a reasoning model to deliberately perturb its internal state into more creative realms. Though these a fine line between insight and hallucination.
In some respects creativity is hallucination. As a human, you’re effectively internally postulating creative ideas “hallucinations” and then one of them “hits” and fires a whole bunch of neurons which indicate: “ok that wild idea actually has grounding and strong connections to the existing knowledge in your brain.”
- If this is your kind of thing and you ever get a chance to see the musical artist Tipper alongside Fractaled Visions driving the visuals, you’re in for a treat.
Most spot on visual depictions of psychedelic artifacts I’ve witnessed.
Saw them together last year and it’s the no. 1 artistic experience of my life. The richness, and complexity of Fractaled Vision’s visuals are almost unbelievable.
Even knowing a lot about shader programming, etc. some of the effects I was like “wtf how did he do that”.
Here’s the set, doesn’t fully capture the experience, but gives a feel: Seeing this in 4k at 60fps was next level.
- I feel like they're conflating "rave" with "clubbing."
Friday, Saturday club attendance has been dropping across the world, and many electronic music focused club venues have shut down (at least in Australia and the UK).
My word association of "rave" is "festival" though. Festivals feel like they're still booming, or at least not in dramatic decline.
From a small personal sampling: Coachella, Portola, Outside Lands, Proper, Lightning in a Bottle, festivals are still going strong. For some (Coachella, Lightning in a Bottle) attendance felt like it dropped 2023 -> 2024, but perhaps 10-20%, and this is likely economically correlated (inflation, etc). Late 2024- festivals (Portola, Proper) were packed.
- Pet theory is that our universe is run on some external computational substrate. A lot of the strangeness we see in quantum physics are side effects of how that computation is executed efficiently.
The inability to reconcile quantum field theory and general relativity is the that gravity is a fundamentally different thing to matter: matter is an information system that's run to execute the laws of physics, gravity is a side effect of the underlying architecture being parallelized across many compute nodes.
The speed of light limitation is the side-effect of it taking a finite time for information to propagate in the underlying computational substrate.
The top-level calculation the universe is running is constantly trying to balance computation efficiently among the compute nodes in the substrate: e.g. the universe is trying to maintain a constant complexity density across all compute nodes.
Black holes act as complexity sinks, effectively "garbage collection." The matter than falls below the event horizon is effectively removed from the computation needs of the substrate. The cosmological constant can be explained by more compute power being available as more and more matter is consumed by black holes.
This can be introduced into GR by adding a new scalar field whose distribution encodes "complexity density." e.g. some metric of complexity like counting micro-states, etc. This scalar field attempts to remain spatially uniform in order to best "smooth" computation across the computational substrate. If you apply this to a galaxy with a large central supermassive black hole, you end up with almost a point sink of complexity at the center, then a large area of high complexity in the accretion disk, and then a gradient of complexity away towards the edges of the galaxy. That is, the scalar field has strong gradients along the radius of the galaxy, and this gives rise to varying gravitational effects over the radius (very MOND-like).
Some back of the napkin calculations show that adding this complexity density scalar field to GR does replicate observed rotation curves of galaxies. Would love to formalize this and run some numerical simulations.
Would hope that fitting the free parameters of GR with this complexity density scalar field would yield some testable predictions that differ from current naive assumptions around dark matter and dark energy.
- Certain the end state is "one model to rule them all" hence the "transitional."
Just that the pragmatic approach, today, given current LLM capabilities, is to minimize the surface area / state space that the LLM is actuating. And then gradually expand that until the whole system is just a passthrough. But starting with a passthrough kinda doesn't lead to great products in December 2024.
- Have been building agents for past 2 years, my tl;dr is that:
Agents are Interfaces, Not Implementations
The current zeitgeist seems to think of agents as passthrough agents: e.g. a lite wrapper around a core that's almost 100% a LLM.
The most effective agents I've seen, and have built, are largely traditional software engineering with a sprinkling of LLM calls for "LLM hard" problems. LLM hard problems are problems that can ONLY be solved by application of an LLM (creative writing, text synthesis, intelligent decision making). Leave all the problems that are amenable to decades of software engineering best practice to good old deterministic code.
I've been calling system like this "Transitional Software Design." That is, they're mostly a traditional software application under the hood (deterministic, well structured code, separation of concerns) with judicious use of LLMs where required.
Ultimately, users care about what the agent does, not how it does it.
The biggest differentiator I've seen between agents that work and get adoption, and those that are eternally in a demo phase, is related to the cardinality of the state space the agent is operating in. Too many folks try and "boil the ocean" and try and implement a generic purpose capability: e.g. Generate Python code to do something, or synthesizing SQL based on natural language.
The projects I've seen that work really focus on reducing the state space of agent decision making down to the smallest possible set that delivers user value.
e.g. Rather than generating arbitrary SQL, work out a set of ~20 SQL templates that are hyper-specific to the business problem you're solving. Parameterize them with the options for select, filter, group by, order by, and the subset of aggregate operations that are relevant. Then let the agent chose the right template + parameters from a relatively small finite set of options.
^^^ the delta in agent quality between "boiling the ocean" vs "agent's free choice over a small state space" is night and day. It lets you deploy early, deliver value, and start getting user feedback.
Building Transitional Software Systems:
1. Deeply understand the domain and CUJs, 2. Segment out the system into "problems that traditional software is good at solving" and "LLM-hard problems", 3. For the LLM hard problems, work out the smallest possible state space of decision making, 4. Build the system, and get users using it, 5. Gradually expand the state space as feedback flows in from users. - +1, the second and third order effects aren't trivial.
We're already seeing escape velocity in world modeling (see Google Veo2 and the latest Genesis LLM-based physics modeling framework).
The hardware for humanoid robots is 95% of the way there, the gap is control logic and intelligence, which is rapidly being closed.
Combine Veo2 world model, Genesis control planning, o3-style reasoning, and you're pretty much there with blue collar work automation.
We're only a few turns (<12 months) away from an existence proof of a humanoid robot that can watch a Youtube video and then replicate the task in a novel environment. May take longer than that to productionize.
It's really hard to think and project forward on an exponential. We've been on an exponential technology curve since the discovery of fire (at least). The 2nd order has kicked up over the last few years.
Not a rational approach to look back at robotics 2000-2022 and project that pace forwards. There's more happening every month than in decades past.
- Convinced that Apple has shot themselves in the foot / tied their hands behind their back / insert analogy here, re: privacy <> AI.
Interesting to see how it plays out. Meta and Google have much more permissive privacy policies / stances, which means Meta + Google models are going to get much better faster.
Apple does potentially have an edge with their Mx series of chips re: inference flops. I bet they're hoping that the model quality vs. model size curve continues dropping such that they can run sufficiently powerful LLMs on-device.
We'll see.
- Short term defense is learning about, and becoming an expert in, using LLMs in products.
Longer term defense doesn't exist. If Software Engineering is otherwise completely automated by LLMs, we're in AGI territory, and likely recursive self-improvement plays out (perhaps not AI-foom, by huge uptick in capability / intelligence per month / quarter).
In AGI territory, the economy, resource allocation, labor vs. capital all transition into a new regime. If problems that previously took hundreds of engineers working over multiple years can now be built autonomously within minutes, then there's no real way to predict the economic and social dynamics that result from that.
- One sample point of one person's experience:
Product's built based on brainstorming: NEVER worked.
Product built around solution to actual problems I've experienced: Always worked, BUT may not have a large enough TAM.
The best way to actually find problems to solve is to replicate something else (as a learning process to actually find the real product you need to build), or talk to people who have problems: e.g. Try an replicate a product with relatively simple core functionality all the way-end-to-end.
During that process you'll probably discover paint point that aren't addressed by any product.
Example from building: Build some GenAI based app that allows users to upload a video and have it "horrified." There may only be a small market in that, but the framework you build to do it could be productized (or subsets of the problem). A simple to use full stack solution where folks can focus on the image / video generation model prompting fine tuning, and then one-click deploy that model wrapped in a whitelabeled ios app with monetization built-in, etc. <-- sell the shovels.
Example from talking: Lot's of buddies have challenges with bringing AI based coding tools into the organization because most don't support on-prem. Is there an on-prem solution that acts like a DMZ? That is, Cursor team could securely deploy their models to containers running on on-prem hardware (with layers of physical and software security to prevent exfiltration of the model weights), while companies can load their propriety data into another container. Win-win.
- Feels different to past hype cycles (Internet bubble, Crypto bubble).
LLMs with meaningful capabilities arrive very quickly. e.g. One week they were not that useful, the next week they gained meaningful capabilities.
A function that takes text and returns text isn't that useful without it being integrated into products, and this takes time.
Next 12-24 months will be the AIfication of many workflows: that is, discovering and integrating LLM-based reasoning into business processes. Assuming even a gradual improvement in capabilities of LLMs over time, all of these AI enhanced business processes will simply get better.
Diffusion of technology is slow slow slow, and then fast. As I become more capable with AI (e.g. what tasks as an engineer are helped using AI) I'm getting better and better at it. So there's a non-linear learning curve where, as you learn to use the technology better, you can unlock more productivity.
- One of the few strategies that I've seen works is "I'm going to do it this way, and I'll take all responsibility for it failing." and then if it fails, actually take responsibility for the failure.
You have to have a certain confidence in your opinion, and you have to be prepared to destroy yourself mentally and physically to deliver if things go off the rails. But once your deliver something that matches your vision, usually worth it.
- No.
The race is, can OpenAI innovate on product fast enough to get folks to switch their muscle memory workflows to something new?
It doesn't matter how good the model is, if folks aren't habituated to using it.
At the moment, my muscle memory to go to Claude, since it seems to do better at answering engineering questions.
The competition really is between FAANG and OpenAI, can OpenAI accumulate users faster than Apple, Google, Meta, etc layer in AI-based features onto their existing distribution surfaces.