1
point
jgord
Joined 1,108 karma
- I feel the same frustration, seen from another angle : I think we can use current ML techniques to solve 3 or 4 hard problems in 3D reconstruction, and doing so would unlock a vast amount of value - we could turn lidar scans and photos of buildings and industrial plants into accurate 3D models automatically.
BUT I think the bottleneck is _funding_ of small early risky startups to do the needed engineering work.
My notes on this : https://quantblog.wordpress.com/2025/10/29/digital-twins-the...
LLMs, GPU datacenters attract all the big money, and the med and small VCs seem to be leaving their money in the bank earning high interest rates, unless there is a slam dunk opportunity with guaranteed traction and MRR growth.
We seem to be betting that only the large companies will innovate, when historically this has not been the case - Deepseek is a recent counterexample.
- My take, after working on some algos to detect geometry from pointclouds, is that its solvable with current ML techniques, but we lack early stage VC funding for startups working on this :
https://quantblog.wordpress.com/2025/10/29/digital-twins-the...
I have no doubt FeiFei and her well funded team will make rapid progress.
- busywork ... but maybe good marketing - people somehow believe that ISO has some relationship to quality.
- From my vantage point wrangling algos to extract 3D geometry from pointclouds - I really think this is a domain where ML can solve the missing pieces of the puzzle in the next 18 months.
The new tech will unlock a lot of value, but we need an opinionated investor to fund the engineering effort to get us there.
Funding is the bottleneck, not talent or fundamental science.
tl:dr We need a couple of visionary Angel Investors to fund these 3D Skunkworks projects and bring the real world onto the internet.
- Basically true, but there are other potential sources of growth :
- using technology to unlock cheaper energy - using technology to automate boring manual labor - using technology to extend healthy lifespan
Given the demographics collapse and ageing population in most 'developed' countries, we need to look at these other ways of generating economic growth.
- Before I read the article, Ill summarize the facts that seem to be hard and true about climate :
- we are nearing or at +1.5C above pre-industrial baseline
- human carbon burning CO2 emissions are at a max and likely long plateau
- mean temp is rising by around +0.3C per decade
- we will be nearing +2.0C in around 15 years, 2040 give or take
- warming is mainly caused by us humans burning carbon, emitting CO2 and some CH4
- if we reach net-zero, we will be at peak CO2 and thus peak heat, for a long while
In addition, the only economically viable way to bring down the temp seems to be deliberate pollution by emitting sulphur or other particles aka Solar Radiation Management to brighten clouds, reduce heat absorption by the ocean. Volcanoes and shipping fuels have essentially proven that this brings down the temperature, in the short term.
We geo-engineered our way into this hot mess, and we will need to geo-engineer our way out of it.
If the temp reaches +2.5 or +3C .. I think that means quite a lot of crop failure, forced migration, geopolitical tension, lack of stable food supply.. and death to a large number of humans seems to follow logically from that.
So, now Ill look at the article to see if any of these tough truths were mentioned .. sorta-kinda no-so-much, it seems like he thinks things are not that urgent. ?!?
- I think Oxide should be renting out time on their hardware racks, as well as selling them to big orgs.
Oxide looks to be superb engineering up and down the whole stack, and if it drives more rust code into linux all the better.
Now that linode has been consumed by Akamai, we need an alternative.
- Any hypothesis for why eating only one thing would solve IBD ?
Would that select for a particular biota / micro-fauna environment ? or cause less mechanical / chemical inflammation ?
glad it worked in your case.
- that is spectacular .. thx for link.
- afaict, the situation can be roughly summarized as :
Even if we do a great job of electrifying everything, moving away from fossil/carbon fuels by 2060, we still have a heat problem to deal with - will we be able to grow our normal crops under +2.5C, and deal with extreme heatwaves, floods, storms ?- global climate is warming by around +0.3C per decade - its caused mainly by humans burning carbon chains for energy, emitting CO2 - we are currently sailing thru +1.5C above pre-industrial mean temp - Methane CH4 is also a strong warming agent, around 20x more potent than CO2, on decade timescales - humans are emitting all time high levels of C02, around 40Gtonnes / yr - Carbon capture / CCS / DAC need to be millions of times more efficient to be significant - we dont have enough room or time to plant trees to remove the CO2 - net-zero when we reach it, corresponds to max-Co2 which means peak-heat - net-zero aka peak-heat might occur bu 2060, by which time we'll be near +2.5C - extreme events are not linear in increase in temp [ think of shifting the mean of a bell curve ]It seems we will need Solar Radiation Management Geo-engineering "SRM" in order to survive that peak-heat and buy us a few decades in which to slowly remove CO2 [ even as we move full steam ahead to de-carbonize our energy system with wind, solar, battery packs, hydro, fission, geothermal and hopefully fusion power ]
Particulates from volcanoes are well known to cause a cooling effect, and its now becoming more obvious that particulates in pollution in Asia, and sulphur impurities in shipping fuels were having a measurable cooling effect - we seem to be warming faster now that Asia and shipping fuels are not producing as much particulate pollution [ thus less cooling effect ]
It seems to me the only "Hail Mary" we have to address the heat problem, is to use SRM to exert a cooling effect - we humans geo-engineered a warm planet over 150 years of burning carbon fuels, and we will need to geo-engineer our way out of this mess.
tldr : Abundant clean energy is needed, but we also have to address the heat problem - with SRM geo-engineering
- not to mention the mandatory cloudflare "are you human" pre-vetting page Im seeing on 15% of sites.
jesus wept.
- short answer yes .. I tried a _lot_ of approaches, many worked partially. I think I linked to a YT video screencast showing edges of planes that my algo had detected in a sample pointcloud ?
Efficient re-meshings are important, and its worth improving on the current algorithms to get crisper breaklines etc, but you really want to go a step further and do what humans do manually now when they make a CAD model from a pointcloud - ie. convert it to its most efficient / compressed / simple useful format, where a wall face is recognized as a simple plane. Even remeshing and flat triangle tesselation can be improved a lot by ML techniques.
As with pointclouds, likewise with 'photogrammetry', where you reconstruct a 3D scene from hundreds of photos, or from 360 panoramas or stereo photos. I think in the next 18 months ML will be able to reconstruct an efficient 3D model from a streetview scene, or 360 panorama tour of a building. An optimized mesh is good for visualization in a web browser, but its even more useful to have a CAD style model where walls are flat quads, edges are sharp and a door is tagged as a door etc.
Perhaps the points Im trying to make are :
- the normal techniques are useful but not quite enough [ heuristics, classical CV algorithms, colmap/SfM ] - NeRFs and gaussian splats are amazing innovations, but dont quite get us there - to solve 3D reconstruction, from pointclouds or photos, we need ML to go beyond our normal heuristics : 3D reality is complicated - ML, particularly RL, will likely solve 3D reconstruction quite soon, for useful things like buildings - this will unlock a lot of value across many domains - AEC / construction, robotics, VR / AR - there is low hanging fruit, such as my algo detecting planes and pipes in a pointcloud - given the progress and the promise, we should be seeing more investment in this area [ 2Mn of investment could potentially unlock 10Bn/yr in value ] - I kind of did a version of what you suggest - I think I linked to a video showing plane edges auto-detected in a pointcloud sample.
Similarly I use another algo to detect pipe runs which tend to appear as half cylinders in the pointcloud, as the scanner usually sees one side, and often the other side is hidden, hard to access, up against a wall.
So, I guess my point is the devil is in the details .. and machine learning can optimize even further on good heuristics we might come up with.
Also, when you go thru a whole pointcloud, you have a lot of data to sift thru, so you want something fairly efficient, even if your using multiple GPUs do do the heavy matmull lifting.
You can think of RL as an optimization - greatly speeding up something like monte carlo tree search, by learning to guess the best solution earlier.
- ps. its handy to compare the relative data sizes of [ models of ] the same scene : typically for something like a house, the data will be ballpark :
Im guessing gaussian-splat would be something like 20x to 40x more efficient than the pointcloud. I achieved similar compression for building scans, using flat textured mini-planes.- 15GB of pointcloud data ( 100Mn xyzRGB points from a lidar laser scanner ) - 3 GB of 360 panorama photos - 50MB obj 3D textured model - 2MB CAD model - yeah, see my other comment.
To me its totally obvious that we will have a plethora of very valuable startups who use RL techniques to solve realworld problems in practical areas of engineering .. and I just get blank stares when I talk about this :]
Ive stopped saying AI when I mean ML or RL .. because people equate LLMs with AI.
We need better ML / RL algos for CV tasks :
These might be used by LLMs but are likely built using RL or 'classical' ML techniques, tapping into the vast parallel matmull compute we now have in GPUs / multicore CPUs, and NPUs.- detecting lines from pixels - detecting geometry in pointclouds - constructing 3D from stereo images, photogrammetry, 360 panoramas - makes sense - humans have evolved a lot of wetware dedicated to 3D processing from stereo 2D.
I've made some progress on a PoC in 3D reconstruction - detecting planes, edges, pipes from pointclouds from lidar scans, eg : https://youtu.be/-o58qe8egS4 .. and am bootstrapping with in-house gigs as I build out the product.
Essentially it breaks down to a ton of matmulls, and I use a lot of tricks from pre-LLM ML .. this is a domain that perfectly fits RL.
The investors Ive talked to seem to understand that scan-to-cad is a real problem with a viable market - automating 5Bn / yr of manual click-labor. But they want to see traction in the form of early sales of the MVP, which is understandable, especially in the current regime of high interest rates.
Ive not been able to get across to potential investors the vast implications for robotics, AI, AR, VR, VFX that having better / faster / realtime 3D reconstruction will bring. Its great that someone of the caliber of Fei-Fei Li is talking about it.
Robots that interact in the real world will need to make a 3D model in realtime and likely share it efficiently with comrades.
While a gaussian splat model is more efficient than a pointcloud, a model which recognizes a wall as a quad plane is much more efficient still, and needed for realtime communication. There is the old idea that compression is equivalent to AI.
What is stopping us from having a google street-view v3.0 in which I can zoom right into and walk around a shopping mall, or train station or public building ? Our browsers can do this now, essentially rendering quake like 3D environments - the problem is with turning a scan into a lightweight 3D model.
Photogrammetry, where you have hundreds of photos and reconstruct the 3D scene, uses a lot of compute, and the colmap / Structure-from-Motion algorithm predates newer ML approaches and is ripe for a better RL algorithm imo. Ive done experiments where you can manually model a 3D scene from well positioned 360 panorama photos of a building, picking corners, following the outline of walls to make a floorplan etc ... this should be amenable to an RL algorithm. Most 360 panorama photo tours have enough overlap to reconstruct the scene reasonably well.
I have no doubt that we are on the brink of a massive improvement in 3D processing. Its clearly solvable with the ML/RL approaches we currently have .. we dont need AGI. My problem is getting funding to work on it fulltime, equivalently talking an investor into taking that bet :)
- any group strategy to push back against the overuse of whole-page captchas ?
Do we all need to run an AI browser plugin now that auto-fills cloudflare captchas ?
- detecting geometry from point cloud scans of buildings using ML/RL techniques :
flat planes and edges : https://youtu.be/-o58qe8egS4
semi-cylinder pipes : https://youtu.be/8fjHNDGKeu4
Aim to automate that TAM of 5Bn/yr of manual labor, growing at 12% cagr
SOM : ~100Mn
Could also use it to play media - so a phone or tablet could act as a remote control from anywhere in wifi reach, and play music on the main TV screen / speakers or on the local device.
Was pretty cool, but didnt have the funds to commercialize it.