• 3 Posts
  • 89 Comments
Joined 2 years ago
cake
Cake day: January 30th, 2025

help-circle
  • The video isn’t about how this is all BS, conclusion from section 1:

    They reviewed roughly 1,250 papers on AI self-improvement and found that 74% of them were published in 2026 alone. This field is moving extremely fast. Anyone telling you this is all nonsense has just not read the literature. Is this recursive self-improvement RSI in the bounded sense? Absolutely yes. Al systems are improving themselves within well-defined tasks and the improved versions are producing further improvements. This is a real thing. It is demonstrated that is in the papers and none of this existed 5 years ago.

    The caveat in the second section is:

    We have much weaker evidence that it can decide what the research problem should be. And that gap, I cannot stress this enough, between optimization and judgment, between execution and scientific taste is the gap that we need to close before anything resembling an autonomous AI scientist could exist. And this is not a small gap

    That’s still scary, yeah it probably won’t lead to human extinction, but it has serious implications for both economic and social life moving forward











  • Explain to me how the analogy is wrong then.

    I have accepted that llms have hard limits, like you said with regards to information theory and godels incompleteness theorem. I don’t have enough knowledge to articulate those limits, just like I don’t know the physics that prevents you from shrinking the dye further but I accept that they exist.

    So we agree there are hard limits, what I am saying is:

    • within those limits is a huge problem space
    • millions of people are putting labor in that problem space today
    • LLMs have not saturated that problem space yet
    • LLMs are making gains in that problem space, like with math and reasoning.

    I’m not arguing where that “finish line” is because I know I don’t know enough about that. I do understand that 99.999% of problems aren’t passed that finish line and there’s a lot to go before LLMs reach that line.

    Almost all use cases for llms don’t require creating novel math techniques, so why are you focusing on them? If “can’t create genuinely new techniques” disqualifies something from being economically transformative or intellectually significant, that standard would disqualify most human knowledge labor too.


  • Bringing up information theory and godels incompleteness theorem in regards to ai advancement is like saying there are hard physical limits on shrinking the dye in the 1980s. Yes hard axiomatic limits exist, but within those limits is a huge problem space. Even if we limit it to solving problems on a screen using existing techniques, that problem space is huge and occupying a large amount of time for the knowledge workers who make up ~25% of the labor force.

    Unlike shrinking the dye though we are not near that limit, we still have a lot of improvement to make before AI can solve all the problems in that space. I think we can both agree the current LLMs aren’t able to replace all screen work. But it is making big strides in that problem space, such as going from failing basic arithmetic to solving the hardest math problems in a couple years.

    Like with shrinking the dye there’s not going to be a singular breakthrough, like recursive self improvement, that’ll throw us up against those limits almost immediately. Progress is done incrementally over years with a bunch of different smaller improvements, like opaque recurrence. This does have some acceleration as the technology helps with the design of the technology, computers help chip architects design better chips, which make better computers, but a human is still required. Same with AI, every lab is probably heavily relying on coding agents to assist them with developing and deploying new techniques. The AI may not be creating these techniques, but it is accelerating the creation of these techniques.


  • How are you defining “new information”? If I ask Claude "what is 235 x 567 + 57,899? " There is a high likelihood that that question was not in its training data, but it still gets it right and generates new information. It will be using existing methods from its training data to “reason” to get the answer, and it hasn’t shown the ability to devise new methods. But 99.99% of the problems in the world can be solved by using existing methods.

    Open AI isn’t going to go bust because it can’t make novel math techniques, almost all of the knowledge work requiring math doesn’t require new methods. If it’s able to master every existing technique used by knowledge workers without creating new methods, ie. your finish line, then they’ll have automated a huge chunk of labor and probably become a hugely profitable company.

    Recursive self improvement may not be panning out currently so we won’t see any singularity style exponential take off, but we’re still seeing steady linear improvement in tasks such as math and reasoning through other techniques, like the opaque recurrence that astra is using.