There’s also the issue of things not necessarily having to be the MOST profitable thing ever. Memory manufacturers decided to go to the highest bider because money. I suspect Chinese company will be quite happy to just flood the market with cheaply priced (relatively speaking) products and just get everyone’s money.
If you ask any big LLM a mundane question, it’s likely to get it right. And as you put it, it’s one of the way people get sucked in. But it does work, very inefficiently, for that purpose. Saying it does not is lying to yourself.
You can have IPv6 on your router, and an isolated network. It’s just not necessarily what’s done by default.
The issue is not usually with the seed (although being able to determine the seed used for a PRNG completely defeat its purpose). The biggest issue (aside from deterministic seed) is that not all (P)RNG are created equal. Some have seemingly infinitely long, non repeating sequences, that varies greatly for a minuscule change in the seed (the good ones). Some have relatively short, repeating sequences. Some are so badly implemented that actually seeing a few numbers of the sequence allows reverse-engineering their internal state, so you can get the next numbers for free afterward. If you don’t use a big function with an impossible to probe initial state (like any good cryptographic hash function), you could even go backward in the sequence, finding previously generated secrets.
There’s plenty of attack surface on bad implementation of a (P)RNG. And the hardware ones combines “nuh-uh, we won’t tell you the seed” with “nuh-uh, you’ll have to trust our implementation”. That’s why, as far as I know, hardware RNG is usually used as one of the source for the seed of a PRNG.
For example, although it was deemed a bug, some AMD processor, for a while, failed to generate actual randomness under some circumstances. That’s a problem if you only rely on them. That’s why dumping that in a tumbler of other sources is a good idea. (sorry for the reddit link: https://old.reddit.com/r/AMDHelp/comments/itfgf4/cpu_random_generator_seem_to_be_failing/ )
The way you get your seed also matter. Time-based seed, if you have a vague idea of when it was acquired, is easy to check. Sure, there might be a window of a few hundred of thousands of milliseconds, but that’s nothing too hard to check if you’re sure this would yield you a secure key you really need. Multiplying the source, from sources that seemingly can’t be under the control of an attacker is important, but you have to be careful. An anecdote I was told at a conference was about using the system’s delay on read. Since reading happened at seemingly random (with a busy system), on a HDD, where the head have to actually move and the platters have to spin, it might be a good idea. When people moved to SSD, the reading delay became almost constant. Not such a good source now.
An example of failure on properly seeding your PRNG would be the random SSH keys generated by Debian (a long time ago). The random generation of keys moved from almost the full space of possibility to around 32.000 keys. That’s low enough that those keys are actually listed and banned from being used by some systems. More info about that here: https://www.hezmatt.org/~mpalmer/blog/2024/04/09/how-i-tripped-over-the-debian-weak-keys-vuln.html
If your PRNG is implemented correctly, with proper impossible (well, extremely hard) to reverse functions, and a really good seed that can’t be exfiltrated, periodically augmented (and not replaced!) with many source of seemingly out of control randomness, there’s little to do against that beyond just dumping memory beforehand and taking it out the hard way. Unfortunately, a lot of people think “outside the box” and wants to reimplement their own. Or rely on faulty hardware. Or rely on tampered hardware. And so on.
I don’t know where you think I implied it was ok for surface search, but it’s the exact opposite of what I wanted to say.
I don’t know about a “revolt”, but it sure is disappointing.
I like the guy, and most of what he does. I also like, contrary to some, that he usually keep quiet on some controversial topics. Well, I liked that; it seems that he wasn’t just “staying quiet”, unfortunately.
And, the worst part is just that he, as we know him (remember, parasocial stuff and so on), should have been on the side of people that “know better”. I don’t understand how you can be that curious of a person, and not seeing the issues here. I understand trying the tools, to see what it’s about and what it can actually do; in a sort of experimentation kind of way. I would even understand (but not condone) regularly using them in some ways, once you find something that you can confidently say “it works well enough”.
This, however, is not it. And maybe I’m putting him on a sort of pedestal when I say he should have known better, but that’s how I feel it. There’s a large list of things LLM are known to not be reliable for: searching anything beyond surface informations, properly summarizing actually complex subjects, picking out what’s pertinent and what isn’t, being consistent, not having blind side, and being truthful (assuming we can decide what truthful means in the first place). All of this is intrinsic to the technology at hand. And while some people argument that “humans are imperfect too”, the difference is that humans can spot their mistakes.
And yet, here we are; one of the most prominent science communicator, heavily leaning on this to start his researches, undoubtedly being led astray on some topics by hallucination, missing some big issues on others because the models simply didn’t get something out this time (for whatever reasons), getting very heavy confirmation bias (because of course at some point you ask a prompt that’s oriented, and the LLM will oblige). And recently, a few things he kept saying, like, we’re needing more code… sounded silly. In fact, most of the time I hear something in a field I’m reasonably knowledgeable about, I find things that I don’t really agree with.
That’s the part that worries me about this. Not that he used large, corporate LLMs “responsibly” (assuming that’s a thing one can do, which I’m not convinced of). But that he muddled the image of a reliable source (and we have to decide, at some point, that this or that source is reliable or not) with lots of approximations wrapped in a pretty ribbon. And now, the little things I was slightly weirded out in his videos are taking way more space in my mind, and I’m starting to doubt topics I’m absolutely not knowledgeable about that he’s talked about, too, because that’s just how trust works.
At the end of the day, reliable people come and goes, but the “disclosure” sure sounded like he didn’t actually grasp the issues, only barely that there might have been issues.
They’ve started killing/selling them though.