Is it really feasible to run it 100% locally? I know there’s plenty of people with very powerful rigs indeed, but still. Or are 99% of these people really saying “it would, in theory, be possible to run that locally, therefore your concerns are invalid”?

There’s usually a lot of caveats they don’t mention on this. Like saying it** is feasible to run it** locally! (** it depends on your rig, memory and storage constraints, model licensing constraints blah-di-dah, blah-di-dee)

Yes, there are super small and simple models so this is technically true.

Computers are used to do math, hosted agents are doing big math for people from far away, but some people only need a little math they can do at home.

cant be used ethically when the AI datacenters are polluting asf.

There is no argument to use AI ethically. Only bullshit they tell themselves so as to circumvent responsibility.

I think it’s at least way more ethical to use locally hosted open weight AI. Mostly because a lot of these models are usually fine tuned and distilled to run on a single GPU and that is the direction the technology needs to develop to be sustainable.

Obviously for most things it’s way more ethical to just not use AI or LLMs.

I think people should at least experiment with running local models on their own hardware. It really puts it into perspective how much compute they use. Like the power, speed, and performance ratios of different size LLM and how absurd anything involving generating video is. I have a pretty beefy gaming PC and it took it like 6 minutes to generate 10 seconds of 720p video. And then that time goes up exponentially past 10 seconds. Like 20 minutes for a 15 second video

Running on a small model is basically like playing a modern full sized video game. When I measure my PC with a power meter

Let me just answer this part

Is it really feasible to run it 100% locally? I know there’s plenty of people with very powerful rigs indeed, but still. Or are 99% of these people really saying “it would, in theory, be possible to run that locally, therefore your concerns are invalid”?

Yes absolutely, and you don’t even need an extremely powerful machine. Basic text generation will work on a macbook pro. If the hardware prices were at normal levels, buying a machine for 2-3k USD/EUR and self-hosting powerful models would be feasible. Right now the same hardware will run you about 10k, that why I don’t think it’s practical at this moment.

However training is where the real cost lies. It’s pretty much impossible to train a model from 0 on anything resembling a personal machine. First you need all the data, measuring in millions of terabytes. And then you need to train the model on all of that.

On a side note - I once tried training a GPT2 replica on my laptop with about 3GB of random text. The estimated time was 3 months. That’s the level of requirements we’re talking about.

When it doesn’t use external disinformation causing data centres and runs from home generated solar power.

An llm is just a set of rules for cobbling together text. In and of itself, the technology is ethically neutral. Mind you, all ethics are both situational and relative, but what llms do really isn’t ethically any different from something like how your game handles enemy actions, except in scale.

What makes them a nightmare is the wonton fuckery involved in how they’re trained. The open sourced ones lack the abusive and nigh maniacal plundering of human effort that the commercial ones do.

So, when it comes to a locally hosted, open source option, there’s no fundamental difference ethically between an llm, your music player, your word processor, whatever.

Even data centers for generative ai could conceivably be run ethically, given some massive assumptions regarding resource allocation and usage. Mainly in that the energy it takes, and the physical resources, are being allocated to those centres rather than something else. But if everyone agrees in the benefits, then using renewable resources carefully would make it no better or worse than any of the non ai data centers already in existence. I mean, most VPNs are renting either space or hardware from some company that has a data center. “The cloud” is just a giant computer owned and operated by some company somewhere, and that’s a data center.

Yeah, generative systems are way more resource intensive, but we can’t pretend that AWS isn’t doing the same basic thing, just with a different use case. Again, there’s the caveat that that details aren’t the same, I’m talking about principles of ethics, nor the specific technology. We’ve decided that building spaces, or filling existing space, for computing is useful enough to be worth the problems it causes. The consensus hasn’t really been reached that generative ai is also worth it, but in ethical terms, the same framework applies.

So, your concerns are absolutely valid. They are, however, partially met by sustainable implementation

Yes. “open-weights” means they publish the AI model file to the public and people can download and run it themselves.

And there’s very different AI models out there. Some are smaller, some bigger… Bigger usually means more “intelligent” and you also need more resources to run them. On average, if you own something like a beefy gaming computer, you can do a reasonably “clever” AI at reasonable speed. Something like ChatGPT needs a datacenter, and at the other end we also have some small models which run on some smartphones or average laptops. They surely won’t be able to do the same things ChatGPT does… And if your laptop is slow, it’ll output text very slowly. But maybe you’re okay with less “intelligent” AI.

And no, they don’t come from nowhere. They’re mainly made by the AI companies. They’re trained the same way all large language models are trained with the same problematic procedure. Oftentimes they publish some more information like a scientific paper alongside. There might be information inside about the energy used / carbon footprint of the training steps. Sometimes/rarely they also give information on what data they used.

And internet people have all sorts of weird opinions on (FOSS) philosophy and ethics. I think we’d need to be more specific than that. Dirty gas turbines and allowing companies to be exempt from copyright while other people get sued for copying a single movie isn’t really ethical by any means.