this post was submitted on 12 Mar 2025
48 points (80.8% liked)

Selfhosted

44512 readers
1072 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
48
What AI services are you selfhosting? Or, have tested and passed on (discuss.james.network)
submitted 6 days ago by [email protected] to c/[email protected]
53 comments fedilink hide all child comments
 

Wondering about services to test on either a 16gb ram "AI Capable" arm64 board or on a laptop with modern rtx. Only looking for open source options, but curious to hear what people say. Cheers!

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 17 points 6 days ago* (last edited 6 days ago)

I run kobold.cpp which is a cutting edge local model engine, on my local gaming rig turned server. I like to play around with the latest models to see how they improve/change over time. The current chain of thought thinking models like deepseek r1 distills and qwen qwq are fun to poke at with advanced open ended STEM questions.

STEM questions like "What does Gödel's incompleteness theorem imply about scientific theories of everything?" Or "Could the speed of light be more accurately refered to as 'the speed of causality'?"

As for actual daily use, I prefer using mistral small 24b and treating it like a local search engine with the legitimacy of wikipedia. Its a starting point to ask questions about general things I don't know about or want advice on, then do further research through more legitimate sources.

Its important to not take the LLM too seriously as theres always a small statistical chance it hallucinates some bullshit but most of the time its fairly accurate and is a pretty good jumping off point for further research.

Lets say I want an overview of how can I repair small holes forming in concrete, or general ideas on how to invest financially, how to change fluids in a car, how much fat and protein is in an egg, ect.

If the LLM says a word or related concept I don't recognize I grill it for clarifying info and follow it through the infinite branching garden of related information.

I've used an LLM to help me go through old declassified documents and speculate on internal gov terminalogy I was unfamiliar with.

I've used a speech to text model and get it to speek just for fun. Ive used multimodal model and get it to see/scan documents for info.

Ive used websearch to get the model to retrieve information it didn't know off a ddg search, again mostly for fun.

Feel free to ask me anything, I'm glad to help get newbies started.

[–] [email protected] 19 points 6 days ago (1 children)

I use OLlama & Open-WebUI, OLlama on my gaming rig and Open-WebUI as a frontend on my server.

It's been a really powerful combo!

[–] [email protected] 5 points 6 days ago (4 children)

Would you please talk more about it. I forgot about Open-webui, but intending to start playing with. Honestly, what do you actually do with it?

[–] [email protected] 2 points 2 days ago

I have this exact same setup. Open Web UI has more features than I've been able to use such as functions and pipelines.

I use it to share my LLMs across my network. It has really good user management so I can set up a user for my wife or brother in law and give them general use LLM while my dad and I can take advantage of Coding-tuned models.

The code formatting and code execution functions are great. It's overall a great UI.

Ive used LLMs to rewrite code, help format PowerPoint slides, summarize my notes from work, create D&D characters, plan lessons, etc

[–] [email protected] 5 points 6 days ago* (last edited 6 days ago) (3 children)

I have the same setup, but its not very usable as my graphics card has 6gb ram. I want one with 20 or 24, as the 6b models are pain and the tiny ones don’t give me much.

Ollama was pretty easy to set up on windows, and its eqsy to download and test the models ollama has available

load more comments (3 replies)
[–] [email protected] 4 points 6 days ago* (last edited 6 days ago)

I have Linkwarden pointed at my ollama deployment, so it auto tags links that I archive which is nice.

I've seen other people send images captured on their security cameras on frigate to ollama to get it to describe the image

There's a bunch of other use cases I've thought of for coding projects, but haven't started on any of them yet

[–] [email protected] 3 points 6 days ago

Sex chats. For other uses, just simple searches are better 99% of the time. And for the 1%, something like the Kagis FastGPT helps to find the correct keywords.

[–] [email protected] 8 points 6 days ago (2 children)

None currently. Wish I could afford a GPU to play with some stuff.

[–] [email protected] 1 points 5 days ago (2 children)

Yeah. I have a mini PC with an AMD GPU. Even if I were to buy a big GPU I couldn't use it. That frustrates me, because I'd love to play around with some models locally. I refuse to use anything hosted by other people.

[–] [email protected] 1 points 2 days ago

Most LLM projects support Vulkan if you have enough VRAM

[–] [email protected] 1 points 5 days ago (1 children)

Your M.2 port can probably fit an M.2 to PCIe adapter and you can use a GPU with that - ollama supports AMD GPUs just fine nowadays (well, as well as it can, rocm is still very hit or miss)

[–] [email protected] 1 points 5 days ago (1 children)

Oh, then I need to give it another try.

[–] [email protected] 1 points 4 days ago

Check rocm's supported cards, oh and after you install rocm, restart your computer - made that mistake when I was doing it and couldn't figure out why it wasn't working.

[–] [email protected] 2 points 6 days ago

Well, let me know your suggestions if you wish. I took the plunge and am willing to test on your behalf, assuming I can.

[–] [email protected] 7 points 6 days ago (4 children)

I messed around with home assistant and the ollama integration. I have passed on it and just use the default one with voice commands I set up. I couldn't really get ollama to do or say anything useful. Like I asked it what's a good time to run on a treadmill for beginners and it told me it's not a doctor.

[–] [email protected] 12 points 6 days ago (1 children)

Like I asked it what's a good time to run on a treadmill for beginners and it told me it's not a doctor.

Kirkland brand meseeks energy.

[–] [email protected] 3 points 5 days ago

Hey now Kirkland brand is respectable, usually premium brands repackaged. Such as how Costco vodka was secretly ("secretly") Grey Goose

[–] [email protected] 5 points 6 days ago* (last edited 6 days ago)

There are some experimental models made specifically for use with Home Assistant, for example home-llm.

Even though they are tiny 1-3B I've found them to work much better than even 14B general purpose models. Obviously they suck for general purpose questions just by their size alone.

That being said they're still LLMs. I like to keep the "prefer handling commands locally" option turned on and only use the LLM as a fallback.

[–] [email protected] 3 points 6 days ago

Sounds like ollama was loaded up with an either overly censored or plain brain dead language model. Do you know which model it was? Maybe try mistral if it fits in your computer.

[–] [email protected] 2 points 6 days ago (1 children)

Haha, that is hilarious. Sounds like it gave you some snark. afaik you have to clarify by asking again when it says such things. "I'm not asking for medical advice, but..."

load more comments (1 replies)
[–] [email protected] 5 points 6 days ago (1 children)

I have Immich that has AI searching for my photos. Pretty useful for finding stuff actually

[–] [email protected] 1 points 5 days ago (1 children)

Once I changed the default model, immich search became amazing. I want to show it off to people but alas, way too many NSFW pics in my library. I would create a second "clean" version to show off to people but I've been too lazy.

[–] [email protected] 1 points 2 days ago

Can't you tag the NSFW to filter it out?

[–] [email protected] 4 points 5 days ago (1 children)

LMStudio is pretty much the standard. I think it's opensource except for the UI. Even if you don't end up using it long-term, it's great for getting used to a lot of the models.

Otherwise there's OpenWebUI that I would imagine would work as a docker compose, as I think there's ARM images for OWU and ollama

[–] [email protected] 2 points 2 days ago (1 children)

Well they are fully closed source except for the open source project they are a wrapper on. The open source part is llama.cpp

[–] [email protected] 1 points 2 days ago (1 children)

Fair enough, but it's damn handy and simple to use. And I don't know how to do speculative decoding with ollama, which massively speeds up the models for me.

[–] [email protected] 1 points 2 days ago (1 children)

Their software is pretty nice. That's what I'd recommand to someone who doesn't want to tinker. It's just a shame they don't want to open source their software and we have to reinvent the wheel 10 times. If you are willing to tinker a bit koboldcpp + openewebui/librechat is a pretty nice combo.

[–] [email protected] 1 points 2 days ago (1 children)

That koboldcpp is pretty interesting. Looks like I can load a draft model for spec decode as well as a pile of other things.

What local models have you been using for coding? I've been disappointed with things like deepseek-coder and the qwen-coder, it's not even a patch on Claude, but that damn cost for anthropic has been killing me.

[–] [email protected] 1 points 1 day ago (1 children)

As much as I'd like to praise the open-weight models. Nothing comes close to Claude sonnet in my experience too. I use local models when info are sensitive and claude when the problem requires being somewhat competent.

What setup do you use for coding? I might have a tip for minimizing claude cost you depending on what your setup is.

[–] [email protected] 2 points 1 day ago (1 children)

I'm using vscode/Roocode with Gosucoder shortprompt, with Requesty providing models. Generally I'll use R1 to outline a project and Claude to implement. The shortprompt seems to reduce the context quite a bit and hence the cost. I've heard about Cursor but haven't tried it yet.

When you're using local models, which ones are you using? The ones I mention don't seem to give me much I can use, but I'm also probably asking more of them because I see what Claude can do. It might also be a problem with how Roocode uses them, though when I just jump into a chat and ask it to spit out code, I don't get much better.

[–] [email protected] 1 points 1 day ago* (last edited 1 day ago)

If you are willing to pay 10$ a month. You should get GithubCopilot, it provides near unlimited claude 3.5 usage. RooCode can hook into the github copilot api, and use it for its generations.

I use Qwen Coder and Mistral small locally too. It works ok, but its nowhere near GPT/Claude in terms of response quality.

[–] [email protected] 4 points 6 days ago (1 children)

I run ollama and auto1111 on my desktop when it's powers on. Using open-webui in my homelab always on, and also connected to openrouter. This way I can always use openwebui with openrouter models and it's pretty cheap per query and a little more private that using a big tech chatbot. And if I want local, I turn on the desktop and have local lamma and stab diff.

I also get bugger all benefit out of it., it's a cute toy.

[–] [email protected] 1 points 5 days ago (1 children)

How do you like auto1111 as I've never head of it

[–] [email protected] 2 points 2 days ago

The project is a bit out of date for newer models, Though Older ones work great.

I recommand ComfyUi if you want fine grained control over the generation and you like to tinker.

Swarm / Reforge / Invoke if you want neat, up to date UI.

[–] [email protected] 3 points 6 days ago (2 children)

I’ve an old gaming PC with a decent GPU laying around and I’ve thought of doing that (currently use it for linux gaming and GPU related tasks like photo editing etc) However ,I’m currently stuck using LLMs on demand locally with ollama. Energy costs of having it powered on all time for on demand queries seems a bit overkill to me…

[–] [email protected] 1 points 6 days ago (1 children)

I put my Plex media server to work doing Ollama - it has a GPU for transcoding that's not awful for simple LLMs.

[–] [email protected] 2 points 6 days ago

That sounds like a great way of leveraging existing infrastructure! I host Plex together with other services in a server with intel transcoding capable CPU. I’m quite sure I would get much better performance with the GPU machine, might end up following this path!

load more comments (1 replies)
[–] [email protected] 3 points 6 days ago

I have immich machine learning and ollama with openwebui

I use immich search a lot to find things like pictures of the side of the road to post on my community [email protected]

I almost never use the ollama though, not really sure what to do with it other than ask it dumb questions just to see what it says

I use the duckduckgo one when it auto has an answer to something I searched but its not too reliable

[–] [email protected] 2 points 6 days ago (1 children)

Try running an AI Horde worker, it's a really great service!

[–] [email protected] 1 points 5 days ago (1 children)

Not sure I know what that is. As in Hoarder?

[–] [email protected] 4 points 5 days ago

It's a cluster of workers where everyone can generate images/text using workers connected to the service.

So if you ran a worker, people could generate stuff using your PC. For that you would gain kudos, which in turn you can use to generate stuff on other people's computers.

Basically you do two things: help common people without access to powerful machines and use your capacity when you have time to use the kudos whenever you want, even on the road where you can't turn on your PC if you fancy so.

[–] [email protected] 2 points 6 days ago (1 children)

I was able to run a distilled version of DeepSeek on Linux. I ran it inside a PODMAN container with ROCM support (I have an AMD GPU). It wasn't super fast but for a locally deployed and self hosted option the performance was okay. Apart from that I have deployed Fooocus for image generation in a similar manner. Currently, I am working on deploying Stable Diffusion with either ComfyUI or Automatic1111 inside a PODMAN container with ROCM support.

[–] [email protected] 2 points 6 days ago

Didn't know about these image generation tools, besides Stable Diffusion. Thanks!

[–] [email protected] 1 points 5 days ago (2 children)

Can anyone suggest a model for light coding? I'm on a 3070 mobile.

[–] [email protected] 1 points 2 days ago

Qwen coder or the new gemma3.

But at this size using privacy respecting api might be both cheaper and lead to better results.

[–] [email protected] 1 points 5 days ago

Claude is the standard that all others are judged by. But it's not cheap.

Gemini is pretty good, and Qwen-coder isn't bad. I'd suggest you watch a few vids on GosuCoder's YT channel to see what works for you, he reviews a pile of them and it's quite up to date.

And if you use VScode, I highly recommend the Roocode extension. Gosucoder also goes into revising the roocode prompt to reduce costs for Claude. Another extension is Cline.

[–] [email protected] 1 points 5 days ago (1 children)

I am curious about trying an application specific AI. Like just for coding, for instance. I assume the memory requirements would be much lower.

[–] [email protected] 1 points 5 days ago (1 children)

afaik Ollama would fit that bill, but perhaps others can chime in. You could probably run it on your local computer with a small model based on CPU alone.

[–] [email protected] 2 points 5 days ago

I haven’t sunk much time at it, but I’m not aware of any training data focusing on code only. There’s nothing preventing me from running with general purpose data, but I imagine I’d have a snappier response with a smaller, focused dataset, without losing accuracy.

load more comments
view more: next ›