this post was submitted on 23 Nov 2024
300 points (99.7% liked)

Technology

60112 readers
1798 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 2 points 1 month ago (1 children)

Interesting. A few questions, if I may.

Are you running ollama in the same system as the one consuming it ? If yes does it always run in background ? Does it impact performance of other applications when it runs in background?

[–] [email protected] 6 points 1 month ago

No, Ollama is running on an old PC with a GeForce 1060 and 16gig of ram...

Yes, it's a "webserver" running in the background exposing an API.

However, if I "top" my system, without chatting, it sits at 0% usage; it's only when asking that the system peeks at around 55-70% CPU.

You have to understand there is 2 things here: the server and the model. The server is always running, but requires next to nothing in terms of resources.

The model is what computing your questions, this is the heavy part. It's started on use, then after a delay, it's closing.

TL;DR To answer your real question, you could use Ollama on the same system that you are using.