A 3B Model Fit On My VPS But It Still Couldnt Run My Hermes Agent
A 3B Model Fit On My VPS But It Still Couldnt Run My Hermes Agent

A 3B Model Fit on My VPS but It Still Couldn’t Run My Hermes Agent

Back in June, I spent about a month trying to self-host an AI agent called Hermes. Most of that month went into learning, one failure at a time, that “the model runs” and “the model works” are two very different claims.

This isn’t a tutorial where everything clicks by paragraph four. It’s what broke, what I tried next, and the math behind it. Read it before you buy a VPS, not after.

Why I Switched From OpenAI to Hermes Agent

OpenAI’s ChatGPT was fine at first. Then I’d come back the next day to find it had forgotten our context, or was doing the opposite of what I’d asked.

So I moved to Hermes Agent, an open-source agent from Nous Research, for its persistent memory and self-learning.

What I missed is that Hermes needs a big context window. Its docs say it requires at least 64,000 tokens and rejects smaller models at startup because the system prompt, tool schemas, and working state all need room. That one rule shaped everything that followed.

The VPS Setup and What It Really Costs

I chose Contabo’s Cloud VPS 6: 6 vCPU cores, 12GB RAM, 200GB SSD, and 2 snapshots, at € 6/month before VAT for the first 24 months. With backups, my real spend came to about $9.18 a month before any inference.

Inside it, I installed Docker and Ollama. That part was easy.

Small Local Models Hit the Context Wall

I pulled Qwen 3B, Mistral, and Llama. With a small context window, Hermes gave me nothing usable back, not even for “hello, how are you doing.” Hermes’ provider docs explain part of it: Ollama defaults to just 4,096 tokens of context on machines with under 24GB of VRAM.

When the models did answer, replies took close to a minute, which is unworkable for an agent that processes and classifies information.

My 16GB Core i7 laptop did worse. Models looped or produced near nonsense. I didn’t save those logs, which I regret.

The Qwen 2.5 3B Lesson

Qwen 2.5 3B got furthest. I ran a custom build with num_ctx set to 65536, yet Ollama reported a model context length of 32768. A direct test passed:

ollama run qwen2.5:3b "Reply with one word: ok"

It proved Ollama worked and nothing more.

The 32768 figure wasn’t a bug. Qwen’s model card lists a full context of 32,768 tokens for the 3B model. A bigger setting doesn’t create a window the model was never trained for. It just reserves more memory.

Many guides say memory grows with the square of context. That describes attention compute, which the Transformer paper puts at quadratic cost. The key-value cache grows linearly, as the vLLM PagedAttention paper shows. Using Qwen’s specs, I estimate about 36KB per token at 16-bit, roughly 1.2GB at 32K and 2.4GB at 64K. Memory was never the problem. Speed and capability were.

Why CPU-Only Inference Broke the Agent Loop

Inside the full Hermes and OpenClaw stack, Qwen sat at 100% CPU. OpenClaw would give up with “this response is taking longer than expected,” or Hermes would appear to type forever.

Setting OLLAMA_KEEP_ALIVE=-1 removed cold starts but not slow generation. Keeping a model warm doesn’t make it fast.

OpenRouter, Owl and a Model That Vanished

Next I moved to OpenRouter with a $10 monthly budget. The model that worked was OWL, which I believe was Owl Alpha, a free “stealth” model with a 1 million token context. Decrypt reported that Meituan later revealed it as LongCat-2.0. This model could be unavailable on OpenRouter, so please cross-check,

For three weeks, it ran my full workflow and classified jobs accurately. Then it returned a flat 404. Stealth models are temporary tests, and OpenRouter’s model API carries an expiration date field for deprecations. Every model I pick now has a tested fallback.

Free Models and Other Detours

NVIDIA’s free models were weaker, and OpenRouter’s limits page shows the ceiling: free variants allow 20 requests a minute and 1,000 a day once you’ve bought 10 credits. I needed about 1,000 completions a day, right at the cap.

Hermes offers MiniMax OAuth with no API key or card, but real usage pointed me to MiniMax’s $10 Starter plan, which it has since stopped selling to new buyers. OpenCode Go was $5 for the first month, then $10; it’s now a flat $10. I stayed on OpenRouter. Step 3.5: Flash through Nous Portal or StepFun is on my list, untested.

One win: a profile called salesy logged in through OpenAI Codex using my ChatGPT plan, which includes Codex within usage limits. It covers that profile only, and a subscription isn’t unlimited API capacity.

Cost Per Job, Not Cost Per Token

I found an OpenRouter model at $0.126 per million input tokens and $0.252 per million output tokens. For 1,000 jobs at 2,000 input and 500 output tokens each:

  • Input: 2,000,000 × $0.126/M = $0.252
  • Output: 500,000 × $0.252/M = $0.126
  • Total: $0.378 per 1,000 jobs, or $11.34 for 30,000 a month

That’s optimistic. Real requests carry system prompts, tool definitions, and retries. The figure I log now is total spend divided by jobs actually completed.

Small Operational Fixes That Paid Off

A YAML error at line 532 once broke my config. Now I validate before every gateway restart:

python3 -c "import yaml,pathlib; yaml.safe_load(open(pathlib.Path.home()/'.hermes/config.yaml')); print('YAML OK')"

Shell aliases don’t exist under sudo, so sudo cop gateway install --system failed. Closing an SSH tunnel kept killing dashboards until each profile got its own systemd service.

What to Check Before You Self-Host

Before paying for a VPS, answer four questions. Was your model trained for at least 64K tokens of context? Can it reply inside your gateway’s timeout? Do you have a tested fallback for when a provider disappears? Can you predict cost per completed job?

I’m researching a Mac Mini for local inference but haven’t bought one. Whatever you run, test it on a real job, not “reply with ok.”

I later came across OpenCode; it is cheap, and the $10 plan gives access to almost all models.

 

Author

  • George M

    George M. is a hands-on developer and architect. He holds a B.S. in Computer Science and is a certified specialist with a Google Cloud ML certification. He founded OnNetPulse, where he shares his thoughts on the future of technology and helps readers move beyond theory to real-world implementation.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *