
Local LLM Server: What Hardware Does a Company Need?
Local LLM server: workstation, RTX PRO 6000 card or H200. What each tier serves with Qwen 3.8 27B, measured throughput and prices checked in November 2026.
A local AI server is a machine owned by the company that runs a language model (a local LLM) without sending data to an outside API. The reference model in this article, Qwen 3.8 27B, fits on a 24 GB graphics card and generates 40 to 75 tokens per second there for one user. On an H100 server GPU, with 50 concurrent requests, each user gets 35 tokens per second with SGLang, and less than one with Ollama.
In other words, it is the number of concurrent requests that determines the hardware to buy, and therefore the budget, together with the length of the documents sent. The same model runs on a card that costs €2,333 excluding VAT and on an H200 NVL at €32,533 excluding VAT; only the second one serves a whole company.
We gathered the published measurements for three tiers (a workstation, a one-card server, an H100 or H200), with prices checked on 16 November 2026 at French retailers, and for two uses that expect different things from a local LLM: software development, where the reference is Claude Code, and chatbots or business agents, where the reference is the speed of a chat assistant. Whether to host a model yourself at all, with the full cost and the break-even point against an API, is covered in our article on on-premise LLMs.
Why use Qwen 3.8 27B as the reference?
Because, as far as we know, it is the best-rated open model that fits on a single card. Qwen 3.8 27B is a dense model with 27 billion parameters, released by Alibaba under the Apache 2.0 license. On the Artificial Analysis index, it scores about five times as much as gpt-oss-120b, a model four times larger: our article Qwen 4 27B vs Qwen 3.8 27B covers its performance and architecture. At ATG, it is the model that drafts replies to messages and runs our FinanceBench benchmark.
Why a dense model rather than an MoE, the "mixture of experts" models that only activate part of their parameters for each token? The MoE is faster: on the same consumer card, a developer measured 31 tokens per second for the dense Qwen 3.6 27B and 98 for the Qwen 3.6 35B-A3B MoE. But an MoE has to keep all its experts in memory, since any of them may be needed for the next token: this MoE weighs 71.9 GB in BF16, against 55.6 for the dense model. And the dense model answers better, 77.2 against 73.4 on SWE-bench Verified according to Qwen. On a one-card server, memory and quality matter more than the speed of a single user: our mixture of experts guide explains the difference.
That changes the hardware math. In our article on on-premise LLMs, a model of about 100 billion parameters was still the minimum once tasks get complex, with two server GPUs to match. A 27-billion-parameter model that beats it brings the question down to a single card. Its weights take 55.6 GB in BF16 and 30.9 GB in FP8, and about 17 GB in 4-bit (the Q4_K_S version weighs 16.68 GiB).
Qwen 4 27B, announced by Alibaba, has not been released: as of 16 November, the Qwen organization on Hugging Face has no Qwen 4 repository. Hardware chosen for Qwen 3.8 27B will probably run it if it keeps its size, but its real memory footprint is unknown, so some headroom is worth paying for.
Development or chatbot: is Qwen 3.8 27B good enough?
The two uses do not need the same quality. On the Artificial Analysis index, Qwen 3.8 27B scores 20 points, on par with Claude Haiku 5.5 at low effort (19), but far from Claude Sonnet 5.5 (55), Claude Opus 5.5 (59) and Claude Fable 5.2 (65).
For software development, Claude remains the reference, and the gap is clear. On Terminal-Bench 2.1, a set of command-line programming tasks, Vals.ai measures 87.6% for Claude Opus 5.5, 83.2% for Claude Sonnet 5.5 and 58.4% for Qwen 3.8 27B, under the same conditions. Qwen claims better results on its own tests: 61.7 on SWE-bench Pro, against 53.4 for Claude Opus 4.6, a Claude model from an earlier generation. We would not replace Claude Code with a local Qwen to have an agent work on a large codebase; a local workstation is better suited to code that must not leave the company and to simple tasks.
For a chatbot or a business agent, Qwen 3.8 27B is very good, except for very complex cases. That is our view as users: at ATG, it drafts replies to messages, and our FinanceBench benchmark reaches 92% correct answers with it, on 150 analyst questions over 368 financial reports. Replacing it with Gemini 3.8 Flash, or giving it more thinking time, does not change the score significantly, as we explain in our article on auditing a RAG system: quality comes first from retrieving the right passages. The cases that justify a frontier model are long reasoning across several documents and agents that chain many steps.
What decides the hardware for a local LLM?
Memory comes down to one rule: the model weights, plus the KV cache, the working memory of each conversation, which grows with the length of the text read and the number of users. Our article on on-premise LLMs walks through the calculation. Qwen 3.8 27B has a small KV cache: only 16 of its 64 layers keep one, about 2.1 GB for a 32,000-token conversation.
Three questions then decide the tier:
- How many requests arrive at the same time at peak hours. A concurrent request is not an employee: a user spends most of their time reading or writing, not waiting for an answer.
- How much text the model reads per request. A chat question is a few hundred tokens; a RAG system or an agent reading contracts sends tens of thousands.
- What speed each user will accept. Because Qwen 3.8 27B thinks before answering, it produces far more text than comparable models on average: in our view, below 20 to 30 tokens per second, the wait becomes noticeable. The benchmarks below show what users are already used to.
What speed do users expect?
Two references set expectations. For a developer used to Claude Code, it is Anthropic's API, which Claude Code calls. For a chat user, it is the speed of consumer assistants, and that has changed scale: Mistral's Le Chat has answered at over 1,100 tokens per second with Mistral Large 2 since February 2025, on Cerebras chips, and in January 2026 OpenAI signed for 750 megawatts of Cerebras compute, aiming for "faster responses, more natural interactions".
| Service | Model | Speed for one user | Type of figure |
|---|---|---|---|
| Anthropic API (Claude Code) | Claude Opus 5.5 | 105 to 116 tokens/s | Artificial Analysis measurement |
| Anthropic API | Claude Sonnet 5.5 | 126 to 152 tokens/s | Artificial Analysis measurement |
| Anthropic API | Claude Haiku 5.5 | 209 to 257 tokens/s | Artificial Analysis measurement |
| Mistral API | Mistral Large 4 | 81 to 88 tokens/s | Artificial Analysis measurement |
| Alibaba API | Qwen 3.8 27B | about 40 tokens/s | Artificial Analysis measurement |
| Groq | Qwen 3.8 27B | 450 tokens/s | Groq's claim |
| Cerebras | Qwen 3.8 27B | about 1,850 tokens/s | Cerebras's claim |
| Local workstation (RTX 3090 to 5090) | Qwen 3.8 27B | 40 to 75 tokens/s | Hardware Corner measurement |
Artificial Analysis measurements checked on 16 November 2026 (five-day average), depending on the model's effort level.
Chart: speed for one user, in tokens per second, of the services users already know. Claude Opus 5.5 on the Anthropic API, used by Claude Code: 105 to 116. Claude Sonnet 5.5: 126 to 152. Claude Haiku 5.5: 209 to 257. Mistral Large 4 on the Mistral API: 81 to 88. Qwen 3.8 27B on the Alibaba API: about 40, at Groq: 450 (claimed), at Cerebras: about 1,850 (claimed), on a local workstation: 40 to 75. Reference bands: developer 100 to 150, chat user 500 and above
We take two benchmarks from this: 100 to 150 tokens per second for a developer, 500 and above for a chat user, the threshold we already use for demanding uses in our article on on-premise LLMs. A local workstation is therefore half as fast as Claude Opus 5.5, and twenty-five to forty-five times slower than the same model at Cerebras. A shared server under load, at 32 to 42 tokens per second per user, gives about a third of Claude's speed. Claude Code's fast mode even promises up to 2.5 times more. A developer moving from Claude Code to an in-house server will notice the difference; a chat user used to Le Chat will notice it even more.
A server's total throughput is shared between requests: a server producing 1,600 tokens per second for 50 requests gives each about 32 tokens per second. The figures below always keep the two apart.
A workstation for one user: which PC for local AI?
For a developer, a pilot or personal use, a 24 or 32 GB graphics card is enough. Hardware Corner measured Qwen 3.8 27B in 4-bit with llama.cpp, updated on 17 August, at a 4,000-token context:
| Machine | Memory | Measured speed (one user) | Price on 16/11/2026 (France) |
|---|---|---|---|
| RTX 5090 | 32 GB | 75 tokens/s | €7,750 excl. VAT the card, €10,833 excl. VAT as a full workstation |
| RTX 4090 | 24 GB | 46 tokens/s | no longer produced |
| RTX 3090 | 24 GB | 40 tokens/s | second-hand only |
| Apple M5 Max | 128 GB | 31 tokens/s | €5,616 excl. VAT for a 128 GB Mac Studio |
| GB10 (the DGX Spark platform) | 128 GB | 12 tokens/s | €6,548 incl. VAT (ASUS Ascent GX10) |
Chart: speed of Qwen 3.8 27B in 4-bit for one user on a workstation, against reference speeds. RTX 5090 32 GB: 75 tokens per second (€7,750 excluding VAT the card, €10,833 the workstation). RTX 4090 24 GB: 46. RTX 3090 24 GB: 40. Radeon AI PRO R9700 32 GB: 20 to 30 (€2,333). Apple M5 Max 128 GB: 31 (€5,616 the Mac Studio). GB10, the DGX Spark platform, 128 GB: 12 (€6,548 incl. VAT). Every bar stays below the developer band, 100 to 150 tokens per second, and far from the chat user band, 500 and above
AMD's card, the 32 GB Radeon AI PRO R9700, is the cheapest of the lot at €2,333 excluding VAT. ServeTheHome measured it at 20 to 30 tokens per second for a single user; AMD claims up to 51.8 with speculative decoding, a technique that guesses several tokens at once and then checks them.
128 GB unified-memory machines buy capacity, not speed. NVIDIA's DGX Spark, the Mac Studio or Ryzen AI Max+ mini PCs load a 27-billion-parameter model without trouble, but their memory is slow for a dense model, which reads all its parameters for every token. On DGX Spark, a developer measured 11.2 tokens per second without speculative decoding, 19 to 28 with it. Worse, reading a 128,000-token document takes 150 seconds before the first word. These machines suit MoE models better, which only read part of their parameters for each token.
Prices have followed the memory shortage. NVIDIA raised the DGX Spark from $3,999 to $4,699 in February, citing "worldwide constraints in memory supply", then to $6,950 in October. In France, the RTX 5090 reached €8,899.95 on 3 November according to the GinjFo price tracker, before falling back to €5,609.90 on 14 November according to Pro-gamer. At this tier, a price checked on a Monday does not always hold the following Monday.
4-bit costs almost nothing in quality. The INT4 version published by Red Hat keeps 98.5 to 101% of the scores of the original model depending on the task (87.9 against 89.2 on GPQA Diamond). Quesma measured no notable loss in Q4_K_M, and a clear drop only at 2 bits. We prefer the versions published by Qwen, NVIDIA or Red Hat: some community versions have shown reasoning loops.
Without a graphics card, only a small model remains an option, for simple tasks. For Olala mini 7A1B, its first LLM unveiled on November 17, 2026, OVH claims 50 tokens per second on a standard processor: it is a mixture of experts that puts only one billion of its 7 billion parameters to work on each token. The figure is self-reported, with no test conditions, and a model of this size does not play in the same league as Qwen 3.8 27B: it is aimed at classification or extraction, not drafting or code. We analyze it in our article on Olala mini.
A workstation has no redundancy and no access control, and only serves one person well at a time: it is a pilot tool.
Ollama, vLLM or SGLang: which inference software to choose?
The software that runs the model, the inference engine, deserves as much attention as the card: they are not all equal, and the gap is one to fifty on the same GPU. Ollama and LM Studio are desktop tools: they serve one person very well. For several users you need a server engine, which batches requests to process them together. Phil Winder (Winder.AI) compared five engines on the same rented H100, with Qwen 3.8 27B in 8-bit and requests of about 900 tokens. Here is the speed each user gets, in tokens per second; in brackets, the GPU's total throughput, which we divide by the number of requests:
| Engine | 1 request | 10 concurrent | 50 concurrent | Median wait at 50 |
|---|---|---|---|---|
| SGLang 0.5.20 | 81 | 63 (627 in total) | 35 (1,725 in total) | 1.5 s |
| vLLM 0.30.0 | 76 | 59 (589) | 32 (1,610) | 2.1 s |
| llama.cpp | 56 | 14 (138) | 1.9 (97) | 8.2 s |
| Ollama 0.34.4 | 33 | 3.3 (33) | 0.7 (33) | 381 s |
Chart: speed of Qwen 3.8 27B in 8-bit for each user, on the same H100, by engine, at 1, 10 and 50 concurrent requests. SGLang 0.5.20: 81, 63 and 35 tokens per second, median wait of 1.5 s at 50. vLLM 0.30.0: 76, 59 and 32, 2.1 s. llama.cpp: 56, 14 and 1.9, 8.2 s. Ollama 0.34.4: 33, 3.3 and 0.7, 381 s. Total throughput measured by Winder.AI in September 2026, divided by the number of requests
"Ollama served 33," as Winder.AI puts it: with this architecture, Ollama handles requests one at a time. Seen from each user, at 50 concurrent requests, SGLang delivers 35 tokens per second and Ollama 0.7: fifty times less, on the same card. An Ollama pilot that works for one person slows down for everyone once the team joins in, and the card is not to blame. Tuning matters too: with its default settings, SGLang capped itself at 20 requests at a time, and the wait before the first word rose to 10 seconds.
A one-card server for a team
The next tier is a server card in a hosted machine. NVIDIA's RTX PRO 6000, with 96 GB of memory, holds the model and many conversations in memory. Paralon Cloud measured it with vLLM in 4-bit, with 256-token answers: 106 tokens per second for one request, 636 in total for 8 concurrent requests, 1,342 for 32. That is, per user, about 80 tokens per second at 8 and 42 at 32 (our calculation).
A card like this serves about thirty concurrent requests at the speed of the vendor's own API. It costs between €16,142 and €16,600 excluding VAT at LDLC Pro depending on the version, and more elsewhere. No French integrator publishes the price of a one-card rack server: the only GPU server in LDLC Pro's catalogue, with three RTX PRO 6000 cards, is listed at €89,562 excluding VAT. A paradox spotted by Pause Hardware: in the United States, Dell sells a complete workstation with this card for about €12,600, less than the card alone at European retail.
The older server card, the 48 GB L40S, rents for €1.40 an hour at OVHcloud, but we found no published measurement of Qwen 3.8 27B on it.
H100 and H200: serving the whole company
Beyond a few dozen concurrent requests, data center GPUs take over. On an 80 GB H100, Winder.AI measures 1,610 to 1,725 tokens per second at 50 concurrent requests, or about 32 to 35 tokens per second per user, with 1.5 to 2.1 seconds of wait before the first word. The H100 is no longer sold new, but it rents for €2.80 an hour at OVHcloud, or €1,940 a month around the clock.
Its successor, the 141 GB H200 NVL, costs €32,533 excluding VAT at LDLC Pro. In 4-bit with vLLM, Paralon Cloud measures 3,342 tokens per second in total for 128 requests, about 26 per user; at 64 requests, about 48. The newer B200 does no better on this model under load: according to Paralon, it falls 12 to 14% behind the H200 between 32 and 128 requests.
Document length matters as much as the number of users. On an H200 with SGLang in FP8, measurements published on the project's repository give, for 1,000-token requests, 98 tokens per second for a single user, 83 at 16 users and 50 at 64. With 8,000 input tokens, speed drops to 29 tokens per second at 64 users, and the wait before the first word goes from 2.6 to 19 seconds. A chat assistant and a RAG system reading contracts do not need the same server, even with the same number of users.
What about chat speed?
No published measurement of Qwen 3.8 27B on a GPU reaches 500 tokens per second per user, even for a single user. Speculative decoding, which guesses several tokens at once and then checks them, gets closer: on an H200 with SGLang, it takes a single user from 98 to 196 tokens per second, and DataPacket reports up to 360 tokens per second on an RTX PRO 6000 with the DFlash2 method. But the advantage fades under load: on the same H200, only 66 tokens per second per user remain at 32 users. As the HyperQwen authors put it, "speculation wins below ~8 concurrent users, plain batching above".
A fast local chat therefore costs cards: to get close to 200 to 360 tokens per second, each GPU serves only a few users at a time, and the budget follows the number of concurrent users rather than the headcount. Beyond that, specialised hardware is needed: the speed of Mistral's Le Chat or of Cerebras comes from chips designed for inference, available through APIs, not from a standard GPU server. For an internal chatbot, we consider an answer streaming at 40 to 80 tokens per second comfortable: a 300-word answer, about 400 tokens, is written in five to ten seconds (our calculation).
What a company can take away
Chart: three hardware tiers for Qwen 3.8 27B in 4-bit against reference speeds. RTX 5090 workstation (€10,833 excluding VAT, full workstation): 75 tokens per second for one user. RTX PRO 6000 96 GB team server (€16,142 excluding VAT, card only): 80 tokens per second per request at 8 concurrent requests, 42 at 32. H200 NVL 141 GB (€32,533 excluding VAT, card only): 48 at 64 requests, 26 at 128. Every bar stays below the developer band, 100 to 150 tokens per second (Claude in Claude Code), and far from the chat user band, 500 and above (Le Chat, Cerebras)
| Tier | Hardware | Price on 16/11/2026 (France) | Measured concurrent requests | Speed per user |
|---|---|---|---|---|
| Workstation | RTX 5090, full workstation | €10,833 excl. VAT | 1 | 75 tokens/s |
| Workstation | Radeon AI PRO R9700, card | €2,333 excl. VAT | 1 | 20 to 30 tokens/s |
| Team server | RTX PRO 6000 96 GB, card | €16,142 to €16,600 excl. VAT | 8 / 32 | 80 / 42 tokens/s |
| Company | H100 80 GB, rented | €2.80 an hour | 50 | 32 to 35 tokens/s |
| Company | H200 NVL 141 GB, card | €32,533 excl. VAT | 64 / 128 | 48 / 26 tokens/s |
Measurements in 4-bit, except the H100 (8-bit); short requests; per-user figures calculated by ATG from the published total throughput.
Our reading of these figures: the load is what to measure before buying. The number of concurrent requests and the length of the documents can be logged during a pilot, and they decide the tier far more than the model's name. Renting an H100 at OVHcloud for a few days is enough to replay that load with vLLM before ordering a card whose lead time sometimes exceeds two weeks. An Ollama pilot shows the quality of the answers, but nothing about how the system holds under load.
The card is only one line of the budget. Hosting, electricity, maintenance and above all staff weigh more: our article on on-premise LLMs puts numbers on the full cost and the break-even point against an API.
The same model is also available through APIs hosted in France: OVHcloud serves Qwen 3.8 27B and Scaleway offers it, which allows a pilot with no hardware at all. Our page on sovereign AI describes what European hosting settles, and what it does not.
Frequently asked questions about local AI servers
What PC do you need to run AI locally?
For a model like Qwen 3.8 27B in 4-bit, a 24 or 32 GB graphics card is enough for one user: Hardware Corner measures 40 tokens per second on an RTX 3090, 46 on an RTX 4090 and 75 on an RTX 5090. Machines with 128 GB of unified memory, like the DGX Spark or the Mac Studio, load larger models but generate this dense model more slowly.
Can Qwen 3.8 27B replace Claude for coding?
Not for agentic coding on a large codebase. On Terminal-Bench 2.1, Vals.ai measures 87.6% for Claude Opus 5.5, 83.2% for Claude Sonnet 5.5 and 58.4% for Qwen 3.8 27B, and the Artificial Analysis index puts it at 20 points against 59 for Opus 5.5. A local Qwen is better suited to code that must not leave the company and to simple tasks. For a chatbot or a business agent, on the other hand, it is usually enough.
How many users can a local AI server handle?
It depends on the GPU, the engine and the length of the documents. With vLLM and Qwen 3.8 27B in 4-bit, an RTX PRO 6000 serves 32 concurrent requests at about 42 tokens per second each, according to Paralon Cloud; an H200 NVL serves 128 at about 26. The number of employees served is higher, since most of them are reading or writing while others wait for an answer.
Is a local LLM as fast as Claude or Le Chat?
Generally not. On a workstation, Qwen 3.8 27B runs at 40 to 75 tokens per second; Artificial Analysis measures Claude Opus 5.5 at about 110 tokens per second on Anthropic's API, and Cerebras announces about 1,850 tokens per second for Qwen 3.8 27B. Cerebras also provides the compute behind Mistral's Le Chat, which answers at over 1,100 tokens per second, and OpenAI signed with it for 750 megawatts. A shared server under load drops to 30 or 40 tokens per second per user.
How much does the hardware for a local AI server cost?
As of 16 November 2026, at the French retailer LDLC Pro: €2,333 excluding VAT for a 32 GB Radeon AI PRO R9700, €10,833 for a full workstation with an RTX 5090, €16,142 to €16,600 for a 96 GB RTX PRO 6000 and €32,533 for an H200 NVL, card only. A server with three RTX PRO 6000 cards is listed at €89,562. Renting an H100 costs €2.80 an hour at OVHcloud.
Is a DGX Spark enough for a local LLM?
For testing models, yes; for serving Qwen 3.8 27B quickly, no. Its 128 GB of memory loads large models, but it generates this dense model at 11 or 12 tokens per second without speculative decoding, and takes 150 seconds to read a 128,000-token document. Its price rose from $3,999 to $6,950 in a year.
Should you quantize a local LLM to 4-bit?
For Qwen 3.8 27B, the loss is small: Red Hat's INT4 version keeps 98.5 to 101% of the original model's scores, and Quesma measures no notable loss in Q4_K_M. Quality drops clearly from 2 bits. Versions published by Qwen, NVIDIA or Red Hat are safer than some community versions, which have shown reasoning loops.
Should you wait for Qwen 4 before buying hardware?
Qwen 4 27B has not been released as of 16 November 2026, and its real memory footprint is unknown. Hardware sized for Qwen 3.8 27B will probably run it if it keeps its size, but the preview of the Qwen 4 architecture adds large embedding tables: keeping memory headroom is safer than waiting.
Before choosing a card, one question deserves a week of measurement: how many requests really arrive at the same time, and how much text they send. Analysing a pilot's logs is how you get an answer that can save you tens of thousands of euros!
Qwen 3.8 27B without running a server, with ATG
With ATG, all of this is already taken care of. Choosing the card, the inference engine and the quantization, measuring the load, maintenance: our enterprise RAG comes ready to use, hosted by OVHcloud in France. The model that writes its answers is in fact Qwen 3.8 27B, and those answers arrive fast while staying relevant, which a demo shows on your own questions.


