The question we get asked most often lately: “Can our company deploy its own AI? We don’t want our data going outside.”
The answer is yes — and it’s far easier than it was two years ago. But before you spend money on equipment, there are four things worth thinking through, and getting them straight will save you a lot of wasted money — especially the third one.
We deployed our own large model on the company intranet, to handle technical support and day-to-day Q&A. After working through a full round of pitfalls, the biggest takeaway wasn’t “what hardware should we buy” — it was this: most of what makes people feel that “local AI runs slow and dumb” has nothing to do with inadequate hardware. It comes down to configuration done wrong. And that gap in understanding directly determines whether you waste tens of thousands of yuan.
First: confirm whether you should build it in-house at all
Building in-house is not the default option. For most companies, an online API is actually the better value; in-house only makes sense in the following cases:
- Your data genuinely cannot leave the intranet. That means customer records, contracts, drawings, code, financial data — or an industry with its own compliance requirements. This is the strongest reason to build in-house, and basically the only one that doesn’t require doing the math.
- Long-term, high-frequency use by many people. For occasional use, API costs are negligible; but if dozens of people use it every day and it has to plug into business systems for batch jobs, the math needs to be redone.
- You need long-term stability and control. Online services adjust models, change prices, and retire versions; a self-hosted model stays exactly as it is, which has real value in scenarios that demand long-term consistency.
Conversely: if you just want employees to draft some copy and do some translation, and the data isn’t sensitive — buy an online service and be done with it. A few hundred yuan a month; don’t overcomplicate it.
Second: hardware isn’t as simple as “buy a good GPU”
This is the most widespread misconception. Many people assume that picking the most expensive GPU is enough; in reality, whether you can run a model at all — and how fast — comes down to two other metrics:
- VRAM / memory capacity — decides whether the model fits. However large the model file is, you need that much space to hold it. If it doesn’t fit, it won’t run at all, and no amount of compute power will help. This is a hard threshold, not a matter of “running a little slower.”
- Memory bandwidth — decides how fast it produces text. Generating each token requires moving the relevant data through memory; if bandwidth falls short, extra compute power just sits there idle. The speed bottleneck in local deployment is usually bandwidth, not compute.
There’s a counterintuitive rule for choosing a model: don’t look only at the total parameter count.
Many models today use an architecture where “the total parameter count is large, but only a small fraction is actually invoked on each pass.” What determines speed is how large that invoked fraction is, not the total. So a model with a large total that only uses a small part each time can be several times faster than a smaller model whose parameters all have to be computed. When selecting a model, make sure you get both numbers — looking at the total alone will lead you to a completely wrong conclusion.
Third: the real culprit behind “slow” is probably not hardware
This is the one that took us the longest to figure out, and it’s the most valuable.
After deployment, colleagues reported it was “too slow, not usable.” The first instinct was that the equipment wasn’t good enough and we should spend more on an upgrade. But after checking them one by one, not one of the three “slow” complaints turned out to be a hardware problem:
- Culprit one: it spends all its effort on “thinking” and then produces no output. Most new-generation models have a “deep thinking” capability, and it may be on by default. The result is that it burns a large amount of time and compute on internal reasoning, and after a long wait, the actual response you get is empty. It looks like it has “frozen,” when in fact the parameters simply weren’t set correctly.
- Culprit two: it did answer, but the answer was filed in the wrong place. This one is subtler — the model output normally, but because of some parsing setting, the entire response was classified into the “reasoning process” field while the content field stayed empty. The symptoms look exactly like the first case, and it is very easy to misjudge as “the model isn’t good enough.”
- Culprit three: what’s slow is the first word, not the overall speed. How fast or slow something feels to users comes mainly from the wait between asking a question and the first word appearing, not the output speed afterward. After we enabled a caching optimization aimed at repeated content, that wait dropped from over ten seconds to under two — without touching the hardware at all.
So when you run into “local AI is slow,” the right order is: first confirm whether it’s a configuration problem, then consider adding hardware. Get the order backwards and you can spend tens of thousands of yuan with the problem still there.
There’s another easy trap to fall into: some “optimization parameters” actually do more harm than good on certain hardware. Following generic advice, we increased one parameter and it ended up nearly half as fast as the default value. Any tuning has to be measured and compared on your own equipment — don’t just copy advice off the internet.
Fourth: do the math carefully
The cost of building in-house isn’t just the money spent on equipment:
| Cost item | In-house | Online API |
|---|---|---|
| Upfront investment | One-time equipment cost, starting in the tens of thousands of yuan | Essentially zero |
| Usage cost | Electricity only; heavy or light use makes little difference | Pay per usage, the more you use it, the more it costs |
| Operations staffing | The most commonly underestimated item: deployment, tuning, upgrades, and troubleshooting all require people | Barely anything to manage |
| Data security | Data never leaves the intranet | Depends on the provider’s terms; sensitive data requires caution |
| Capability updates | You have to track and swap in new models yourself | New versions adopted automatically |
The item most often left out of the calculation is operations staffing. Buying the equipment is only the beginning: setting up the environment, tuning parameters, updating models, and troubleshooting all require someone who knows what they’re doing. If your company doesn’t have that person, you either outsource it — or the equipment slowly turns into a machine in the corner of the server room that nobody dares touch.
Recommended approach: don’t buy equipment right away
Start by running an online service for a month or two, and get a clear picture of which use cases people actually use, how often they use them, and whether sensitive data is involved. The cost at this stage is minimal, but it prevents most impulse decisions.
Once you have confirmed “genuinely high frequency + data that truly cannot leave the company,” choose your equipment based on verified, real requirements. By then you know how large a model you need to run and how many people will use it at the same time, so your selection will be far more accurate — and you will not pay for performance you never use.
Common questions
Q: How does the performance of an on-premises deployment compare with large online models?
A: The gap is smaller than many people expect, especially for everyday tasks such as Q&A, summarisation, document processing, and coding. But the very best reasoning capability still belongs to the largest online models. The pragmatic approach is to use both: keep sensitive data and high-frequency tasks on-premises, and send the occasional demanding task to an online model.
Q: Do we need to hire someone dedicated to managing it?
A: Day-to-day operation does not require constant attention, but deployment, tuning, and upgrades need someone who knows what they are doing. For small and medium-sized businesses, the usual approach is to have an IT service provider manage it together with your servers and network under a single operations arrangement, which is more realistic than hiring a dedicated person.
Q: If the data sits on our own machines, is it automatically safe?
A: Not necessarily. On-premises deployment only solves “data does not leave the company.” Internal access permissions, log retention, and who can look up what still need to be designed separately. We have seen cases where internal AI was opened up to all staff, and sensitive documents ended up freely searchable.
Q: Is it too early to invest now — will the equipment be obsolete in two years?
A: That concern is reasonable. That is why we recommend starting from a real, ongoing need rather than deploying AI for its own sake. If you have a clear high-frequency use case, investing now will generate value continuously; if you have not yet worked out what you would use it for, waiting is genuinely the better choice.
Thinking about deploying AI on your company intranet, but unsure whether it is worth it or how to choose?
Siwenlide provides IT Operations Outsourcing and Network Equipment Package services for small and medium-sized businesses in Beijing and the surrounding area, and also helps clients with the selection, deployment, and ongoing operations of on-premises AI: we first help you decide whether to build in-house, size the configuration to your real use cases, and then include tuning and day-to-day maintenance in our managed operations, so the equipment does not sit unused because no one knows how to run it. The pitfalls described in this article are ones we hit ourselves, one by one, on our own setup. Visit siwenlide.com (or take a look at our Company Profile first) or call 400-686-2011 to talk it over.
