AI Running in Closed Networks — On-Premises Deployment Checkpoints
A record of building an AI Hub for a major domestic card company demo with zero external API calls. Validated against 877 chunks of actual terms and laws, 11/11 golden set accuracy, and audit trails for rejections — plus four pitfalls encountered due to local deployment.
This article outlines the design of an On-Premises AI Hub (demo for a major domestic card company) built for network-segregated environments, and summarizes the pitfalls encountered due to local deployment. The client's name is not disclosed.
Problem
In environments with strict network segregation, such as financial institutions or public agencies, cloud-based LLM services cannot be utilized. External API access is fundamentally blocked, causing many AI adoption attempts to fail at the initial stage. The practical challenge of implementing AI features using only internal resources under these constraints is the starting point of this article.
Why It Is Difficult
Since external APIs cannot be used, all components—including language models, embeddings, and speech recognition—must be run directly on internal infrastructure. This goes beyond simple technical difficulty and requires a fundamental change in system architecture. Additionally, demos based on synthetic data cannot guarantee reliability in actual business environments, so performance validated against real terms and laws is essential.
From an audit perspective, it is more important to clearly record which questions were rejected and why, rather than simply generating answers. Consistency in rejection logs has emerged as a key challenge for regulatory compliance and transparency. These requirements demand a design philosophy entirely different from development methods in cloud environments.
What We Did
We built an AI Hub consisting of document processing, knowledge base, assistant, call analysis, and audit modules for a demo for a major domestic card company. It was designed to correspond 1:1 with the client's five-project roadmap. The corpus was composed of 13 publicly available laws and terms documents, ranging from personal membership terms to the Credit Information Act and Debt Collection Act, to ensure realism.
We used a local 35B model for answer generation, a 3B model for classification, and local STT for speech recognition. We adhered to security regulations by enforcing a code-level block on external calls via the OFFLINE_MODE setting. We established a validation system measuring accuracy and citation consistency through 11 golden set questions.
Blocked or rejected questions were designed to be recorded in audit logs with reason badges. The call analysis feature scores compliance with mandatory disclosures and the use of prohibited expressions to manage risks. These decisions were essential steps to prove the feasibility of AI operations in network-segregated environments.
Empirical pitfalls included a rate limiter failing to recognize role headers, treating all requests as guests and causing 429 errors. There was also an issue where early returns in input validation and blocking paths occurred without logging, resulting in missing rejection records. Response cache TTL settings can lead to performance illusions during re-measurement, and resource contention on shared memory equipment was confirmed to cause sharp drops in inference speed.
By the Numbers
| Metric | Value | Notes |
|---|---|---|
| Corpus Size | 13 documents / 877 chunks | Based on actual public terms and laws |
| Golden Set Accuracy | 11/11 | Including citation consistency |
| Response Time | 6–10 seconds | Based on local 35B model |
| External API Calls | 0 | Enforced block via OFFLINE_MODE |
| Call Analysis Score | Normal 92·95 / Defect 15 | Evaluation of mandatory disclosures and prohibited expressions |
| Audit Records | 11/11 | Including 4 blocked cases |
What This Article Says
AI can function effectively even in network-segregated environments. However, since cloud models cannot be used, the accuracy of evidence citation, transparency of rejection records, and discipline during re-measurement must be at the center of the design. As pre-deployment checkpoints, you must always perform: creating 10 golden set questions based on actual documents, enforcing code-level blocks on external calls, enabling rejection logging, clearing cache before re-measurement, and checking for equipment sharing.
Further Reading
- Case Study: On-Premises AI Hub
- Enterprise AI Solutions — On-Premises AI Adoption
- Request a 30-minute demo with your documents
CoreCode is a Korean development team with a legal entity in Ho Chi Minh City. We build enterprise AI, web and apps for Vietnamese market entrants, and Toss Apps in Toss mini-apps, delivering products that are operational, not just demos. Tell us your current status, and we will summarize it with numbers in a 30-minute consultation — Request a 30-minute consultation.