Adding AI to Software You Already Have: LLM Integration in Practice

LLM integration means adding language-model capabilities to the software your business already runs, and in practice it's an API call wrapped in ordinary engineering: your application sends text and data to a model, gets a structured response back, validates it, and acts on it. You don't need to replace your systems, adopt a platform, or train anything. Most of the work is the wrapping, not the model, and that's good news, because wrapping is a solved discipline. This post covers the patterns that work, the ones that waste money, and the numbers to expect.

Start With an API Call, Not a Platform

There's a strong gravitational pull, the moment a company decides to "do AI," toward buying something large. An AI platform. An enterprise suite. A vendor ecosystem with its own certification program. Resist it, at least at first. The core capability you're after is available as a metered API from several providers, callable from any codebase written in the last twenty-five years, and you can prove or disprove your use case with a few hundred lines of code before signing anything.

This matters because the failure mode of platform-first thinking is spending six months on procurement and architecture for a use case nobody validated. The API-first path inverts that: take one real task, wire up the narrowest possible integration, run it against a few hundred real examples from your business, and measure. If the model can't do the task acceptably, you've spent weeks, not quarters, finding out. If it can, you now know exactly what to build properly.

Cost is rarely the obstacle people expect. API pricing is per token, roughly per word processed, and typical business tasks like summarizing a service ticket or extracting fields from an email cost fractions of a cent to a few cents each. A feature used a thousand times a day typically runs tens of dollars a month in model fees, sometimes less with smaller models, which are entirely adequate for most classification and extraction work. The bill that matters is engineering time. Budget accordingly, and spend model dollars freely during prototyping, because a week of engineer time costs more than a year of most SMB inference bills.

How Do You Integrate an LLM With Software You Already Have?

Three patterns cover most of what small and mid-sized businesses actually need.

The first is the sidecar feature: an AI-powered addition inside an existing screen. A "summarize this customer's history" button in your CRM. A "draft a response" action in your ticketing tool. Architecturally it's a new endpoint in your existing application that gathers the relevant records, sends them to the model with instructions, and displays the result. The user stays in the software they already know, which does wonders for adoption. Nobody has to learn a new tool to get the benefit.

The second is the background pipeline: no user interface at all. Documents, emails, or records flow in; the model classifies, extracts, or enriches them; validated results flow into your database or ERP. Our piece on AI document processing describes the fullest version of this pattern. Pipelines are where the highest ROI usually hides, because they run all day without anyone clicking anything.

The third is conversational access to your data: a chat interface that answers questions using your actual records and documents. This is the pattern people picture first and should usually build last. It's the hardest to evaluate, the easiest to oversell internally, and the most sensitive to data quality. When it's right, it's genuinely valuable; we cover doing it properly in the post on building an AI knowledge base from company documents.

Whichever pattern you pick, the integration touches your systems of record, and that's where the real engineering lives: authentication, permissions (the model must never retrieve data the requesting user couldn't see themselves), rate limits, retries, and the mundane plumbing of getting clean data out of a database that's accreted quirks since 2009. This is why LLM integration projects are staffed like software engineering projects with an AI specialist, not the other way around.

Retrieval Beats Fine-Tuning for Most SMB Cases

At some point someone will ask: shouldn't we train the model on our data? Almost always, no. What you want is retrieval: at the moment of each request, your system looks up the relevant records, documents, or history and hands them to the model as context. The model reasons over your data without ever being trained on it.

Retrieval wins for most SMB cases on every axis that matters. It's current: change a price in the database and the very next answer reflects it, whereas a fine-tuned model knows only what it was trained on months ago. It's inspectable: you can see exactly what the model was shown, which makes wrong answers debuggable. It's cheaper by an order of magnitude or two. And it doesn't create a maintenance liability, a custom model that must be retrained, re-evaluated, and re-deployed every time your business changes.

Fine-tuning has legitimate uses: enforcing a rigid output format at very high volume, or squeezing latency and cost on a narrow task once retrieval has already proven the use case. Those are optimizations, not starting points. If a proposal leads with fine-tuning before retrieval has been tried, ask why. The honest answer is usually that fine-tuning sounds more impressive in a deck.

Structured Outputs, Guardrails, and Failure Modes

The single most consequential technical decision in an LLM integration is refusing to accept freeform text. Every response the model returns to your system should be structured output, JSON conforming to a schema you define, with typed fields and enumerated values. Providers support enforcing this at the API level now. A response that says the ticket priority is "urgent-ish" simply cannot exist; the field is an enum and the model must pick from your list.

Structure enables the second layer: validation in plain deterministic code. Extracted part numbers get checked against the item master. Dates get sanity-checked. Totals get re-added. Referenced customers must exist. This layer is your primary hallucination guardrail, and it's just code. A model can invent a plausible-looking PO number; it cannot invent one that survives a lookup against your actual open POs.

Then design for the failure modes you'll actually see. Models occasionally return something malformed under load; wrap calls in retries with validation between attempts. They perform worse on inputs unlike anything in your prompt examples; route low-confidence results to a human review gate instead of guessing. They're nondeterministic; the same input can produce slightly different output, so anything requiring exact repeatability belongs in regular code, not in a prompt. And provider outages happen; decide up front whether the feature degrades gracefully or blocks a business process, and never put an LLM call in the synchronous path of something like order intake without a fallback.

One anti-pattern deserves its own sentence: if the task is a deterministic transformation of already-structured data, a script or spreadsheet macro beats an LLM on cost, speed, and correctness, every time.

Evaluation: How You Know It Still Works Next Month

A prompt that works today is not a system. Providers update models. Your document formats drift. Someone "improves" the prompt on a Friday. Without evaluation, you find out about regressions from angry users; with it, you find out from a failing test.

The practice is unglamorous and effective. Collect a few hundred real examples of the task, including the ugly ones, with known-correct answers. Before any change ships, whether a prompt tweak, a model version bump, or a retrieval adjustment, run the full set and compare scores against the current baseline. Store every production request and response so you can replay incidents and mine new hard cases for the eval set. Track the human-correction rate from your review gates as a live accuracy signal.

None of this is exotic. It's regression testing, applied to a component that happens to be probabilistic. Teams that do it ship changes confidently for years. Teams that don't end up afraid to touch their own system, which is its own kind of failure. Running these pipelines on monitored, versioned infrastructure rather than someone's laptop is part of the same discipline, and it's where our cloud infrastructure work typically plugs in.

FAQ

Do we need to move our data to the cloud to use LLMs?

No. The common pattern sends only the relevant slice of data per request over an encrypted API call, with major providers offering terms that exclude your data from training. Fully local, self-hosted models exist for stricter requirements, trading some capability and more engineering effort for complete data control.

Which model should we use?

The honest answer: it matters less than your prompt, retrieval, and validation, and it will change during your project's lifetime anyway. Build so the model is a swappable component behind your own interface, evaluate two or three candidates on your actual task, and pick the cheapest one that passes your eval set.

How long does a first LLM integration take?

A proof of concept on real data: typically two to four weeks. A production feature with structured outputs, validation, review gates, and monitoring: typically eight to fourteen weeks depending on how cooperative your existing systems are. Integration with legacy databases is the usual schedule risk, not the AI.

Can we build this with our existing dev team?

Often, yes, especially the application side, which is standard engineering. The parts that benefit from experience are prompt and schema design, retrieval quality, evaluation methodology, and knowing the failure modes in advance. Many teams pair their developers with a specialist for the first project and run subsequent ones alone.

If you have working software and a hunch about where AI could take real work off your team's plate, the distance to a validated answer is shorter than you think. Willowark's AI and automation practice does exactly this kind of integration, from two-week feasibility spikes to production systems. Talk to us about the feature you have in mind, and we'll tell you what it would honestly take.