We shipped an LLM-powered support ticket triage feature for a SaaS client last year, and the version that worked beautifully in our demo failed in a genuinely embarrassing way about one time in twenty once real customer tickets started flowing through it. Real tickets are messier than demo tickets: typos, half-finished sentences, customers pasting in a stack trace with no context, tickets written in a mix of English and another language. The gap between "impressive prompt in a notebook" and "reliable feature in production" turned out to be most of the actual engineering work.

The first lesson: stop writing prompts like instructions to a person

Our first version of the triage prompt read like a memo: "Please categorize the following support ticket into one of these categories, and explain your reasoning." It worked fine on clean test cases and produced wildly inconsistent output formats on real ones — sometimes a paragraph of reasoning before the category, sometimes after, sometimes the model would helpfully add a fourth category we hadn't asked for because a ticket seemed to warrant it. Every one of those variations broke our downstream parsing code.

What actually fixed it was treating the prompt less like an instruction and more like an API contract, with the output format constrained as tightly as the input.

const systemPrompt = `You are a support ticket classifier. Respond with ONLY valid JSON matching this exact schema, no other text:
{
  "category": "billing" | "technical" | "account" | "other",
  "confidence": number between 0 and 1,
  "needs_human_review": boolean
}

If the ticket doesn't clearly fit one category, set needs_human_review to true rather than guessing.`;

That last line mattered more than anything else in the prompt. Explicitly giving the model permission to say "I'm not confident" instead of forcing a confident-sounding wrong answer cut our misclassification rate substantially, because it stopped forcing a binary choice on genuinely ambiguous tickets. This is a pattern we now use everywhere: an escape hatch in the schema is worth more than a longer, more detailed instruction trying to cover every edge case in advance, because you will never actually cover every edge case in advance.

Few-shot examples beat clever wording, every time

We spent a full day trying to word-tune the instruction text to fix edge cases — tickets that were really two issues combined, tickets that were sarcastic, tickets in broken English. Marginal gains, a lot of wasted time. What actually moved the needle was adding four or five real, anonymized example tickets directly in the prompt, each with the correct classification already worked out, right before the actual ticket to classify. Showing the model what "good" looks like, concretely, consistently beat trying to describe "good" in the abstract. If you're stuck tuning a prompt through wording alone, stop and add examples instead. It's a more reliable lever than almost anything else you can pull.

The thing that actually made it production-ready: validation, not better prompting

Here's the part that felt like giving up but was actually the right call: no amount of prompt engineering gets you to 100% reliable structured output from an LLM, and trying to chase that last percentage point with ever more elaborate prompts is a losing trade against just validating the output programmatically and having a clear fallback.

Parse the response as JSON; if it fails, that's a signal, not a crash — log it and route to human review
Validate the category against the actual allowed enum, don't trust the model didn't invent a new one
If confidence is below a threshold we tuned empirically against real support agent agreement, route to human review regardless of what category the model picked

Once we added that validation layer, the *feature* became reliable even though the model's raw output never became perfectly reliable. That distinction matters. You're not trying to make the model never make a mistake. You're trying to build a system around the model that catches mistakes before a customer or a downstream process sees them.

Temperature is not a knob you set once and forget

We initially used the same temperature setting, 0.7, copied from an example in the API docs, for both a creative product-description generator and this classification task. That was wrong in both directions. Classification wants near-zero temperature — you want the same ticket to get the same category every time, consistency over creativity. The product description generator wanted more variation between calls so it didn't produce the same three sentence structures repeatedly. Match the temperature to whether you actually want determinism or variety; the default in whatever example code you copied from was tuned for someone else's use case, not yours.

The honest summary

Getting an LLM feature to work in a demo is maybe 20% of the actual effort. The other 80% is structured output contracts, validation layers, sensible fallback paths, and being honest with yourself about where the model will be wrong so you can catch it before your customer does. Prompt engineering, in the narrow sense of clever wording, is a real skill, but it's a much smaller part of shipping a reliable AI feature than most getting-started guides suggest.