Using LLM for Text Classification with Transformers: Best Practices

We are going to build a production-ready support ticket classifier that categorizes unstructured text into predefined labels using an LLM. I will use Oxlo.ai as the inference backend because its request-based pricing keeps costs flat even when tickets contain long conversation threads, and the OpenAI-compatible API means we can use JSON mode with zero client changes. This tutorial is for engineers who want to replace a traditional ML pipeline with a maintainable prompt-based system. What you'll need Python 3.10 or newer An Oxlo.ai API key from https://portal.oxlo.ai The OpenAI SDK installed with pip install openai Step 1: Configure the Oxlo.ai client I start by instantiating the OpenAI SDK against Oxlo.ai's endpoint. This is a literal drop-in replacement: only the base URL and API key change. from openai import OpenAI import json import os client = OpenAI( base_url="https://api.oxlo.ai/v1", api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY") ) # Verify connectivity response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Say OK"}, ], max_tokens=5 ) print(response.choices[0].message.content) Step 2: Define the classification schema and system prompt The system prompt is the only "training" we need. It constrains the model to a strict JSON schema with allowed categories and a confidence score, so downstream code can parse the output without regex. SYSTEM_PROMPT = """You are a support ticket classifier. Analyze the user's message and return a JSON object with exactly these keys: - category: one of ["Billing", "Technical", "Account", "General"] - confidence: a float between 0.0 and 1.0 - reasoning: one sentence explaining why you chose this category Rules: - Return only the JSON object, with no markdown formatting. - If the ticket mentions payment, invoices, or charges, use Billing. - If it mentions bugs, errors, or integrations, use Technical. - If it mentions login, password, or user settings, use Account. - Use General only if none of the above apply.""" Step 3: Build the classifier function using JSON mode Oxlo.ai supports the OpenAI SDK's JSON mode via response_format, which forces valid JSON output. I keep temperature low to reduce variance across repeated classifications. def classify_ticket(text: str) -> dict: response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Classify this support ticket:\n\n{text}"}, ], response_format={"type": "json_object"}, temperature=0.1, max_tokens=256 ) raw = response.choices[0].message.content return json.loads(raw) # Test on a short example sample = "I was charged twice for my Pro plan this month. Please refund the extra payment." result = classify_ticket(sample) print(json.dumps(result, indent=2)) Step 4: Evaluate against a small labeled dataset Before deploying, I sanity-check the classifier on a handful of known examples. This catches prompt drift without requiring a heavy test harness. test_set = [ ("My credit card was declined but I still got charged.", "Billing"), ("The API returns a 500 error when I send large payloads.", "Technical"), ("I forgot my password and the reset email never arrives.", "Account"), ("What are your office hours?", "General"), ("Can I downgrade from Premium to Pro mid-cycle?", "Billing"), ] correct = 0 for text, expected in test_set: prediction = classify_ticket(text) predicted_label = prediction.get("category", "Unknown") status = "PASS" if predicted_label == expected else "FAIL" if status == "PASS": correct += 1 print(f"{status}: expected {expected}, got {predicted_label} (confidence: {prediction.get('confidence')})") print(f"\nAccuracy: {correct}/{len(test_set)} ({correct/len(test_set):.0%})") Step 5: Add confidence thresholds and batch processing In production, low-confidence predictions should be flagged for human review. I also process tickets in a simple loop, which stays fast because Oxlo.ai serves popular models with no cold starts. def classify_with_fallback(text: str, threshold: float = 0.85) -> dict: result = classify_ticket(text) confidence = float(result.get("confidence", 0.0)) if confidence list[dict]: results = [] for ticket in tickets: try: results.append(classify_with_fallback(ticket)) except Exception as e: results.append({"category": "Error", "error": str(e)}) return results long_tickets = [ "I noticed something strange on my invoice. " * 20 + "Can you help?", "The webhook keeps timing out after 30 seconds. " * 20 + "Is there a fix?", ] batch_results = batch_classify(long_tickets) for r in batch_results: print(r["category"], "-", r.get("confidence") or r.get("error")) Run it Save the finished script as classifier.py, export your OXLO_API_KEY, and run it against a new ticket. Here is the end-to-end call: import os import json from openai import OpenAI client = OpenAI( base_url="https://api.oxlo.ai/v1", api_key=os.environ["OXLO_API_KEY"] ) SYSTEM_PROMPT = """You are a support ticket classifier. Analyze the user's message and return a JSON object with exactly these keys: - category: one of ["Billing", "Technical", "Account", "General"] - confidence: a float between 0.0 and 1.0 - reasoning: one sentence explaining why you chose this category Rules: - Return only the JSON object, with no markdown formatting. - If the ticket mentions payment, invoices, or charges, use Billing. - If it mentions bugs, errors, or integrations, use Technical. - If it mentions login, password, or user settings, use Account. - Use General only if none of the above apply.""" def classify_ticket(text: str) -> dict: response = client.chat.completions.create( model="llama-3.3-70b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Classify this support ticket:\n\n{text}"}, ], response_format={"type": "json_object"}, temperature=0.1, max_tokens=256 ) return json.loads(response.choices[0].message.content) if __name__ == "__main__": ticket = "My SSO integration stopped working after the latest update. Users cannot log in through Okta." out = classify_ticket(ticket) print(json.dumps(out, indent=2)) Example output: { "category": "Technical", "confidence": 0.94, "reasoning": "The ticket describes an SSO integration failure and login issue caused by a software update, which falls under technical problems." } Next steps If you need hierarchical or multi-label classification, swap in Qwen 3 32B or DeepSeek V3.2 from Oxlo.ai's model catalog. The flat per-request pricing becomes a major advantage here, because long tickets or threaded conversations do not inflate your bill the way token-based providers do. See https://oxlo.ai/pricing for details. A concrete next step is to pipe every low-confidence prediction into a review queue, then feed those human-verified labels back into your system prompt as few-shot examples. This closed loop will tighten accuracy without touching any model weights.