The request shape
Model APIs are ordinary HTTP APIs. If you can read the API lesson, you can read these. The request carries a model name, a list of messages with roles, and a few parameters.
curl https://api.example.com/v1/messages \ -H "x-api-key: $MODEL_API_KEY" \ -H "content-type: application/json" \ -d '{ "model": "some-model-name", "max_tokens": 512, "system": "You extract structured data. Reply with JSON only.", "messages": [ { "role": "user", "content": "Extract the date and total from: ..." } ] }'- System instruction — the standing rules for this call, sent every time. There is no server-side memory
- Messages — the conversation, in order. To continue a conversation you resend it
- max_tokens — a cap on the reply length, and therefore on the cost of one call
- The response returns the generated text plus a usage object with input and output token counts
The parameters worth knowing
| Parameter | Effect | Use when |
|---|---|---|
| temperature | Higher is more varied, lower more deterministic | Low for extraction and classification; higher for drafting |
| max_tokens | Caps reply length | Always set it — it's your cost ceiling per call |
| stop sequences | Ends generation at a marker | You're generating into a structured format |
| streaming | Returns tokens as they're produced | A user is waiting and watching |
| tools / function calling | The model requests a structured call you execute | You need it to act, not just to write |
Temperature is the one people over-tune. For anything where a correct answer exists — pulling a date out of a document, classifying a support ticket — keep it low and spend the effort on the instruction instead. Variety is not a virtue when there's a right answer.
Cost, concretely
Billing is per token, in and out, usually at different rates. The mental model that keeps you out of trouble: every call pays for everything you send, every time. A long system instruction resent on each of a hundred thousand calls is a hundred thousand copies of it, billed.
- Measure one real callRead the usage object. Multiply by your expected volume before shipping, not after the first invoice.
- Set max_tokens deliberatelyIt's the only hard cap on a runaway reply.
- Trim what you resendLong conversation histories dominate cost. Summarise old turns rather than resending them verbatim.
- Use prompt caching if the provider offers itA large, unchanging instruction block can often be cached across calls at a reduced rate.
- Try the smaller model firstExtraction and classification frequently work on a cheaper model. Test it rather than assuming the largest is required.
The failures that only appear in production
| Failure | Handling |
|---|---|
| 429 rate limited | Retry with exponential backoff and jitter — never a tight loop |
| Timeouts on long generations | Stream, or raise the client timeout above the default |
| Reply isn't valid JSON | Validate against a schema; retry once with the parse error included |
| Reply truncated mid-sentence | It hit max_tokens — detect the stop reason rather than parsing a fragment |
| Provider outage | A queue and a visible degraded state beat a spinner that never resolves |
| Prompt injection from user content | Treat model output as untrusted input; never execute it or interpolate it into SQL |
Log inputs and outputs from day one, with anything sensitive redacted. When a user reports a bad answer, the exact prompt that produced it is the only useful evidence — and it can't be reconstructed later.
Common mistakes
- Calling the API from frontend code, exposing the key
- Leaving max_tokens unset, so one call can generate for a long time
- Assuming the reply parses as JSON without validating it
- Retrying a 429 immediately in a loop
- Interpolating model output into a query or executing it as code
Key takeaways
- It's an ordinary HTTP API: model, messages, a few parameters, and a usage object
- Keep temperature low wherever a correct answer exists
- Every call pays for everything you send — history and system prompts dominate cost
- Rate limits, truncation, invalid JSON, and outages are the production failures to design for
Try it yourself
Make one API call from a script, print the raw response including the usage object, and multiply the token counts by your expected monthly volume. That number decides most of your design choices.