The request shape
Model APIs are ordinary HTTP APIs. If you can read the API lesson, you can read these. The request carries a model name, a list of messages with roles, and a few parameters.
curl https://api.example.com/v1/messages \ -H "x-api-key: $MODEL_API_KEY" \ -H "content-type: application/json" \ -d '{ "model": "some-model-name", "max_tokens": 512, "system": "You extract structured data. Reply with JSON only.", "messages": [ { "role": "user", "content": "Extract the date and total from: ..." } ] }'- System instruction, the standing rules for this call, sent every time. There is no server-side memory
- Messages, the conversation, in order. To continue a conversation you resend it
- max_tokens, a cap on the reply length, and therefore on the cost of one call
- The response returns the generated text plus a usage object with input and output token counts
The parameters worth knowing
| Parameter | Effect | Use when |
|---|---|---|
| temperature | Higher is more varied, lower more deterministic | Low for extraction and classification; higher for drafting |
| max_tokens | Caps reply length | Always set it, it's your cost ceiling per call |
| stop sequences | Ends generation at a marker | You're generating into a structured format |
| streaming | Returns tokens as they're produced | A user is waiting and watching |
| tools / function calling | The model requests a structured call you execute | You need it to act, not just to write |
Temperature is the one people over-tune. For anything where a correct answer exists, pulling a date out of a document, classifying a support ticket, keep it low and spend the effort on the instruction instead. Variety is not a virtue when there's a right answer.
Cost, concretely
Billing is per token, in and out, usually at different rates. The mental model that keeps you out of trouble: every call pays for everything you send, every time. A long system instruction resent on each of a hundred thousand calls is a hundred thousand copies of it, billed.
- Measure one real callRead the usage object. Multiply by your expected volume before shipping, not after the first invoice.
- Set max_tokens deliberatelyIt's the only hard cap on a runaway reply.
- Trim what you resendLong conversation histories dominate cost. Summarise old turns rather than resending them verbatim.
- Use prompt caching if the provider offers itA large, unchanging instruction block can often be cached across calls at a reduced rate.
- Try the smaller model firstExtraction and classification frequently work on a cheaper model. Test it rather than assuming the largest is required.
The failures that only appear in production
| Failure | Handling |
|---|---|
| 429 rate limited | Retry with exponential backoff and jitter, never a tight loop |
| Timeouts on long generations | Stream, or raise the client timeout above the default |
| Reply isn't valid JSON | Validate against a schema; retry once with the parse error included |
| Reply truncated mid-sentence | It hit max_tokens, detect the stop reason rather than parsing a fragment |
| Provider outage | A queue and a visible degraded state beat a spinner that never resolves |
| Prompt injection from user content | Treat model output as untrusted input; never execute it or interpolate it into SQL |
Log inputs and outputs from day one, with anything sensitive redacted. When a user reports a bad answer, the exact prompt that produced it is the only useful evidence, and it can't be reconstructed later.
Common mistakes
- Calling the API from frontend code, exposing the key
- Leaving max_tokens unset, so one call can generate for a long time
- Assuming the reply parses as JSON without validating it
- Retrying a 429 immediately in a loop
- Interpolating model output into a query or executing it as code
Key takeaways
- It's an ordinary HTTP API: model, messages, a few parameters, and a usage object
- Keep temperature low wherever a correct answer exists
- Every call pays for everything you send, history and system prompts dominate cost
- Rate limits, truncation, invalid JSON, and outages are the production failures to design for
Try it yourself
Make one API call from a script, print the raw response including the usage object, and multiply the token counts by your expected monthly volume. That number decides most of your design choices.
CHECK YOUR UNDERSTANDING
Pass the quiz below to unlock this.