Configuration

Where a running server takes its values from, and which one wins.

A value comes from one of three places: a flag at start, a field in the request, or the checkpoint itself. A flag beats the checkpoint, a request beats both. Hard caps only go down: a request may lower --max-prompt-tokens, never raise it, and --max-context never exceeds the model’s own window. /health names every default with its source.

Start: beam serve flags

Flags take mlx-lm’s names where mlx-lm has one. beam serve --help prints the same list; the tables are generated from the parser and checked in CI. Hover a shortened description for the whole text.

Server

Flag	Value	Default	What it does
--model	text	—	local path or Hugging Face repo id
--model-alias	text	—	model id shown to clients
--reasoning-field	reasoning / reasoning_content / both / none	reasoning	where a chat completion carries the model’s thinking
--host	text	127.0.0.1	address to listen on
--port	integer ≥ 1	8000	TCP port to listen on
--allowed-origins	ORIGIN (one or more)	—	pages a browser may call the server from and read the answers of
--max-queued	integer ≥ 1	—	requests allowed to wait for a batch slot
--trust-remote-code	switch	—	run a model_file shipped inside the checkpoint
--log-level	DEBUG / INFO / WARNING / ERROR	INFO	how much the server log says

Access

Flag	Value	Default	What it does
--api-key	KEY	—	the key every request must carry
--skip-api-key	switch	—	serve without a key on a non-loopback –host, on purpose
--allowed-hosts	HOST (one or more)	—	Host header values accepted next to localhost and –host, e.g. the machine’s name or LAN address behind a wildcard bind

Chat template

Flag	Value	Default	What it does
--chat-template	text	—	Jinja text, or the path of a .jinja file, used instead of the model’s own template
--use-default-chat-template	switch	—	give a model that ships no chat template a plain ChatML one
--chat-template-args	JSON	—	handed to every template render, e.g. ‘{“enable_thinking”: false}’

Token limits

Every value counts tokens; each one counts a different set.

Flag	Value	Default	What it does
--max-context	integer ≥ 1	—	prompt plus generated tokens, a hard cap
--max-prompt-tokens	integer ≥ 1	—	prompt tokens, a hard cap below the context
--max-completion-tokens	integer ≥ 1	—	generated tokens when the client sends no max_tokens / max_completion_tokens / max_output_tokens
--max-reasoning-tokens	integer ≥ 0	—	reasoning tokens
--min-response-tokens	integer ≥ 0	0	tokens kept for the answer after the think block when the client sends no min_response_tokens

Sampling defaults

Used when the client sends nothing; a flag beats the model’s generation_config.json, which beats mlx-lm’s defaults.

Flag	Value	Default	What it does
--temp	number	—	temperature
--top-p	number	—	nucleus sampling
--top-k	integer ≥ 0	—	top-k sampling
--min-p	number	—	min-p sampling

KV cache

Flag	Value	Default	What it does
--kv-bits	4 / 8	—	quantize the full-attention KV cache to this many bits
--kv-group-size	32 / 64 / 128	—	group size of the KV quantization
--kv-config	text	—	JSON file, bits per layer
--kv-prefill	exact / quantized	—	when a quantized layer becomes quantized

Batching

Flag	Value	Default	What it does
--decode-concurrency	integer ≥ 1	8	sequences decoded in one batch
--prompt-concurrency	integer ≥ 1	2	prompts prefilled in one batch
--prefill-step-size	integer ≥ 1	2048	prompt tokens per model call while prefilling
--prefill-slice	integer ≥ 1	512	prompt tokens a prefill runs before decode gets a turn
--decode-share	SHARE	0.5	share of the worker’s time decode keeps while a prefill runs

Speculative decoding

Off unless asked; a checkpoint that bundles a draft head says so at start.

Flag	Value	Default	What it does
--draft-model	text	—	the proposer that drafts tokens for the verify pass
--exact-verify	off / kernels / positions	off	how the verify runs
--max-draft-tokens	integer ≥ 1	3	cap on the drafts verified per cycle

Prompt cache

Flag	Value	Default	What it does
--prompt-cache-size	integer ≥ 1	16	stored prefixes
--prompt-cache-bytes	integer ≥ 1	—	RAM budget in bytes for the stored prefixes
--prompt-cache-dir	DIR	—	directory the store is written to on a clean stop and read back from after start, one folder per model revision and KV layout
--prompt-cache-save-seconds	SECONDS	8.0	time budget for writing the store on a stop
--prompt-cache-disk-bytes	integer ≥ 1	107374182400	cap on the whole –prompt-cache-dir across model revisions and layouts

--prompt-cache-dir keeps the prompt cache across restarts: on a clean stop the engine writes its store there, one folder per model revision and KV layout with a manifest and one file per entry, and reads it back after the start. The folder grows to the size of the store in RAM (--prompt-cache-bytes) per configuration, and whole folders of other configurations are removed when the directory would exceed --prompt-cache-disk-bytes. The files hold the conversations in another form: the directory is created with mode 0700 and must stay private.

Request: fields a call may send

/v1/chat/completions takes the fields below. /v1/completions takes the sampling, penalty and stop fields plus prompt, echo and an integer logprobs; /v1/responses takes them in its own shape; /v1/messages takes Anthropic’s fields and answers in Anthropic’s blocks, events and errors. cache_control markers are accepted; the prompt cache checkpoints every message boundary on its own and reports hits in usage.cache_read_input_tokens. Roles are system, developer, user, assistant and tool; developer becomes system for a template that does not know it. A field left out falls back to the server’s default.

Field	Values	What it does
max_completion_tokens max_tokens	integer ≥ 1	Generated tokens for this call.
max_prompt_tokens	integer ≥ 1	A prompt cap for this call
max_reasoning_tokens thinking_token_budget reasoning.max_tokens	integer ≥ 0	Reasoning tokens the think block may take
min_response_tokens	integer ≥ 0	Tokens kept for the answer after the think block.
reasoning_effort reasoning.effort	none / off / false, minimal, low, medium, high, xhigh, max / ultra	none turns thinking off
enable_thinking	boolean, or the words true / false	Thinking on or off for this call
chat_template_kwargs	object	Extra variables for the template render, on top of --chat-template-args and the aliases above.
temperature	0-2	Sampling temperature; 0 is greedy.
top_p	0-1	Nucleus sampling.
top_k	integer (0 or -1 = off)	Top-k sampling.
min_p	0-1	Min-p sampling.
min_tokens_to_keep	integer ≥ 1	Tokens min_p may never filter away.
xtc_probability	0-1	Exclude-top-choices sampling
xtc_threshold	0-0.5	The probability above which a token counts as a top choice.
seed	integer	A seeded call samples with its own random key, so it repeats.
repetition_penalty	≥ 0 (1 = off)	Sign-aware multiplicative penalty on tokens seen in the last n.
repetition_context_size	integer ≥ 1 (20)	The n the repetition penalty looks back.
presence_penalty	-2 to 2	Additive penalty on tokens present in the last n.
presence_context_size	integer ≥ 1 (20)	The n the presence penalty looks back.
frequency_penalty	-2 to 2	Additive penalty in proportion to how often a token appeared in the last n.
frequency_context_size	integer ≥ 1 (20)	The n the frequency penalty looks back.
logit_bias	object, token id → -100 to 100	Added to the logits of those ids at every step.
logprobs	boolean	Log-probabilities per token.
top_logprobs	integer 0-20	The most likely alternatives per token; needs logprobs: true.
stop	string or list of strings	Sequences that end the answer
tools	list of function tools	Tools the template renders and the answer is parsed for.
tool_choice	auto or none	none sends no tools
stream	boolean	Server-sent events.
stream_options.include_usage	boolean	With stream, the last event carries usage.

Refused with an error: n above 1, best_of, suffix, response_format other than text, and stored state on /v1/responses (previous_response_id, conversation). Structured output is an extra.

Checkpoint: what the model ships

A standard MLX checkpoint can carry settings of its own:

• generation_config.json - its sampling values become the server’s defaults, below the flags and above mlx-lm’s own; do_sample: false means greedy.
• The chat template - tool-call markers are inferred from the template that renders, think markers from the vocabulary. --chat-template replaces it; --use-default-chat-template gives a model without one plain ChatML.
• A KV profile - bits per layer, as a list or an object, read only through --kv-config. Listed layers take their bits, the rest follow --kv-bits. The prefill mode comes from the profile, else from the mode the package was measured with (the manifest’s file, SHA-256 checked); --kv-prefill beats both.
• A draft head - named by the manifest (parts.mtp, SHA-256 checked), else by mtp_file in config.json, else found in the shards (Qwen’s mtp.* tensors, GLM-4.7 / DeepSeek’s NextN layer). Used only with --draft-model bundled; a checkpoint that bundles one says so when started without it. Its bits come from the manifest, else config.json, else it is quantized to 4 bits at load.
• A package manifest - extras/manifest.json in the B.E.A.M. package layout names the draft head and the KV profile, each with file and SHA-256. A plain checkpoint has none.

/health shows what was built from all of this: the KV layout per layer, batching and cache settings, the draft head and its counters, and every default with its source.