Agentic LLM Inference Tuning Reference for Qwen 3.6 and Gemma 4
This page is a practical reference for agentic LLM inference tuning (temperature, top_p, top_k, penalties, and how they interact in multi-step and tool-heavy workflows).
It sits alongside the broader LLM performance engineering hub and matches best with a clear LLM hosting and serving story—throughput and scheduling still dominate when the model is starved, but unstable sampling burns retries and output tokens before the GPU does.
This page consolidates:
- Vendor recommended parameters
- Embedded defaults from GGUF and APIs
- Real-world community findings
- Agentic workflow optimizations
Current Focus:
- Qwen 3.6 (dense and MoE)
- Gemma 4 (dense and MoE)
If you run terminal agents such as OpenCode, pair this reference with local LLM behavior in OpenCode so workload-level results and sampler defaults stay aligned.
Goal: Provide a single place to configure models for agent loops, coding, and multi-step reasoning.
TLDR Reference Table: Agentic Defaults
| Model | Mode | Temp | Top_p | Top_k | Presence Penalty |
|---|---|---|---|---|---|
| Qwen 3.5 27B | thinking general | 1.0 | 0.95 | 20 | 0.0 |
| Qwen 3.5 27B | coding | 0.6 | 0.95 | 20 | 0.0 |
| Qwen 3.5 35B MoE | thinking | 1.0 | 0.95 | 20 | 1.5 |
| Qwen 3.5 35B MoE | coding | 0.6 | 0.95 | 20 | 0.0 |
| Gemma 4 31B | general | 1.0 | 0.95 | 64 | 0.0 |
| Gemma 4 31B | coding | 1.2 | 0.95 | 65 | 0.0 |
| Gemma 4 26B MoE | general | 1.0 | 0.95 | 64 | 0.0 |
| Gemma 4 26B MoE | coding | 1.2 | 0.95 | 65 | 0.0 |
What "Agentic Inference" Actually Means
Most parameter guides assume chat, single-shot completion, or human interaction. Agentic systems are different; they require multi-step reasoning, tool calling, consistent outputs, and low error propagation.
Core Shift in Priorities
| Use Case | Priority |
|---|---|
| Chat | Natural language quality |
| Creative | Diversity |
| Agentic | Consistency + reasoning stability |
Qwen 3.6 Tuning
Dense vs MoE
The 27B dense and 35B-A3B MoE models differ significantly regarding MTP (Multi-Token Prediction) speculative decoding: MTP is worthwhile for the 27B dense model but collapses the usable context window on 16 GB VRAM for the 35B MoE. See the Qwen 3.6 MTP vs Standard benchmark for measured generation speeds and VRAM overhead.
Qwen is one of the few families where MoE requires different penalties.
Dense (27B)
- Stable and predictable; no routing complexity.
- Recommended:
presence_penalty = 0.0
MoE (35B-A3B)
- Expert routing per token creates a risk of repetition loops.
- Recommended:
presence_penalty = 1.5(general),0.0(coding).
Why this matters: MoE models can get stuck reusing the same experts. Presence penalty helps diversify token paths and improve reasoning exploration.
Qwen Agentic Coding Setup
Correct setup:
temperature = 0.6top_p = 0.95top_k = 20presence_penalty = 0.0
Why low temperature works: Coding agents need deterministic outputs, repeatable tool calls, and stable formatting. Higher temperature often breaks JSON and introduces hallucinated APIs.
Gemma 4 Tuning
Gemma behaves differently. Because official model cards are often implicit, real tuning comes from Google AI Studio, GGUF defaults, and community benchmarks.
The Counter-Intuitive Finding
Gemma 4 performs better with higher temperature.
| Temp | Result |
|---|---|
| 0.5 | Poor reasoning |
| 1.0 | Stable baseline |
| 1.2 to 1.5 | Best coding performance |
Hypothesis: Training distribution favors exploration, and the model compensates for a lack of explicit chain-of-thought control by expanding the solution search space.
Gemma Agentic Coding Setup
Recommended:
temperature = 1.2top_p = 0.95top_k = 65penalties = 0.0
Note: Do not apply the traditional "low temp for code" rule blindly to Gemma.
Thinking Mode and Agent Systems
Both Qwen and Gemma support reasoning modes. Agent loops require intermediate reasoning, error recovery, and multi-step planning.
Practical rule: Always enable thinking mode for coding agents, tool use, and multi-step tasks.
Parameter Strategy by Use Case
- Coding agents: Prioritize determinism, minimize penalties, and use stable sampling.
- Reasoning agents: Use moderate temperature to allow exploration while preserving structure.
- Tool calling: Require strict formatting, low randomness, and consistent token patterns.
Schema and JSON tooling are orthogonal to logits; combine these rules with structured output patterns for Ollama and Qwen3 to reduce validator retries.
Vendor Defaults vs Reality
Vendor defaults are generally safe and generic, but not optimized. Community findings often show better task-specific tuning:
- Gemma: Official guidance is sparse; community suggests high temp for coding.
- Qwen: Official sections are inconsistent; community values have converged.
Practical Deployment Notes
Under concurrency, queueing and memory splits interact with retries as much as sampling does. Read how Ollama handles parallel requests to tune OLLAMA_NUM_PARALLEL.
- Ollama: Works well for both families; verify GPU compatibility.
- vLLM: Stable for production with support for advanced sampling.
- llama.cpp: Requires careful sampler ordering; always enable Jinja for modern models.
Key Takeaways
- There is no universal parameter set.
- Architecture matters more than model size.
- Agentic systems require different tuning than chat.
- Community benchmarks often lead vendor documentation.
Final Opinion: Treat inference tuning as a first-class system design problem, not just a config file. Most outdated guides assuming "low temp for code" do not apply to modern models.