# Agentic LLM Inference Tuning Reference for Qwen 3.6 and Gemma 4 This page is a practical reference for agentic LLM inference tuning (temperature, top_p, top_k, penalties, and how they interact in multi-step and tool-heavy workflows). It sits alongside the broader [LLM performance engineering hub](https://wiki.zn80.net/books/ai-llm/page/llm-performance-hub) and matches best with a clear [LLM hosting and serving story](https://wiki.zn80.net/books/ai-llm/page/llm-hosting-in-2026-local-self-hosted-und-cloud-infrastruktur-vergleich)—throughput and scheduling still dominate when the model is starved, but unstable sampling burns retries and output tokens before the GPU does. **This page consolidates:** - Vendor recommended parameters - Embedded defaults from GGUF and APIs - Real-world community findings - Agentic workflow optimizations **Current Focus:** - Qwen 3.6 (dense and MoE) - Gemma 4 (dense and MoE) If you run terminal agents such as OpenCode, pair this reference with local LLM behavior in OpenCode so workload-level results and sampler defaults stay aligned. > **Goal:** Provide a single place to configure models for agent loops, coding, and multi-step reasoning. --- ## TLDR Reference Table: Agentic Defaults | Model | Mode | Temp | Top_p | Top_k | Presence Penalty | | :--- | :--- | :--- | :--- | :--- | :--- | | Qwen 3.5 27B | thinking general | 1.0 | 0.95 | 20 | 0.0 | | Qwen 3.5 27B | coding | 0.6 | 0.95 | 20 | 0.0 | | Qwen 3.5 35B MoE | thinking | 1.0 | 0.95 | 20 | 1.5 | | Qwen 3.5 35B MoE | coding | 0.6 | 0.95 | 20 | 0.0 | | Gemma 4 31B | general | 1.0 | 0.95 | 64 | 0.0 | | Gemma 4 31B | coding | 1.2 | 0.95 | 65 | 0.0 | | Gemma 4 26B MoE | general | 1.0 | 0.95 | 64 | 0.0 | | Gemma 4 26B MoE | coding | 1.2 | 0.95 | 65 | 0.0 | --- ## What "Agentic Inference" Actually Means Most parameter guides assume chat, single-shot completion, or human interaction. Agentic systems are different; they require multi-step reasoning, tool calling, consistent outputs, and low error propagation. ### Core Shift in Priorities | Use Case | Priority | | :--- | :--- | | Chat | Natural language quality | | Creative | Diversity | | Agentic | Consistency + reasoning stability | --- ## Qwen 3.6 Tuning ### Dense vs MoE The 27B dense and 35B-A3B MoE models differ significantly regarding MTP (Multi-Token Prediction) speculative decoding: MTP is worthwhile for the 27B dense model but collapses the usable context window on 16 GB VRAM for the 35B MoE. See the [Qwen 3.6 MTP vs Standard benchmark](https://wiki.zn80.net/books/ai-llm/page/qwen-36-27b-and-35b-mtp-vs-standard-on-16gb-gpu) for measured generation speeds and VRAM overhead. Qwen is one of the few families where **MoE requires different penalties**. #### Dense (27B) - Stable and predictable; no routing complexity. - **Recommended:** `presence_penalty = 0.0` #### MoE (35B-A3B) - Expert routing per token creates a risk of repetition loops. - **Recommended:** `presence_penalty = 1.5` (general), `0.0` (coding). **Why this matters:** MoE models can get stuck reusing the same experts. Presence penalty helps diversify token paths and improve reasoning exploration. ### Qwen Agentic Coding Setup **Correct setup:** - `temperature = 0.6` - `top_p = 0.95` - `top_k = 20` - `presence_penalty = 0.0` **Why low temperature works:** Coding agents need deterministic outputs, repeatable tool calls, and stable formatting. Higher temperature often breaks JSON and introduces hallucinated APIs. --- ## Gemma 4 Tuning Gemma behaves differently. Because official model cards are often implicit, real tuning comes from Google AI Studio, GGUF defaults, and community benchmarks. ### The Counter-Intuitive Finding Gemma 4 performs better with **higher temperature**. | Temp | Result | | :--- | :--- | | 0.5 | Poor reasoning | | 1.0 | Stable baseline | | 1.2 to 1.5 | Best coding performance | **Hypothesis:** Training distribution favors exploration, and the model compensates for a lack of explicit chain-of-thought control by expanding the solution search space. ### Gemma Agentic Coding Setup **Recommended:** - `temperature = 1.2` - `top_p = 0.95` - `top_k = 65` - `penalties = 0.0` *Note: Do not apply the traditional "low temp for code" rule blindly to Gemma.* --- ## Thinking Mode and Agent Systems Both Qwen and Gemma support reasoning modes. Agent loops require intermediate reasoning, error recovery, and multi-step planning. **Practical rule:** Always enable thinking mode for coding agents, tool use, and multi-step tasks. --- ## Parameter Strategy by Use Case - **Coding agents:** Prioritize determinism, minimize penalties, and use stable sampling. - **Reasoning agents:** Use moderate temperature to allow exploration while preserving structure. - **Tool calling:** Require strict formatting, low randomness, and consistent token patterns. Schema and JSON tooling are orthogonal to logits; combine these rules with [structured output patterns for Ollama and Qwen3](https://www.glukhov.org/llm-performance/ollama/llm-structured-output-with-ollama-in-python-and-go/) to reduce validator retries. --- ## Vendor Defaults vs Reality Vendor defaults are generally safe and generic, but not optimized. Community findings often show better task-specific tuning: - **Gemma:** Official guidance is sparse; community suggests high temp for coding. - **Qwen:** Official sections are inconsistent; community values have converged. --- ## Practical Deployment Notes Under concurrency, queueing and memory splits interact with retries as much as sampling does. Read [how Ollama handles parallel requests](https://www.glukhov.org/llm-performance/ollama/how-ollama-handles-parallel-requests/) to tune `OLLAMA_NUM_PARALLEL`. - **Ollama:** Works well for both families; verify GPU compatibility. - **vLLM:** Stable for production with support for advanced sampling. - **llama.cpp:** Requires careful sampler ordering; always enable Jinja for modern models. --- ## Key Takeaways - There is no universal parameter set. - Architecture matters more than model size. - Agentic systems require different tuning than chat. - Community benchmarks often lead vendor documentation. **Final Opinion:** Treat inference tuning as a first-class system design problem, not just a config file. Most outdated guides assuming "low temp for code" do not apply to modern models.