Skip to main content

Agentic LLM Inference Tuning Reference for Qwen 3.6 and Gemma 4

This page is a practical reference for agentic LLM inference tuning (temperature, top_p, top_k, penalties, and how they interact in multi-step and tool-heavy workflows).

It sits alongside the broader LLM performance engineering hub and matches best with a clear LLM hosting and serving story—throughput and scheduling still dominate when the model is starved, but unstable sampling burns retries and output tokens before the GPU does.

This page consolidates:

Current Focus:

  • Qwen 3.6 (dense and MoE)
  • Gemma 4 (dense and MoE)

If you run terminal agents such as OpenCode, pair this reference with local LLM behavior in OpenCode so workload-level results and sampler defaults stay aligned.

Goal: Provide a single place to configure models for agent loops, coding, and multi-step reasoning.

TLDR Reference Table: Agentic Defaults

Model Mode Temp Top_p Top_k Presence Penalty
Qwen 3.5 27B thinking general 1.0 0.95 20 0.0
Qwen 3.5 27B coding 0.6 0.95 20 0.0
Qwen 3.5 35B MoE thinking 1.0 0.95 20 1.5
Qwen 3.5 35B MoE coding 0.6 0.95 20 0.0
Gemma 4 31B general 1.0 0.95 64 0.0
Gemma 4 31B coding 1.2 0.95 65 0.0
Gemma 4 26B MoE general 1.0 0.95 64 0.0
Gemma 4 26B MoE coding 1.2 0.95 65 0.0

What "Agentic Inference" Actually Means

Most parameter guides assume chat, single-shot completion, or human interaction. Agentic systems are different; they require multi-step reasoning, tool calling, consistent outputs, and low error propagation.

Core Shift in Priorities

Use Case Priority
Chat Natural language quality
Creative Diversity
Agentic Consistency + reasoning stability

Qwen 3.6 Tuning

Dense vs MoE

The 27B dense and 35B-A3B MoE models differ significantly regarding MTP (Multi-Token Prediction) speculative decoding: MTP is worthwhile for the 27B dense model but collapses the usable context window on 16 GB VRAM for the 35B MoE. See the Qwen 3.6 MTP vs Standard benchmark for measured generation speeds and VRAM overhead.

Qwen is one of the few families where MoE requires different penalties.

Dense (27B)

  • Stable and predictable; no routing complexity.
  • Recommended: presence_penalty = 0.0

MoE (35B-A3B)

  • Expert routing per token creates a risk of repetition loops.
  • Recommended: presence_penalty = 1.5 (general), 0.0 (coding).

Why this matters: MoE models can get stuck reusing the same experts. Presence penalty helps diversify token paths and improve reasoning exploration.

Qwen Agentic Coding Setup

Correct setup:

  • temperature = 0.6
  • top_p = 0.95
  • top_k = 20
  • presence_penalty = 0.0

Why low temperature works: Coding agents need deterministic outputs, repeatable tool calls, and stable formatting. Higher temperature often breaks JSON and introduces hallucinated APIs.

Gemma 4 Tuning

Gemma behaves differently. Because official model cards are often implicit, real tuning comes from Google AI Studio, GGUF defaults, and community benchmarks.

The Counter-Intuitive Finding

Gemma 4 performs better with higher temperature.

Temp Result
0.5 Poor reasoning
1.0 Stable baseline
1.2 to 1.5 Best coding performance

Hypothesis: Training distribution favors exploration, and the model compensates for a lack of explicit chain-of-thought control by expanding the solution search space.

Gemma Agentic Coding Setup

Recommended:

  • temperature = 1.2
  • top_p = 0.95
  • top_k = 65
  • penalties = 0.0

Note: Do not apply the traditional "low temp for code" rule blindly to Gemma.

Thinking Mode and Agent Systems

Both Qwen and Gemma support reasoning modes. Agent loops require intermediate reasoning, error recovery, and multi-step planning.

Practical rule: Always enable thinking mode for coding agents, tool use, and multi-step tasks.

Parameter Strategy by Use Case

  • Coding agents: Prioritize determinism, minimize penalties, and use stable sampling.
  • Reasoning agents: Use moderate temperature to allow exploration while preserving structure.
  • Tool calling: Require strict formatting, low randomness, and consistent token patterns.

Schema and JSON tooling are orthogonal to logits; combine these rules with structured output patterns for Ollama and Qwen3 to reduce validator retries.

Vendor Defaults vs Reality

Vendor defaults are generally safe and generic, but not optimized. Community findings often show better task-specific tuning:

  • Gemma: Official guidance is sparse; community suggests high temp for coding.
  • Qwen: Official sections are inconsistent; community values have converged.

Practical Deployment Notes

Under concurrency, queueing and memory splits interact with retries as much as sampling does. Read how Ollama handles parallel requests to tune OLLAMA_NUM_PARALLEL.

  • Ollama: Works well for both families; verify GPU compatibility.
  • vLLM: Stable for production with support for advanced sampling.
  • llama.cpp: Requires careful sampler ordering; always enable Jinja for modern models.

Key Takeaways

  • There is no universal parameter set.
  • Architecture matters more than model size.
  • Agentic systems require different tuning than chat.
  • Community benchmarks often lead vendor documentation.

Final Opinion: Treat inference tuning as a first-class system design problem, not just a config file. Most outdated guides assuming "low temp for code" do not apply to modern models.

Back to top