Expert substitution (MoNE)

Mixture of Novices and Experts (MoNE) prunes MoE layers by replacing redundant experts with "novices" that return a constant vector. vLLM loads the retained experts into compact weights and adds the novices' output separately, so pruned experts use no MLP weight memory and no MoE backend needs to know about them.

Checkpoint format

The checkpoint keeps the original architecture. config.json lists the substituted experts of each decoder layer:

{
  "architectures": ["DeepseekV2ForCausalLM"],
  "n_routed_experts": 64,
  "approximate_experts": {
    "1": [1, 6, 8, 12],
    "2": [0, 3, 17, 40]
  }
}

For VLMs, approximate_experts may live on the text config. Retained experts keep their logical checkpoint names. Each substituted expert stores one [hidden_size] tensor instead of its MLP:

model.layers.1.mlp.experts.0.gate_proj.weight
model.layers.1.mlp.experts.1.approx_value

Alternatively, compression_config.transform_config.expert_substitution declares the same information with explicitly named constant tensors:

{
  "compression_config": {
    "transform_config": {
      "expert_substitution": {
        "version": 1,
        "router_semantics": {
          "preserve_logical_expert_ids": true,
          "preserve_router_weights": true,
          "renormalize_after_substitution": false
        },
        "targets": {
          "model.layers.1.mlp.experts": {
            "num_logical_experts": 64,
            "weight_layout": "compact_retained_experts",
            "replacements": {
              "1": {
                "format": "constant-v1",
                "tensors": {"value": "model.layers.1.mlp.expert_replacements.1.value"}
              }
            }
          }
        }
      }
    }
  }
}

Only the values shown for version, router_semantics, weight_layout, and format are supported. Each target path must contain exactly one layer index. This form is not yet accepted alongside compressed-tensors quantization, whose transform_config schema does not include expert_substitution.

Semantics

For a substituted expert j with constant v_j and router weight w_j(x), the expert contributes w_j(x) v_j. Routing is unchanged: router weights are neither renormalized nor redistributed to retained experts.

Execution

FusedMoEFactory matches each MoE layer to approximate_experts by the layer index in its prefix and, for a match, builds SubstitutedRoutedExperts, which:

1. allocates weights only for retained experts, in ascending logical order;
2. after routing, sums w_j(x) v_j over substituted routes in FP32;
3. rewrites substituted routes as zero-weight routes to physical expert 0 and retained routes as compact physical IDs, preserving the [num_tokens, top_k] contract of every decomposed backend;
4. adds the constant output on one tensor-parallel rank, before the partial outputs are all-reduced.

Zero-weight routes may schedule some unnecessary GEMM work; the memory savings are unaffected. Monolithic backends route internally and are not selected.

Weight loading

Retained experts are loaded through the standard expert mapping by logical ID and placed in their compact row. Substituted IDs map to no local row, the same way expert parallelism skips non-local experts.

Constant tensors are consumed at the model's load_weights boundary, so initial loading and direct weight updates share one path. approx_value tensors of layers that the model does not build (other pipeline stages, MTP layers) are passed through to the model's loader, which skips them like any other weight it does not own. Explicitly named constants are always consumed, and dropped when their layer is built elsewhere. Loading fails if a local substituted expert has no constant.

Direct load_weights updates may change individual constants. Layerwise reloads (reload_weights in checkpoint format) are rejected, and the model keeps its previous weights.

Supported configurations

• Models whose MoE layers use FusedMoEFactory with the default RoutedExperts, and whose MoE prefix contains exactly one layer index.
• Unquantized experts.
• Tensor and pipeline parallelism, including MTP draft models.
• Any decomposed MoE backend.

Expert and data parallelism, EPLB, MoE LoRA, fused shared experts, quantized experts, and routed input/output transforms are rejected at initialization. Deferred MoE finalize and layerwise weight reloads are rejected when used.