Native TP does not shard qwen3_vl_moe experts (weights fully replicated per rank) #48308

Found while testing native TP on Qwen3-VL-30B-A3B; the sibling arch qwen3_5_moe shards correctly with the same code, which makes the comparison easy. Related but distinct from #48204: that draft adds an expert-parallel path; this issue is about the plain tp_size-only path silently replicating the experts.

Reproduction

# torchrun --nproc-per-node 4 repro.py
import torch
from transformers import AutoModelForImageTextToText
from transformers.distributed.configuration_utils import DistributedConfig

model = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-30B-A3B-Instruct", dtype=torch.bfloat16, distributed_config=DistributedConfig(tp_size=4)
)
print(torch.cuda.memory_allocated() / 1e9)  # expected ~15 GB/rank, observed 60.3 GB/rank

Outcome

The model loads without error but each rank holds 60.3 GB: the full 30B-A3B weights. Expert weights are not sharded (for comparison, Qwen/Qwen3.6-35B-A3B, arch qwen3_5_moe, shards to
10.5 GB/rank with the same code). Consequences observed:

• any training attempt OOMs immediately (weights + grads alone exceed 80 GB)
• continuous batching on the TP model crashes in the paged cache with index_copy_(): Destination slice shape: 4 128 ... source slice shape: 1 128: the cache is sized for the config's 4 KV heads while the attention layers are sharded to 1 KV head per rank, so the two disagree (attention appears sharded while the MoE is not).

Root cause: Qwen3VLMoeTextConfig.base_model_tp_plan carries dense-style layers.*.mlp.gate_proj/up_proj/down_proj entries, paths that do not exist in the MoE layers (experts live at layers.*.mlp.experts.*), so no pattern matches the expert weights and they replicate. The sibling qwen3_5_moe plan has the fused-expert entries (experts.gate_up_proj: packed_colwise, experts.down_proj: rowwise, experts: moe_tp_experts), which is why it shards correctly.

Precedent: the sibling arch had the identical bug, reported in #45125 and fixed by #45124 (add _tp_plan to the ForConditionalGeneration wrapper). That fix was never mirrored to qwen3_vl_moe, which is exactly why the comparison above splits the way it does. #41570 (uneven memory on Qwen3-VL-30B-A3B multi-GPU) may be an earlier symptom sighting.

Expected behavior

Either a complete TP plan for qwen3_vl_moe (experts sharded like the other MoE archs), or a load-time error when parts of the model have no TP mapping, instead of silent replication.

System info

transformers main (a353632), torch 2.13.0+cu130, Python 3.12.12, 4-8x NVIDIA H100 80GB, torchrun single node.