Instructions to use AlexHung29629/MyModelCleanFreezeRouter with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/MyModelCleanFreezeRouter with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/MyModelCleanFreezeRouter")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/MyModelCleanFreezeRouter")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/MyModelCleanFreezeRouter", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/MyModelCleanFreezeRouter with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/MyModelCleanFreezeRouter"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/MyModelCleanFreezeRouter

SGLang

How to use AlexHung29629/MyModelCleanFreezeRouter with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/MyModelCleanFreezeRouter" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/MyModelCleanFreezeRouter" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/MyModelCleanFreezeRouter",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/MyModelCleanFreezeRouter with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/MyModelCleanFreezeRouter
```

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

See axolotl config

axolotl version: 0.19.0.dev0

# 4 節點 × 8 張 H100，全參數 FSDP2。clean 版資料，凍結 MoE router。
# 從 train-clean.yml 改來。由 train-clean-freeze-router.sbatch 送出。不要改 train-clean.yml。
# 不要在登入節點跑 preprocess 或 train。
# global batch = 32 * micro_batch_size * gradient_accumulation_steps。
# output 跟 train-clean.yml 分開，避免蓋到 gemma4-26b-fft-clean。
# 資料、長度、packing 跟 clean 65536 一樣，preprocess 可沿用那份 cache。
base_model: google/gemma-4-26B-A4B-it

# Gemma 4 權重是 Gemma4ForConditionalGeneration，不要強制 AutoModelForCausalLM。
# 這次是文字 SFT，凍結 vision / audio tower。
freeze_mm_modules: true

# Axolotl 沒有「只凍結這些名字」的鍵。unfrozen_parameters 會先凍結全部，
# 再把對得上的參數設回 requires_grad。點會被當成字面量，萬用字元是 .+，不是 .*。
# 對照 google/gemma-4-26B-A4B-it 的 index：每個 decoder layer 的 router 只有
# proj.weight、scale、per_expert_scale，共 90 個。這三個不在下面的名單裡。
# expert、attention、MLP、layer_scalar、embed、norm、lm_head 照練。
# vision / audio / embed_vision 也不在名單裡，跟 freeze_mm_modules 一樣保持凍結。
unfrozen_parameters:
  - model.language_model.layers.\d+.(self_attn|mlp|experts|input_layernorm|post_attention_layernorm|pre_feedforward_layernorm|post_feedforward_layernorm|post_feedforward_layernorm_1|post_feedforward_layernorm_2|pre_feedforward_layernorm_2).+
  - model.language_model.layers.\d+.layer_scalar$
  - model.language_model.(embed_tokens|norm).+
  - lm_head.+

plugins:
  - axolotl.integrations.liger.LigerPlugin

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
# multimodal Gemma 4 的 liger 沒有 fused linear CE。
liger_fused_linear_cross_entropy: false
# EAFT 自帶 fused lm_head+loss，不能跟 CCE 同時開。
cut_cross_entropy: false

# 容器內建的 EAFT kernel。這四個鍵要一起開。
use_eaft: true
eaft_alpha: 0.333
eaft_k: 20
eaft_normalize: true

# 全參數。不要加 adapter、4bit、quantize_moe_experts。
load_in_8bit: false
load_in_4bit: false

datasets:
  - path: data/ultradata-sft-clean.jsonl
    type: input_output
  - path: data/ultradata-sft-agent-clean.jsonl
    type: input_output
  - path: data/tw-sft-clean.jsonl
    type: input_output

dataset_prepared_path: ./prepared/gemma4-26b-fft-clean-65536
output_dir: ./outputs/gemma4-26b-fft-clean-freeze-router
dataset_num_proc: 64
train_on_inputs: false
val_set_size: 0
seed: 42

# 跟 train-clean.yml 相同：65536、gradient_accumulation_steps 1。
# 超過這個長度的樣本，Axolotl preprocess 會丟掉。
sequence_len: 65536
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

use_tensorboard: true

# 32 張 H100。global batch = 32 * 1 * 1 * 65536 = 2,097,152 tokens。
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 2

optimizer: adamw_torch_fused
learning_rate: 5.0e-6
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
max_grad_norm: 1.0

bf16: auto
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# 183274 打開後，backward 仍是已佔 72.23GiB 再要 5.50GiB。這段 workspace 沒被 offload 移走。
activation_offloading: false

# FSDP2 把參數、梯度、optimizer state 切到 32 張卡。
# 必須包文字 decoder。不指定時 FSDP 會去找不存在的 Gemma4AudioLayer。
# 存檔用 FULL_STATE_DICT，集中成一份權重。不寫 optimizer state，不能從 optimizer 續訓。
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true

# flex attention。全域層 head_dim=512，FA2 單獨跑不了。
# 參數照 MEGA/train.yml。H100 預設 flex 會 SRAM OOM。
attn_implementation: flex_attention
flex_attn_kernel_options:
  num_stages: 1
  bwd_num_stages: 1

logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2
# 只存模型參數。optimizer、scheduler、rng 都不寫進 checkpoint。
save_only_model: true

special_tokens:
  bos_token: "<bos>"
  eos_token: "<eos>"
  pad_token: "<pad>"
  unk_token: "<unk>"

# preprocess --debug 若把要學的 <turn|> 標成 -100，拿掉這段。
eot_tokens:
  - "<turn|>"

outputs/gemma4-26b-fft-clean-freeze-router

This model is a fine-tuned version of google/gemma-4-26B-A4B-it on the data/ultradata-sft-clean.jsonl, the data/ultradata-sft-agent-clean.jsonl and the data/tw-sft-clean.jsonl datasets.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 32
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 21
training_steps: 712

Training results

Framework versions

Transformers 5.14.1
Pytorch 2.11.0+cu130
Datasets 4.8.4
Tokenizers 0.22.2

Downloads last month: -

Safetensors

Model size

26B params

Tensor type

BF16

Model tree for AlexHung29629/MyModelCleanFreezeRouter

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Finetuned

(228)

this model