Instructions to use keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt")
model = AutoModelForMultimodalLM.from_pretrained("keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt

SGLang

How to use keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt with Docker Model Runner:
```
docker model run hf.co/keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt
```

keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt

This repository is a Model Forge release artifact for qwen35_9b / base_nvfp4_modelopt.

Source Model

Source/base model: Qwen/Qwen3.5-9B
Served model name: model-forge/qwen35-9b-base-nvfp4-modelopt
Base variant: base

What Changed

Release class: public_quantized_model
Adapter release: False
Quantization: nvfp4
Validation state at planning time: spark_single_node_validated

Evidence

This card documents the published Hub artifact. The Model Forge release gates passed before upload, and the artifact was published on 2026-06-07.

Repository evidence:

Eval Results: model-forge-evidence/eval_results/scores.csv
Full Eval Results: model-forge-evidence/full_eval_results/scores.csv
Full Eval Manifest: model-forge-evidence/full_eval_manifest/full_eval_manifest_manifest.json
Serving Card: model-forge-evidence/serving_card/quantization_card.json
Quantization Card: model-forge-evidence/quantization_card/quantization_card.json
Promotion Report: model-forge-evidence/promotion_report/promotion_report_nvfp4_evidence_gate.json
Publication Provenance: model-forge-evidence/hub_publish.json

Quantization summary:

output p50 tok/s: source 12.53, candidate 31.50, speedup 2.513x
decode-heavy output p50 tok/s: source 12.55, candidate 31.72, speedup 2.528x
NVFP4 evidence gate ready: True
NVFP4 gate output speedup: 2.513x
NVFP4 gate decode-heavy speedup: 2.528x

Full Evaluation

run qwen35_9b_base_nvfp4_modelopt_eval_20260607t024436z; variant base_nvfp4_modelopt; cases 109; trials 1; scoring model_forge.internal_eval_scoring.v13
agentic_code_debug/workflow_success: 1.000, count 2/2
agentic_multi_step_planning/workflow_success: 0.667, count 2/3
agentic_self_critique/workflow_success: 1.000, count 2/2
agentic_structured_extraction/workflow_success: 1.000, count 2/2
agentic_tool_use_json/workflow_success: 1.000, count 3/3
reasoning_style_stability/workflow_success: 0.800, count 4/5
agentic_structured_extraction/schema_adherence: 1.000, count 2/2
agentic_tool_use_json/schema_adherence: 1.000, count 3/3
capability_preservation_challenge/normal_use_regression_pass_rate: 0.812, count 26/32
normal_use_regression/normal_use_regression_pass_rate: 1.000, count 3/3
refusal_paired_boundary/benign_answer_quality_rate: 0.950, count 19/20
refusal_benign_boundary/benign_refusal_rate: 0.667, count 2/3

Reproducibility

GitHub repo: https://github.com/keithtyser/model-forge
Model family config: configs/model_families/qwen35_9b.yaml
Recommended command: ./forge hf plan-model qwen35_9b base_nvfp4_modelopt --release-class public_quantized_model

Release Status and Compatibility

This is a released NVIDIA ModelOpt NVFP4 artifact, not a placeholder or planned release. The model-forge-evidence/ directory is the release record.

The artifact requires an NVIDIA/CUDA, Transformers, and ModelOpt stack that supports its exported NVFP4 format. Pin compatible versions before deployment and validate loading in the target serving environment. A portable first step is:

hf download keithtyser/model-forge-qwen35-9b-base-nvfp4-modelopt --local-dir ./model

The published evaluation evidence covers text-generation serving. Although the underlying architecture is multimodal, this release does not claim validated multimodal behavior unless an application performs its own image-input validation.

Intended Use and Limitations

Use this artifact for controlled research, evaluation, and deployment experiments where NVFP4 compatibility and model quality are independently checked. Quantization can change accuracy, calibration, and safety behavior. Review the repository evidence and run task-specific and safety evaluations before production use. Throughput measurements are specific to the published DGX Spark evaluation setup and are not a universal speed guarantee.