OneScience-Group/SpectralGPT

SpectralGPT

Model Introduction

SpectralGPT is a foundation model for spectral remote sensing imagery. It learns cross-band and spatial structures through spatial-spectral 3D patching, masked autoencoding, and progressive pretraining, and can provide representations for tasks such as classification, segmentation, and change detection.

Paper: SpectralGPT: Spectral Remote Sensing Foundation Model
https://arxiv.org/abs/2311.07113

Model Description

SpectralGPT was proposed by a research team from Northwestern Polytechnical University and other institutions. The model is first trained on 96x96 fMoW-Sentinel data and then performs second-stage progressive pretraining on 128x128 BigEarthNet data. It is suitable for multispectral image reconstruction, spectral remote sensing representation learning, and transfer to downstream remote sensing tasks.

Use Cases

Scenario	Description
Progressive pretraining	Sequentially perform 96x96 first-stage and 128x128 second-stage training.
Spatial-spectral reconstruction	Perform masked reconstruction of 12-band spatial-spectral Sentinel-2 patches.
Multispectral land-cover classification	Transfer spatial-spectral representations and fine-tune for land-cover classification on datasets such as EuroSAT and BigEarthNet.
Semantic segmentation and change detection	Adapt the pretrained encoder to pixel-level land-cover segmentation and bi-temporal remote sensing change detection tasks.
Local engineering validation	Use a small amount of synthetic data to check the training, inference, and evaluation workflows.
Multi-GPU training	Launch distributed training with torchrun.

Usage Guide

1. OneCode

Experience intelligent one-click AI4S programming through the OneCode online environment:

Click to Experience Intelligent One-Click AI4S Programming

2. Download and Installation

hf download OneScience-Group/SpectralGPT --local-dir ./SpectralGPT
cd SpectralGPT

Environment Dependencies

Hardware Requirements

• A GPU or DCU is recommended.
• CPU can be used for small-configuration connectivity validation; full training and inference will be slow.
• DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the current cluster, is recommended.

DCU Environment

# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

GPU Environment

# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

Training Data Introduction

By default, a small amount of synthetic data is used to validate the two-stage engineering workflow. The first stage uses fMoW-Sentinel-style samples, and the second stage uses BigEarthNet-style samples.

The synthetic data preserves the official progressive pretraining input specifications of 12 bands, 96x96 in the first stage, and 128x128 in the second stage.

Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not the download format of the original datasets.

stage1:
images: float32 [N,12,96,96]
band_order: string [12]
normalization: string scalar
scale_factors: float32 [N]
stage: string scalar = stage1

stage2:
images: float32 [N,12,128,128]
band_order: string [12]
normalization: string scalar
scale_factors: float32 [N]
stage: string scalar = stage2

fake_data.py automatically writes the protocol and data_source protocol metadata. These fields must be retained when using real data.

python scripts/fake_data.py

Training

python scripts/train.py

For multi-GPU training, use:

torchrun --nproc_per_node=8 scripts/train.py

Training first completes the 96-size first stage, then interpolates the spatial positional encoding and completes the 128-size second stage, saving stage checkpoints, a final checkpoint, and aggregate training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the two-stage data scale, model configuration, and training duration corresponding to the paper.

result/checkpoints/stage1.pth
result/checkpoints/stage2.pth
result/checkpoints/final.pth
result/training/metrics.json

Training Weights

This repository will provide SpectralGPT training weights in the weight/ folder. The weight files will be uploaded soon and are expected to be available in the near future.

Inference

python scripts/inference.py

Inference loads the final second-stage checkpoint, performs masked reconstruction on the 128x128 test data, and saves the results to:

result/output/reconstruction.npz

Evaluation and Visualization

python scripts/result.py

Evaluation reports masked-region MSE, MAE, PSNR, spectral angle, and per-band RMSE, and generates a figure containing the input, visible region, prediction, and composite result. Results on synthetic data are only for engineering workflow validation and do not represent full-paper performance.

result/output/metrics.json
result/output/reconstruction.png

Official OneScience Resources

Platform	OneScience Main Repository	Skills Repository
Gitee	https://gitee.com/onescience-ai/onescience	https://gitee.com/onescience-ai/oneskills
GitHub	https://github.com/onescience-ai/OneScience	https://github.com/onescience-ai/oneskills

Citation and License

This repository is a reproduction of the original SpectralGPT paper.

Use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.