decoderstack-d24 / tokenizer /token_bytes_legacy.pt

tokenizer: keep the pre-fix token_bytes so published bpb numbers stay reproducible

4aad0ae verified 2 months ago

133 kB

This file is stored with Xet . It is too big to display, but you can still download it.

Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.