Index index by Group index by Distribution index by Vendor index by creation date index by Name Mirrors Help Search

rust-tokenizers-devel-0.22.2-1.fc42 RPM for noarch

From Fedora 42 testing updates for ppc64le / Packages / r

Name: rust-tokenizers-devel Distribution: Fedora Project
Version: 0.22.2 Vendor: Fedora Project
Release: 1.fc42 Build date: Fri Dec 5 07:47:10 2025
Group: Unspecified Build host: buildvm-x86-32.rdu3.fedoraproject.org
Size: 888638 Source RPM: rust-tokenizers-0.22.2-1.fc42.src.rpm
Packager: Fedora Project
Url: https://crates.io/crates/tokenizers
Summary: Implementation of today's most used tokenizers in Rust
 
Provides an implementation of today's most used tokenizers, with a focus
on performances and versatility.

This package contains library source intended for building other packages which
use the "tokenizers" crate.

Provides

Requires

License

Apache-2.0

Changelog

* Fri Dec 05 2025 Alexander F. Lent <lx@xanderlent.com> - 0.22.2-1
  - Update to 0.22.2 (rhbz#2419081)
* Tue Oct 21 2025 Alexander F. Lent <lx@xanderlent.com> - 0.22.1-2
  - Allow build against newer monostate
* Sun Sep 28 2025 Alexander F. Lent <lx@xanderlent.com> - 0.22.1-1
  - Initial import of tokenizers 0.22.1 (fixes rhbz#2358553)

Files

/usr/share/cargo/registry/tokenizers-0.22.2
/usr/share/cargo/registry/tokenizers-0.22.2/.cargo-checksum.json
/usr/share/cargo/registry/tokenizers-0.22.2/CHANGELOG.md
/usr/share/cargo/registry/tokenizers-0.22.2/Cargo.toml
/usr/share/cargo/registry/tokenizers-0.22.2/LICENSE
/usr/share/cargo/registry/tokenizers-0.22.2/Makefile
/usr/share/cargo/registry/tokenizers-0.22.2/README.md
/usr/share/cargo/registry/tokenizers-0.22.2/README.tpl
/usr/share/cargo/registry/tokenizers-0.22.2/benches
/usr/share/cargo/registry/tokenizers-0.22.2/benches/added_vocab_deserialize.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/bert_benchmark.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/bpe_benchmark.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/common
/usr/share/cargo/registry/tokenizers-0.22.2/benches/common/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/layout_benchmark.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/llama3_benchmark.rs
/usr/share/cargo/registry/tokenizers-0.22.2/benches/unigram_benchmark.rs
/usr/share/cargo/registry/tokenizers-0.22.2/examples
/usr/share/cargo/registry/tokenizers-0.22.2/examples/encode_batch.rs
/usr/share/cargo/registry/tokenizers-0.22.2/examples/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/bpe.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/byte_fallback.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/ctc.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/fuse.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/sequence.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/strip.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/decoders/wordpiece.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/lib.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe/model.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe/trainer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/bpe/word.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/lattice.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/model.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/trainer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/unigram/trie.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordlevel
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordlevel/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordlevel/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordlevel/trainer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordpiece
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordpiece/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordpiece/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/models/wordpiece/trainer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/bert.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/byte_level.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/precompiled.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/prepend.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/replace.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/strip.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/unicode.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/normalizers/utils.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/bert.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/byte_level.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/delimiter.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/digits.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/fixed_length.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/metaspace.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/punctuation.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/sequence.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/split.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/unicode_scripts
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/unicode_scripts/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/unicode_scripts/scripts.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/pre_tokenizers/whitespace.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors/bert.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors/roberta.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors/sequence.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/processors/template.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/added_vocabulary.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/encoding.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/normalizer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/pattern.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/pre_tokenizer.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/tokenizer/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/cache.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/fancy.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/from_pretrained.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/iter.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/onig.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/padding.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/parallelism.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/progress.rs
/usr/share/cargo/registry/tokenizers-0.22.2/src/utils/truncation.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests
/usr/share/cargo/registry/tokenizers-0.22.2/tests/added_tokens.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/common
/usr/share/cargo/registry/tokenizers-0.22.2/tests/common/mod.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/documentation.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/from_pretrained.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/offsets.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/serialization.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/stream.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/training.rs
/usr/share/cargo/registry/tokenizers-0.22.2/tests/unigram.rs

Generated by rpm2html 1.8.1

Fabrice Bellet, Sun Dec 7 01:26:27 2025