TokLens

community

AI & ML interests

None defined yet.

Recent Activity

gsaltintas updated a model about 2 hours ago

toklens/language-compressibility-bpe128k

gsaltintas updated a model 2 days ago

toklens/fineweb_edu_tokenizers

gsaltintas published a model 2 days ago

toklens/fineweb_edu_tokenizers

View all activity

toklens 's models 41

toklens/language-compressibility-bpe128k

Updated about 2 hours ago

toklens/fineweb_edu_tokenizers

Updated 2 days ago

toklens/fineweb_data_optimal_143M_tok_T8

Updated 30 days ago • 9

toklens/fineweb_data_optimal_143M_tok_T6

Updated 30 days ago • 52

toklens/fineweb_data_optimal_143M_tok_T4

Updated 30 days ago • 9

toklens/fineweb_data_optimal_143M_tok_T2

Updated 30 days ago • 6

toklens/fineweb_data_optimal_143M_tok_T1

Updated 30 days ago • 19

toklens/fineweb_data_optimal_143M_tok_T18

Updated 30 days ago • 11

toklens/fineweb_data_optimal_143M_tok_T12

Updated 30 days ago • 6

toklens/fineweb_edu_12gb_tokenizer_no_pretok

toklens/fineweb_edu_12gb_tokenizer_sentencepiece

toklens/fineweb_edu_12gb_tokenizer_split_on_whitespace

toklens/data_optimal_143M_tok_T8_space

toklens/data_optimal_143M_tok_T6_space

toklens/data_optimal_143M_tok_T4_space

toklens/data_optimal_143M_tok_T2_space

toklens/data_optimal_143M_tok_T18_space

toklens/data_optimal_143M_tok_T12_space

toklens/maddition_eng_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1000_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1000_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_5_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_10_000_lambda_seg_1_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_100_000_lambda_seg_1_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_1_000_000_lambda_seg_1_2000_minimal

toklens/maddition_ita_Latn_lambda_voc_1_lambda_tok_100_lambda_seg_1_2000_minimal

Updated May 27 • 1

toklens/maddition_eng_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal

toklens/maddition_tur_Latn_lambda_voc_1_lambda_tok_1_lambda_seg_1_2000_minimal

Updated May 27 • 1

toklens/maddition_eng_Latn_lambda_voc_5_lambda_tok_.5_lambda_seg_1_2000_minimal

toklens/maddition_tur_Latn_lambda_voc_5_lambda_tok_.5_lambda_seg_1_2000_minimal