mlboydaisuke commited on
Commit
7ca2923
·
verified ·
1 Parent(s): f6a837a

Update LlmMetadata for litert-lm 0.18.0: chat template with content parts, supports_thinking / supports_function_calling (weights unchanged)

Browse files

This updates the metadata (LlmMetadata) of three files, in place so their URLs stay the same: `minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`, `minicpm_wi4b32_wi8_afp32.litertlm` and `MiniCPM5-1B_dynamic_wi8_afp32.litertlm` (the packer rewrites the container header; the two wi4b32 files come out 5.7 KB and 5.5 KB shorter because the zero padding after the last section is not written again).

One default changes, shown in the table: the two wi4b32 files now answer directly unless `enable_thinking` is true, as the dynamic_wi8 file and LiteRT-LM's minicpm5 template do; if they should keep leaving that choice to the model, I will change it.

| file | thinking by default (before → after) | context tokens |
|---|---|---:|
| `minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm` | the model decides → off | 1024 |
| `minicpm_wi4b32_wi8_afp32.litertlm` | the model decides → off | 1024 |
| `MiniCPM5-1B_dynamic_wi8_afp32.litertlm` | off → off | 4096 |

litert-lm 0.18.0 hands the chat template each message's content as a list of parts ([models/README.md](https://github.com/google-ai-edge/LiteRT-LM/blob/v0.18.0/models/README.md), [issue 3688](https://github.com/google-ai-edge/LiteRT-LM/issues/3688)), so with the published templates it renders the user turn of the wi4b32 files empty and prints the list into the dynamic_wi8 prompt as text.

All three now carry LiteRT-LM's [models/minicpm5/chat_template.jinja at b5e34ab1](https://github.com/google-ai-edge/LiteRT-LM/blob/b5e34ab1/models/minicpm5/chat_template.jinja) with a `format_content` macro that reads content as a string or as parts (`text`, and `tool_response` as on the current main version), and set `supports_thinking` and `supports_function_calling`. The two wi4b32 files also take three values from the dynamic_wi8 file: `llm_model_type minicpm5`, so the runtime parses the model's `<function name=…>` tool calls; the reasoning channel `<think>\n` / `</think>`; and the two stop token ids (1 and 130073). The fourteen stop strings they drop all end in `<|im_end|>\n`, and each encodes to tokens that include `<|im_end|>` (id 130073), which stays a stop id, so generation stops at that token before any of those strings could complete. The TFLite and tokenizer sections are byte-identical to the published files (sha256 checked), and the sampler and context lengths do not change.

On litert-lm 0.18.0 (python API, CPU on a Mac), `describe` reports both capabilities, and each file answered a plain prompt, returned a parsed tool call and answered from its result, answered with thinking on (the reasoning on the reasoning channel) and off, and ran two turns; on 0.17.0 each file answered the plain prompt and the two wi4b32 files ran two turns, while two turns on the dynamic_wi8 file, tool calls there, the Kotlin and Swift APIs and the GPU on the Mac were not run. On a Galaxy S26 with litert_lm_main built from the v0.18.0 tag, `minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm` answers the same prompt on the GPU with every node of its prefill and decode signatures delegated (OpenCL, 1044 of 1044 and 953 of 953); the other two files were not run on a phone.

The card gets one sentence under each file's bullet; if a new name suits the repo better, I will change the PR, and if a new export supersedes these files, I will close it.

MiniCPM5-1B_dynamic_wi8_afp32.litertlm CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c3b948cf44754111c574c9d6970a9311f1c07dc021323cf2d7076f76389313be
3
  size 1105464896
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97e59cffe93017b553060f3bb43b113749ae100693b81ff7e0548733a22a06a7
3
  size 1105464896
README.md CHANGED
@@ -24,9 +24,9 @@ This repository hosts the [**LiteRT-LM**](https://ai.google.dev/edge/litert-lm)
24
 
25
  ## Available Models
26
 
27
- * **`minicpm_wi4b32_wi8_afp32.litertlm`**: This model features mixed INT4-block32(linear)/INT8(embed and lmhead) quantization (wi4b32_wi8) with FP32 activations (afp32).
28
- * **`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`**: This model uses same quantization recipe with minicpm_wi4b32_wi8_afp32 but optimized for GPU execution.
29
- * **`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`**: This model features dynamic weight-only INT8 quantization (wi8) with FP32 activations (afp32). It is configured with static prefill memory allocation.
30
 
31
  ## Quantization Benchmark
32
 
 
24
 
25
  ## Available Models
26
 
27
+ * **`minicpm_wi4b32_wi8_afp32.litertlm`**: This model features mixed INT4-block32(linear)/INT8(embed and lmhead) quantization (wi4b32_wi8) with FP32 activations (afp32). Runs on litert-lm 0.17.0 and 0.18.0 (chat content as a string or as parts); thinking is off by default and `enable_thinking` turns it on; on 0.18.0, tool calls are parsed by the runtime.
28
+ * **`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`**: This model uses same quantization recipe with minicpm_wi4b32_wi8_afp32 but optimized for GPU execution. Runs on litert-lm 0.17.0 and 0.18.0 (chat content as a string or as parts); thinking is off by default and `enable_thinking` turns it on; on 0.18.0, tool calls are parsed by the runtime.
29
+ * **`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`**: This model features dynamic weight-only INT8 quantization (wi8) with FP32 activations (afp32). It is configured with static prefill memory allocation. Runs on litert-lm 0.17.0 and 0.18.0 (chat content as a string or as parts); thinking is off by default and `enable_thinking` turns it on; on 0.18.0, tool calls are parsed by the runtime.
30
 
31
  ## Quantization Benchmark
32
 
minicpm_wi4b32_wi8_afp32.litertlm CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:43f41a837c58408192d703a7cf7bb27a3f8b4f8cc1d34c7f79e54f058f966ccc
3
- size 792625152
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c5950e3f6d4d5a898ad03cb5517ccd8f5fb1cb7616d6df2703bb983ddfb6a329
3
+ size 792619648
minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4e15a7cc735ae36e9888d341d44bd53c0579153d7a3379aaf958620eea4816e3
3
- size 793034752
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22f50149b040aa6504b2bb3bcba70ae0dfbb62e501a277e3042b6e12566a9769
3
+ size 793029040