Publish UIT-VSFC-sentiment representative seed 22

README.md

@@ -0,0 +1,111 @@
+---
+language:
+- vi
+library_name: transformers
+pipeline_tag: text-classification
+base_model: "vinai/phobert-base"
+datasets:
+- uitnlp/vietnamese_students_feedback
+tags:
+- vietnamese
+- text-classification
+- uit-vsfc-sentiment
+- transfer
+- eacl-2027
+- uit-vsfc
+- sentiment-analysis
+metrics:
+- f1
+- accuracy
+---
+
+# phobert-base-UITVSFC-S
+
+This model is `vinai/phobert-base` fine-tuned for **UITVSFC-S** on UIT-VSFC.
+
+## Evaluation protocol
+
+- Dataset size: 16,175 examples.
+- Shared fixed splits for S and T: 12,940 train / 1,617 development / 1,618 test.
+- Split seed: 42; jointly stratified by sentiment–topic labels with exact duplicate groups kept in one split.
+- Fine-tuning seeds: [42, 22, 202].
+- Training: 3 epoch(s), AdamW, learning rate 2e-05, weight decay 0.01, warmup ratio 0.1.
+- Effective train batch size: 64.
+- Maximum sequence length: 256.
+- Input mode: VnCoreNLP RDRSegmenter 1.2 word-segmented Vietnamese.
+- No class weighting, resampling, external metadata, images, engagement features, or test-time model selection.
+- Checkpoints are selected by development Macro-F1. The representative published checkpoint is seed **22**, selected only by development Macro-F1.
+
+## Results
+
+Test metrics are reported as mean ± sample standard deviation over seeds [42, 22, 202].
+
+| Metric | Mean ± std |
+|---|---:|
+| Test Macro-F1 | 0.8325 ± 0.0073 |
+| Test accuracy | 0.9491 ± 0.0016 |
+| Test macro precision | 0.8903 ± 0.0184 |
+| Test macro recall | 0.8002 ± 0.0053 |
+| Development Macro-F1 | 0.8588 ± 0.0073 |
+
+### Per-seed results
+
+|       seed |   dev_macro_f1 |   test_macro_f1 |   test_accuracy |   micro_batch_size |   gradient_accumulation_steps |
+|-----------:|---------------:|----------------:|----------------:|-------------------:|------------------------------:|
+|  22.000000 |       0.866755 |        0.837907 |        0.949320 |          64.000000 |                      1.000000 |
+|  42.000000 |       0.852248 |        0.835299 |        0.950556 |          64.000000 |                      1.000000 |
+| 202.000000 |       0.857497 |        0.824158 |        0.947466 |          64.000000 |                      1.000000 |
+
+## Label mapping
+
+```json
+{
+  "0": "negative",
+  "1": "neutral",
+  "2": "positive"
+}
+```
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+model_id = "BaoNhan/phobert-base-UITVSFC-S"
+tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
+model = AutoModelForSequenceClassification.from_pretrained(model_id)
+
+# This checkpoint follows PhoBERT-style preprocessing; segment raw Vietnamese with VnCoreNLP first.
+text = "Đây là văn_bản tiếng_Việt đã được chuẩn_hóa ."
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
+with torch.no_grad():
+    probabilities = model(**inputs).logits.softmax(dim=-1)[0]
+predicted_id = int(probabilities.argmax())
+print(model.config.id2label[predicted_id], probabilities.tolist())
+```
+
+## Files
+
+- `aggregate_metrics.json`: aggregate metrics and training manifest.
+- `artifacts/per_seed_results.csv`: one row per fine-tuning seed.
+- `artifacts/seed_*_confusion_matrix.csv`: confusion matrix for each seed.
+- `artifacts/seed_*_classification_report.json`: per-class metrics.
+- `artifacts/seed_*_test_predictions.csv`: IDs, gold/predicted labels and probabilities; raw text is excluded.
+
+## Limitations
+
+UIT-VSFC contains student feedback from a specific educational context. Performance may not transfer to other institutions, domains, informal writing styles, or newly emerging vocabulary. Predictions should not be treated as explanations of student intent.
+
+## Dataset citation
+
+```bibtex
+@InProceedings{8573337,
+  author={Nguyen, Kiet Van and Nguyen, Vu Duc and Nguyen, Phu X. V. and Truong, Tham T. H. and Nguyen, Ngan Luu-Thuy},
+  booktitle={2018 10th International Conference on Knowledge and Systems Engineering (KSE)},
+  title={UIT-VSFC: Vietnamese Students' Feedback Corpus for Sentiment Analysis},
+  year={2018},
+  pages={19--24},
+  doi={10.1109/KSE.2018.8573337}
+}
+```

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<mask>": 64000
+}

aggregate_metrics.json

@@ -0,0 +1,33 @@
+{
+  "test_macro_f1_mean": 0.8324547653903752,
+  "test_macro_f1_std": 0.007302740931579332,
+  "test_macro_f1_text": "0.8325 ± 0.0073",
+  "test_accuracy_mean": 0.9491141326740832,
+  "test_accuracy_std": 0.0015553841028576004,
+  "test_accuracy_text": "0.9491 ± 0.0016",
+  "test_macro_precision_mean": 0.8902978363806081,
+  "test_macro_precision_std": 0.018442065017888327,
+  "test_macro_precision_text": "0.8903 ± 0.0184",
+  "test_macro_recall_mean": 0.800244788120481,
+  "test_macro_recall_std": 0.00529049423535965,
+  "test_macro_recall_text": "0.8002 ± 0.0053",
+  "dev_macro_f1_mean": 0.8588332191333613,
+  "dev_macro_f1_std": 0.007345678325996933,
+  "dev_macro_f1_text": "0.8588 ± 0.0073",
+  "task": "UIT-VSFC-sentiment",
+  "dataset": "UIT-VSFC",
+  "model_key": "phobert",
+  "model_name": "PhoBERT-base",
+  "base_model": "vinai/phobert-base",
+  "seeds": [
+    22,
+    42,
+    202
+  ],
+  "representative_seed": 22,
+  "selection_rule": "maximum development Macro-F1; seed ascending tie-break",
+  "split_seed": 42,
+  "max_length": 256,
+  "epochs": 3,
+  "effective_batch_size": 64
+}

artifacts/per_seed_results.csv

@@ -0,0 +1,4 @@
+seed,dev_macro_f1,test_macro_f1,test_accuracy,test_macro_precision,test_macro_recall,test_weighted_f1,micro_batch_size,gradient_accumulation_steps,effective_batch_size,wall_seconds
+22,0.8667554102383873,0.8379073008379212,0.9493201483312732,0.8920574123829293,0.8062276401283147,0.9463231800277144,64,1,64,42.48523855209351
+42,0.8522475938819106,0.8352991724918549,0.9505562422744128,0.9077970494769788,0.7983226799568929,0.9468792101938832,64,1,64,44.07595777511597
+202,0.8574966532797857,0.8241578228413494,0.9474660074165636,0.871039047281916,0.7961840442762358,0.9442736951416952,64,1,64,43.962039947509766

artifacts/seed_202_classification_report.json

@@ -0,0 +1,33 @@
+{
+  "negative": {
+    "precision": 0.955026455026455,
+    "recall": 0.9691275167785235,
+    "f1-score": 0.9620253164556962,
+    "support": 745.0
+  },
+  "neutral": {
+    "precision": 0.7045454545454546,
+    "recall": 0.4492753623188406,
+    "f1-score": 0.5486725663716814,
+    "support": 69.0
+  },
+  "positive": {
+    "precision": 0.9535452322738386,
+    "recall": 0.9701492537313433,
+    "f1-score": 0.9617755856966708,
+    "support": 804.0
+  },
+  "accuracy": 0.9474660074165636,
+  "macro avg": {
+    "precision": 0.871039047281916,
+    "recall": 0.7961840442762358,
+    "f1-score": 0.8241578228413494,
+    "support": 1618.0
+  },
+  "weighted avg": {
+    "precision": 0.9436085983352976,
+    "recall": 0.9474660074165636,
+    "f1-score": 0.9442736951416952,
+    "support": 1618.0
+  }
+}

artifacts/seed_202_confusion_matrix.csv

@@ -0,0 +1,4 @@
+,pred_0,pred_1,pred_2
+true_0,722,5,18
+true_1,18,31,20
+true_2,16,8,780

artifacts/seed_202_run_metadata.json

@@ -0,0 +1,41 @@
+{
+  "status": "completed",
+  "task": "UIT-VSFC-sentiment",
+  "dataset": "UIT-VSFC",
+  "model_key": "phobert",
+  "model_name": "PhoBERT-base",
+  "model_id": "vinai/phobert-base",
+  "base_revision": "c1e37c5c86f918761049cef6fa216b4779d0d01d",
+  "seed": 202,
+  "split_seed": 42,
+  "smoke_test": false,
+  "epochs": 3,
+  "num_labels": 3,
+  "max_length": 256,
+  "micro_batch_size": 64,
+  "gradient_accumulation_steps": 1,
+  "effective_batch_size": 64,
+  "per_device_eval_batch_size": 128,
+  "bf16": true,
+  "tf32": true,
+  "text_mode_resolved": "segmented",
+  "tokenizer_class": "PhobertTokenizer",
+  "model_class": "RobertaForSequenceClassification",
+  "model_type": "roberta",
+  "best_checkpoint": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-202/trainer_mb64_ga1/checkpoint-609",
+  "best_metric": 0.8574966532797857,
+  "best_model_dir": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-202/best_model",
+  "train_loss": 0.2576239512275984,
+  "wall_seconds": 43.962039947509766,
+  "dev_accuracy": 0.9499072356215214,
+  "dev_macro_precision": 0.896763726795645,
+  "dev_macro_recall": 0.8306641661907728,
+  "dev_macro_f1": 0.8574966532797857,
+  "dev_weighted_f1": 0.9479756999550458,
+  "test_accuracy": 0.9474660074165636,
+  "test_macro_precision": 0.871039047281916,
+  "test_macro_recall": 0.7961840442762358,
+  "test_macro_f1": 0.8241578228413494,
+  "test_weighted_f1": 0.9442736951416952,
+  "completed_at_utc": "2026-07-20T06:23:47.602837+00:00"
+}

artifacts/seed_22_classification_report.json

@@ -0,0 +1,33 @@
+{
+  "negative": {
+    "precision": 0.9562334217506632,
+    "recall": 0.9677852348993289,
+    "f1-score": 0.961974649766511,
+    "support": 745.0
+  },
+  "neutral": {
+    "precision": 0.7674418604651163,
+    "recall": 0.4782608695652174,
+    "f1-score": 0.5892857142857143,
+    "support": 69.0
+  },
+  "positive": {
+    "precision": 0.9524969549330086,
+    "recall": 0.972636815920398,
+    "f1-score": 0.9624615384615385,
+    "support": 804.0
+  },
+  "accuracy": 0.9493201483312732,
+  "macro avg": {
+    "precision": 0.8920574123829293,
+    "recall": 0.8062276401283147,
+    "f1-score": 0.8379073008379212,
+    "support": 1618.0
+  },
+  "weighted avg": {
+    "precision": 0.9463256732648182,
+    "recall": 0.9493201483312732,
+    "f1-score": 0.9463231800277144,
+    "support": 1618.0
+  }
+}

artifacts/seed_22_confusion_matrix.csv

@@ -0,0 +1,4 @@
+,pred_0,pred_1,pred_2
+true_0,721,4,20
+true_1,17,33,19
+true_2,16,6,782

artifacts/seed_22_run_metadata.json

@@ -0,0 +1,41 @@
+{
+  "status": "completed",
+  "task": "UIT-VSFC-sentiment",
+  "dataset": "UIT-VSFC",
+  "model_key": "phobert",
+  "model_name": "PhoBERT-base",
+  "model_id": "vinai/phobert-base",
+  "base_revision": "c1e37c5c86f918761049cef6fa216b4779d0d01d",
+  "seed": 22,
+  "split_seed": 42,
+  "smoke_test": false,
+  "epochs": 3,
+  "num_labels": 3,
+  "max_length": 256,
+  "micro_batch_size": 64,
+  "gradient_accumulation_steps": 1,
+  "effective_batch_size": 64,
+  "per_device_eval_batch_size": 128,
+  "bf16": true,
+  "tf32": true,
+  "text_mode_resolved": "segmented",
+  "tokenizer_class": "PhobertTokenizer",
+  "model_class": "RobertaForSequenceClassification",
+  "model_type": "roberta",
+  "best_checkpoint": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-22/trainer_mb64_ga1/checkpoint-609",
+  "best_metric": 0.8667554102383873,
+  "best_model_dir": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-22/best_model",
+  "train_loss": 0.2567696720117027,
+  "wall_seconds": 42.485238552093506,
+  "dev_accuracy": 0.9505256648113791,
+  "dev_macro_precision": 0.8899813636655742,
+  "dev_macro_recall": 0.8483658902182034,
+  "dev_macro_f1": 0.8667554102383873,
+  "dev_weighted_f1": 0.9493649950132751,
+  "test_accuracy": 0.9493201483312732,
+  "test_macro_precision": 0.8920574123829293,
+  "test_macro_recall": 0.8062276401283147,
+  "test_macro_f1": 0.8379073008379212,
+  "test_weighted_f1": 0.9463231800277144,
+  "completed_at_utc": "2026-07-20T06:22:26.794364+00:00"
+}

artifacts/seed_42_classification_report.json

@@ -0,0 +1,33 @@
+{
+  "negative": {
+    "precision": 0.9538866930171278,
+    "recall": 0.9718120805369127,
+    "f1-score": 0.9627659574468085,
+    "support": 745.0
+  },
+  "neutral": {
+    "precision": 0.8157894736842105,
+    "recall": 0.4492753623188406,
+    "f1-score": 0.5794392523364486,
+    "support": 69.0
+  },
+  "positive": {
+    "precision": 0.953714981729598,
+    "recall": 0.9738805970149254,
+    "f1-score": 0.9636923076923077,
+    "support": 804.0
+  },
+  "accuracy": 0.9505562422744128,
+  "macro avg": {
+    "precision": 0.9077970494769788,
+    "recall": 0.7983226799568929,
+    "f1-score": 0.8352991724918549,
+    "support": 1618.0
+  },
+  "weighted avg": {
+    "precision": 0.9479121787963952,
+    "recall": 0.9505562422744128,
+    "f1-score": 0.9468792101938831,
+    "support": 1618.0
+  }
+}

artifacts/seed_42_confusion_matrix.csv

@@ -0,0 +1,4 @@
+,pred_0,pred_1,pred_2
+true_0,724,2,19
+true_1,19,31,19
+true_2,16,5,783

artifacts/seed_42_run_metadata.json

@@ -0,0 +1,41 @@
+{
+  "status": "completed",
+  "task": "UIT-VSFC-sentiment",
+  "dataset": "UIT-VSFC",
+  "model_key": "phobert",
+  "model_name": "PhoBERT-base",
+  "model_id": "vinai/phobert-base",
+  "base_revision": "c1e37c5c86f918761049cef6fa216b4779d0d01d",
+  "seed": 42,
+  "split_seed": 42,
+  "smoke_test": false,
+  "epochs": 3,
+  "num_labels": 3,
+  "max_length": 256,
+  "micro_batch_size": 64,
+  "gradient_accumulation_steps": 1,
+  "effective_batch_size": 64,
+  "per_device_eval_batch_size": 128,
+  "bf16": true,
+  "tf32": true,
+  "text_mode_resolved": "segmented",
+  "tokenizer_class": "PhobertTokenizer",
+  "model_class": "RobertaForSequenceClassification",
+  "model_type": "roberta",
+  "best_checkpoint": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-42/trainer_mb64_ga1/checkpoint-609",
+  "best_metric": 0.8522475938819106,
+  "best_model_dir": "/content/EACL_2027_UITVSFC/runs_ml256_a100_fast3/uit-vsfc-sentiment/phobert/seed-42/best_model",
+  "train_loss": 0.25729619419241967,
+  "wall_seconds": 44.07595777511597,
+  "dev_accuracy": 0.9474335188620903,
+  "dev_macro_precision": 0.8936157851751888,
+  "dev_macro_recall": 0.8246244423591201,
+  "dev_macro_f1": 0.8522475938819106,
+  "dev_weighted_f1": 0.9453460273897113,
+  "test_accuracy": 0.9505562422744128,
+  "test_macro_precision": 0.9077970494769788,
+  "test_macro_recall": 0.7983226799568929,
+  "test_macro_f1": 0.8352991724918549,
+  "test_weighted_f1": 0.9468792101938831,
+  "completed_at_utc": "2026-07-20T06:21:07.289690+00:00"
+}

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "vinai/phobert-base",
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "negative",
+    "1": "neutral",
+    "2": "positive"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "negative": 0,
+    "neutral": 1,
+    "positive": 2
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 258,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "tokenizer_class": "PhobertTokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 64001
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01b3b3a5b60b5271f0568869c15a0949d0e13b136a17b328f8d2360066bf509d
+size 540026460

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64000": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "PhobertTokenizer",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab8cc583eabff0d43984400a7ba5b31f52cb58d852a084466cf55ade1978519c
+size 5496