UnrestSent200K · A Bangla crisis-sentiment benchmark · 199,399 human-labelled comments · July–August 2024
Evaluating crisis sentiment in Bangladesh's July uprising. A 200K-comment Bangla benchmark in which every comment keeps its parent post and its moment in the event, so models can be tested on context and on time, not only on words.
North South University, Dhaka, Bangladesh · Charles Sturt University, Australia
* equal contribution · samiul.alim01@northsouth.edu
The event
Between 5 July and 31 August 2024, public conversation in Bangladesh moved through early mobilisation, a nationwide internet blackout, the collapse of the government, a political transition and, finally, a flood crisis. The language, the platforms and the people posting stayed largely the same. The context changed almost overnight.
That is the stress test this benchmark is built around. A short comment that reads as praise on one day can be sarcasm the next, and its meaning often depends on the news post it answers. Most sentiment benchmarks assume language is stable, judge each utterance alone, and are built for English. UnrestSent200K is built to break those assumptions on purpose.
Every comment in the dataset is timestamped and assigned to one of five phases that follow the event itself. The phases are used for temporal analysis and split construction; they are never used as labels. The chart shows how the volume of public comment surged after the blackout ended and the regime fell, then shifted again as floods became the dominant concern.
Two questions follow from this structure. Can a model trained during one phase still read sentiment in the next? And can it read a comment at all without the post that provoked it?
Volume and sentiment across the five phases
Hover or focus a phase for counts · 199,399 comments
| Phase | Period | Days | Negative | Neutral | Positive | Total | Per day |
|---|
The gap
SentNoB, SentiGOLD and BnSentMix made real progress, but none divides a single unfolding event into phases, links comments to their parent posts, or supports controlled comparisons across time.
Bangla social-media benchmarks by size
Comments or posts, thousands · sentiment, hate-speech and offensive-language datasets
UnrestSent200K is 2.8× the size of SentiGOLD, the largest previous Bangla sentiment corpus, and the only one anchored to a single crisis.
Benchmarks treat language as stable over time. A crisis moves faster than retraining cycles, and sentiment can pivot around a single event.
Each utterance is judged in isolation. In a comment thread the post is often part of the meaning, not extra information.
Existing Bangla corpora cover reviews and general social text. Politically sensitive crisis discourse, with sarcasm and implicit references, is different.
The dataset
Public posts and comment threads were collected from verified news pages, public-interest groups and high-engagement discussion threads on Facebook and YouTube, using Apify, the Facebook Graph API and the YouTube Data API v3.
Download
One archive with the full labelled corpus, the parent-post headlines, and the code used for the encoder and LLM experiments in the paper.
Released for research use. Comments are de-identified: usernames, profile links and other identifying information were removed before release. Please cite the paper if you use the data.
Inside the archive
Comments shorter than three tokens, mostly non-Bangla text and obvious spam were removed. Near-duplicates within a thread were dropped at a Jaccard threshold of 0.85. URLs and user mentions were replaced with placeholder tokens and all identifying information was stripped. Splits are 70/15/15, stratified jointly on phase and sentiment so every partition keeps the same class and temporal mix.
Stratified split by phase and sentiment
Train 70% · validation 15% · test 15% · counts
| Phase | Period | Train neg | Train neu | Train pos | Val neg | Val neu | Val pos | Test neg | Test neu | Test pos | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| P1 Pre-Escalation | Jul 5–15 | 2,281 | 853 | 1,576 | 489 | 183 | 338 | 488 | 182 | 337 | 6,727 |
| P2 Crisis & Blackout | Jul 16–Aug 4 | 9,180 | 4,700 | 9,698 | 1,967 | 1,007 | 2,078 | 1,968 | 1,008 | 2,078 | 33,684 |
| P3 Post-Blackout | Aug 5–10 | 14,153 | 7,482 | 10,704 | 3,033 | 1,603 | 2,294 | 3,032 | 1,603 | 2,293 | 46,197 |
| P4 Post-Revolution | Aug 11–20 | 23,664 | 10,532 | 14,479 | 5,071 | 2,257 | 3,103 | 5,070 | 2,257 | 3,103 | 69,536 |
| P5 Flood Crisis | Aug 21–31 | 17,104 | 5,682 | 7,493 | 3,665 | 1,218 | 1,606 | 3,665 | 1,217 | 1,605 | 43,255 |
| Overall | 66,382 | 29,249 | 43,950 | 14,225 | 6,268 | 9,419 | 14,223 | 6,267 | 9,416 | 199,399 |
Besides the comment label, each source post carries a coarse framing label, Hope, Despair or Outrage, that describes the dominant frame of the post. These labels were used as annotation context and are released as metadata. The prediction task stays comment-level.
The corpus supports comment-only and post + comment evaluation side by side. In crisis discourse, removing the post can remove the information a correct label depends on, so the dataset lets future work measure how well a model actually uses discourse context.
Annotation · PiLA
Pilot Label Annotation (PiLA) is a fully human protocol. Fourteen native Bangla-speaking undergraduates with first-hand exposure to the events labelled every comment; ten senior validators, drawn from a separate pool, adjudicated and audited. The whole process took about eight months.
Ambiguity classes identified and documented: sarcasm, rhetorical questions, religious expressions, political references, code-switched profanity, emoji sentiment. Guidelines refined through pilot annotation, then frozen.
Annotators trained on gold examples and admitted only at Cohen's κ ≥ 0.65. All 14 qualified, scoring 0.66 to 0.81.
Two annotators label each comment independently, with the parent post as context but never copying its sentiment. Gold-check probes inserted at 2%; drift triggers retraining.
The 21.6% of comments with disagreement go to a senior validator; 3.1% escalate to a three-validator panel. Both labels, confidences and the cited rule are logged.
Two validators not involved in adjudication relabel 2,000 stratified comments, 400 per phase, blind to gold: 94.2% exact agreement, 96.8% within one step.
Result 1 · Context
A paired ablation with everything held fixed except the input: four fine-tuned encoders see either the comment alone or the parent post and comment joined by a separator token. Every architecture gains 7 to 11 accuracy points.
Accuracy with and without the parent post
Random 70/15/15 split · same corpus, optimiser, schedule and hyperparameters
| Model | Acc, comment | F1, comment | Acc, post + comment | F1, post + comment | Gain (pp) |
|---|
The gain is similar across very different architectures, which makes it a property of the task rather than of any single model. Comment-only sentiment classification in a crisis thread systematically measures the wrong thing.
The best encoder result combines Bangla pretraining and context: grounded BanglaBERT at 74.72% accuracy and 0.717 macro-F1, with balanced precision and recall.
Same comment, two readings
Alone, the comment reads as a flat statement. Under the headline it becomes bitter irony about an “independent” country. Gold label: negative.
Result 2 · Time
Encoders fine-tuned on comments from 5 July to 5 August were tested on two later windows that do not overlap the training period: Test-A (6–20 August) and Test-B (21–31 August). Same language, same platforms, same event. Only the stage of the crisis changed.
Every model pays for the gap. BanglaBERT stays the strongest and degrades least, reaching 54.83% on Test-B, still twenty points below its in-distribution result with the same post + comment input. Across the four encoders the loss is 19 to 28 macro-F1 points.
The lexical distribution itself moves: Jensen–Shannon divergence between phases sits at 0.37 to 0.38, fewer than half of the top words are shared between phase pairs, and vocabulary entropy rises from 7.60 to 7.79. The drop is a property of the data, not of one model.
The implication is direct. A random test split can report a model as reliable while it fails on the same crisis a few weeks later. Crisis sentiment needs temporal splits.
Accuracy under temporal shift
Trained Jul 5–Aug 5 · tested on later windows · in-distribution result shown for reference
| Model | In-dist. acc | Test-A acc | Test-A F1 | Test-B acc | Test-B F1 |
|---|
Sentiment shift between phases, Cohen's d
All ten phase pairs · darker = larger absolute effect · hover a cell
Thresholds: negligible < 0.2 · small < 0.5 · medium < 0.8. Seven of ten contrasts are negligible; the two medium effects both touch the post-blackout phase.
If degradation over time reflected gradual drift, adjacent phases should differ by similar amounts. They do not. The matrix is sparse: the strongest changes both involve Post-Blackout, against Crisis & Blackout (d = −0.556) and against Flood Crisis (d = 0.567). The shift is concentrated at the moment internet access returned, the regime collapsed and people reacted to a new political reality, then again as the floods took over.
For crisis monitoring, smooth temporal modelling is the wrong inductive bias. This is a change-point problem, and the benchmark is built to study it.
Result 3 · LLMs
Fifteen open-source and proprietary LLMs were prompted in Bangla with an expert persona, zero-shot and with seven in-domain examples, all on the same post + comment input as the encoders. Two mid-scale models were then LoRA-fine-tuned on the training split.
Prompted LLM leaderboard
Bangla expert prompt · accuracy, macro-F1, MCC and κ in % · best per section marked
| Model | Accuracy | Macro-F1 | MCC | Cohen's κ |
|---|
Greedy decoding at temperature 0; any response that is not exactly one of the three labels counts as an error. Few-shot examples help the strongest models most: κ rises from 47 to 55 for GPT-4o-mini and from 34 to 40 for GPT-4.1-mini, while several sub-3B open models barely move. LoRA rows use r = 64, α = 32, learning rate 3e-4.
Models at or below 1.5B parameters cluster near chance on the chance-corrected metrics. The 3 to 4B range begins to discriminate, and Gemma4-4B reaches κ 46.3 with examples. Even at 8B, LLaMA-3.1 trails the proprietary models in the same accuracy band. Fine-tuned Gemma-4B beats fine-tuned LLaMA-3.1-8B, so backbone and recipe matter more than parameter count here.
The strongest prompted model, GPT-4o-mini, lands only 1.3 points above grounded BanglaBERT and shares its failure modes. Supervised adaptation on in-domain labels remains the most reliable lever, and the 78.4% ceiling says the task is still open.
The same post and comment pair, judged by Gemma-4B zero-shot and by the same backbone after LoRA fine-tuning on the training split.
Sarcasm
Postআমার বাসায় কাজ করা পিয়ন এখন ৪০০ কোটি টাকার মালিক: প্রধানমন্ত্রী“The peon who worked at my house now owns 4 billion taka: Prime Minister.”
Commentআহ কি গর্বের কথা!“Ah, what a matter of pride!”
Surface “pride” inverts under a corruption headline. Zero-shot follows the literal words; fine-tuning catches the intent.
Neutral collapse
Postদেশে ইন্টারনেট সেবায় ধীরগতি, ফেসবুক–মেসেঞ্জারে প্রবেশেও বিঘ্ন“Internet service is slow nationwide; access to Facebook and Messenger is also disrupted.”
Commentএটা হাসিনার কাজ“This is Hasina's doing.”
Read as a factual attribution by the zero-shot model. In context it is an accusation, and the fine-tuned model learns that pragmatic reading.
Surface-cue confusion
Postবন্যায় এখন পর্যন্ত ১৩ জনের মৃত্যু, ক্ষতিগ্রস্ত ৪৫ লাখ মানুষ“13 dead in floods so far; 4.5 million affected.”
Commentআল্লাহ আপনি সবাইকে হেফাজত করেন।“May Allah protect everyone.”
Disaster vocabulary in the post drags the zero-shot label negative. The prayer is solidarity, and the fine-tuned model reads it that way.
Diagnosis
Aggregate metrics say how much a model fails, not where. Topic modelling over the corpus and over the 2,789 comments BanglaBERT misclassified shows the failures are structured, and that the neutral class carries most of them.
Per-class accuracy, grounded BanglaBERT
Test split · percent of gold-labelled comments recovered
How the 2,789 errors break down
More than half of neutral test comments (57.5%) are mislabelled, with a 2.5 : 1 bias toward negative. The model is confident while wrong: mean confidence on errors is 0.977, and in one topic it reaches 1.000 while 85 neutral comments are labelled positive. Probability thresholds cannot catch these errors.
The corpus itself is hegemonic rather than pluralistic. One topic, league, government and quota, absorbs 56.3% of all documents and carries net-negative polarity, so a model that overfits its surface inherits a negativity prior on everything else. And the blackout phase is the most coherent period in the corpus (Cv 0.78 against 0.38 to 0.42 elsewhere): crisis crystallises discourse rather than fragmenting it.
Volatility is topic-specific: six topics are highly volatile, fourteen medium, five stable. Politically charged themes show three times the sentiment volatility of humanitarian ones. Temporal robustness therefore has to be measured topic-conditionally, and recalibration should follow topic volatility profiles rather than aggregate confidence.
Findings
The best prompted model only slightly improves over the strongest post + comment encoder and shares its failure modes on sarcasm, implicit political reference and phase-dependent meaning.
LoRA fine-tuning on in-domain labels beats every prompted setup, and the smaller backbone beats the larger one. The ceiling is still below 80%.
Every encoder gains when the parent post is included. Many comments cannot be reliably interpreted in isolation.
Models trained on earlier phases perform far worse on later ones, though language, platforms and event are unchanged. Random splits overestimate performance in a crisis.
The largest changes cluster around the post-blackout period rather than accumulating gradually. Crisis sentiment is a change-point problem.
Cite & resources
The release includes the de-identified corpus, fixed splits, parent-post context, timestamps, platform and phase metadata, annotation guidelines, prompt templates, preprocessing and evaluation scripts, and the encoder and LoRA configurations.
@misc{alim2026pulse,
title = {Can LLMs Follow the Pulse of a Crisis? Evaluating Crisis Sentiment in Bangladesh's July Uprising},
author = {Alim, Md. Samiul and Tamim, Mahir Shahriar and Khan, Tanvir Ahmed and Khan, Sharjil and Duti, Rafia Ferdous and Ridoy, Shahriyar Zaman and Moni, Mohammad Ali},
year = {2026},
note = {UnrestSent200K: a Bangla crisis-sentiment benchmark. Dataset and code: https://github.com/sami0055/UNRESTSENT200K}
}
All data were collected from publicly accessible Facebook and YouTube content in accordance with platform terms of service. Usernames, profile links and other identifying information were removed during preprocessing. Annotators gave informed consent and the protocol was reviewed by the host institution. LLMs were used as experimental systems and, in a limited way, for language editing and figure assistance; all such material was reviewed by the authors, who retain full responsibility.