Decoding Digital Discourse

Classification pipeline diagram — **Fig.** Pipeline: preprocessing, data augmentation (MLSMOTE + GPT-3 synthetic data) and fine-tuned classifiers.

0.824

accuracy (Davinci + aug.)

+53%

vs. best prior result

labels, 4 tasks

The task

Within a year of the movement starting, the #MeToo hashtag had been used 19 million times. Simple positive/negative sentiment misses most of what is said. I framed the problem as multi-label classification across four tasks on the public MeTooMA dataset:

Stance: support or opposition
Hate speech: directed or generalized hate vs. neither
Sarcasm
Dialogue acts: allegation, justification, refutation

It's a hard problem. Labelled data is scarce, posts are short and noisy, figurative language and irony are common, and classes are heavily imbalanced. The best team in the BMGC 2020 challenge reached an average AUC of only 0.563.

Handling class imbalance

I compared two augmentation strategies: MLSMOTE (Multilabel Synthetic Minority Oversampling) and synthetic tweets generated with GPT-3 for under-represented labels.

Data imbalance and augmented data per label — **Fig.** MeTooMA label imbalance and augmented data per label.

GPT-3 synthetic tweet generation — **Fig.** Generating synthetic training tweets with GPT-3.

Models

DistilBERT + dropout + a 7-way linear head, trained with BCE-with-logits loss for 4 epochs on an 80/20 split.
GPT-3 Ada and Davinci: seven fine-tuned binary classifiers, one per label, each trained on real-only, augmented and synthetic-only data.

Results

Model	Accuracy	Weighted F1
Best result from literature	0.539	n/a
DistilBERT	0.453	0.418
DistilBERT + MLSMOTE	0.501	0.509
GPT-3 Ada	0.531	0.501
GPT-3 Davinci	0.561	0.520
GPT-3 Davinci + augmented data	0.824	0.816

Effect of augmentation on Davinci

Training data	Acc.	Prec.	Recall	AUROC	F1
MeTooMA only	0.536	0.607	0.653	0.521	0.630
MeTooMA + synthetic	0.824	0.897	0.781	0.605	0.816
Synthetic only	0.968	0.987	0.933	0.975	0.966

The near-perfect synthetic-only scores come with a caveat: GPT-3's synthetic tweets express much stronger, clearer sentiment than real ones. Mixing synthetic and real data improved diversity and class balance while keeping the model grounded in real language.

Word cloud of #MeToo tweets — **Fig.** Most frequent terms in the dataset.

Future work

Next steps include tracking how sentiment changes over time, extending the classifier to other platforms and movements such as #BlackLivesMatter, and using contextual models that pick up irony and implied meaning more reliably.