“All raters are native English speakers from India.”
Surge pulled 1,000 comments from Google’s GoEmotions dataset and found serious errors in 308 of them. Raters saw Reddit comments without the subreddit or the thread, so sarcasm and memes went right past them. Researchers trained models on this anyway. Surge sells labeling services, so take the pitch with salt, but the examples speak for themselves.