Does a tool that flags AI-written social media posts still work on real posts, or only on tidy test data?
How accurate is OSM-Det when applied to real social media posts?
This explores how well OSM-Det, a detector built to spot AI-generated text on social media, holds up on real posts in the wild rather than on curated test sets. The corpus doesn't answer that directly, so the useful question becomes what it says about how far to trust any detector's labels on real social media.
This explores how reliably OSM-Det, a detector that flags AI-generated social media text, works on real posts rather than on benchmark data. To be direct: none of the notes retrieved here reports an accuracy figure for OSM-Det on live social media. The closest material is a study of Medium posts sorted into 'predicted as human-written' and 'predicted as AI-generated' Do readers engage less with AI-generated social media posts?. Posts labeled AI drew about half the likes of posts labeled human (69 vs. 128). The word to notice is *predicted*. Those labels come from a detector's guesses, not from confirmed authorship. So the engagement gap is only as trustworthy as the detector behind it, and any error in the labels carries straight into the finding.
The corpus says a good deal about why detector accuracy on real posts is hard to pin down. LinkedIn says it flags generic AI content with 94% accuracy. But it hasn't published its test setup, how it defined its sample, or how often it wrongly flags human writing How often does LinkedIn wrongly flag legitimate posts?. It isn't even clear whether '94%' counts how many flagged posts were really AI or how many AI posts got caught. Those are very different numbers when most posts are human Does LinkedIn's 94% accuracy apply to human posts wrongly limited?. The same problem applies to any research detector: a single accuracy number from a test set tells you little about how often it mislabels real people's writing.
A second lesson is that detectors break when the kind of text they meet changes. Detectors for fake LinkedIn profiles were trained on genuine profiles and hand-made fakes. They let 42–52% of GPT-written fake profiles through, and only recovered (to 1–7%) after being retrained on GPT output Can fake profile detectors catch GPT-generated LinkedIn profiles?. A social media detector built on one generation of models faces the same risk as newer models write differently. Accuracy measured at launch can quietly decay.
The most surprising failure is that detectors can latch onto style instead of the thing they're supposed to measure. Fake news detectors flag truthful AI-written articles as fake and pass human-written disinformation as real. They learned to treat AI's polished style as a sign of deception Why do fake news detectors flag AI-generated truthful content?. Turn that around for AI-text detection on social media: a careful, comprehensive human writer may look 'AI-like', and a casual AI post may slip through. That matters because the comprehensive, confident tone is exactly what the corpus identifies as AI posts' signature on social platforms Why do AI posts get likes without inviting conversation?.
The takeaway: when a study reports how much of social media is AI-generated, or how AI posts perform, the detector is part of the measurement. Before trusting the headline, ask whether the detector's false-positive rate on real human posts was ever checked. For OSM-Det specifically, this collection doesn't yet hold that evidence.
Sources 6 notes
AI-labeled posts on Medium averaged 69.15 likes versus 127.59 for human-labeled posts, with similar gaps in comments across all follower groups. The paper calls this gap relatively small and suggests AI content still appeals to users.
LinkedIn reports 94 percent accuracy on flagging generic content but has not published independently verifiable data, test parameters, or false-positive rates. The effect on legitimate writers therefore remains unmeasured.
The 94% figure is self-reported from unspecified testing without false-positive rates, sample definitions, or human-post comparisons. The accuracy metric's scope—whether it measures precision or recall—is undefined, making it unsuitable for evaluating whether the policy reliably separates generic AI from thoughtful human writing.
Detectors trained on genuine and manual fakes miss GPT-generated profiles at 42–52% false accept rates, but adversarial training on GPT-generated data restores detection to 1–7% false accepts without raising false rejects.
Fake news detectors flag LLM-generated content as fake while misclassifying human-written disinformation as genuine. The bias arises because detectors trained on human deception patterns mistake AI's distinct linguistic style for falsity, not because they evaluate veracity.
Show all 6 sources
AI-generated posts achieve high engagement metrics through comprehensive, confident phrasing but suppress reply dynamics because they lack human authorship and invite no counter-argument. This creates one-sided recognition divorced from the conversational validation that historically legitimized social proof.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media
- Is it Cake or is it AI? A Systematic Review of Human Uncertainty in Distinguishing Generative Artificial Intelligence Content
- AI Now Writes as Many Online Articles as Humans
- Weak Links in LinkedIn: Enhancing Fake Profile Detection in the Age of LLMs
- AI Content Is Everywhere on Social Media, Especially LinkedIn
- Keeping conversations real on LinkedIn
- The Impact of Generative AI on Social Media: An Experimental Study
- LinkedIn AI Content Study: 81% of Long-Form Posts Are Likely AI