What the Corpus Still Holds
Testing keys, survival and result stability in deposited YouTube comment corpora
Abstract
Researchers often share YouTube comment data as comment IDs plus their own labels and expect others to retrieve the comments through the YouTube Data API. Whether this works, and what is lost, has rarely been measured on real deposits.
I tested three public deposits in October 2026, from the oldest to the youngest: 62,414 comments on COVID-19 videos, collected in 2020, almost half of them labelled hostile; 173,012 comments on the U.S. Army's channel, collected in 2023; and 16,652 comments on precision fermentation, collected in 2024. Each step was fixed before it was run. Every deposited figure was first reproduced from the file, a sample of IDs was looked up one by one, then every comment was checked, and the deposited results were recomputed on what remained.
- The keys still work.
- 74.4%, 93.2% and 94.8% of the comments, from oldest to youngest deposit, can still be retrieved.
- The loss comes in blocks: whole videos and whole threads. On YouTube, a reply almost never outlives the comment it answers, so a reply that corrects a claim disappears with the claim.
- The more hostile the corpus, the more its hostile comments are lost: 4.8 percentage points in the oldest and most hostile corpus (5.4 on videos that still exist), slight in the contested one, absent in the low-conflict one.
- Yet every deposited distribution moves by less than one percentage point.
- In one deposit, the video ID of 1,845 comments has been replaced by
#NAME?, the error value a spreadsheet shows when it reads text as a formula.
For distributions, a loss of this size is a nuisance rather than a threat; for analyses of threads or of individual users it may be one. The paper closes with five checks for anyone who shares or reuses YouTube comment data. These are three deposits measured once; the results are offered as a starting point, not a general law.
The series
This is the third of three measurement papers on YouTube comment corpora. What the Corpus Does Not Say asks what a corpus leaves out when it is collected; What the Corpus Can No Longer Find asks whether a deposited corpus can still be traced back to the platform. A repeat measurement of the three deposits is planned for May 2027.
Comments
Refutations, reviews and comments are most welcome: research@woidwide.net. I have worked as carefully as I could; if I have made a mistake or overlooked relevant work, I apologise and would be glad to be told.
AI use: the author designed the study and set all analytic decisions and decision rules. Claude (Anthropic) was used under his direction to draft text and analysis code, which the author then edited. All numbers come from scripts run on the deposited data. The author is responsible for the content.
Citation
Sedlmair, Max (2026). What the Corpus Still Holds: Testing keys, survival and result stability in deposited YouTube comment corpora. Preprint, version 1.0. Zenodo. https://doi.org/10.5281/zenodo.23805575