Boeker and Urman's sock-puppet audit of TikTok personalization

What happened

Maximilian Boeker (University of Zurich and Technical University of Munich) and Aleksandra Urman (University of Zurich) posted their audit as a preprint on 28 January 2022, and its arXiv page says it was accepted for the Web Conference 2022. Between late June and mid-August 2021 they ran scripted accounts on the TikTok website and compared each account's For You feed with a twin that only scrolled. They collected data on 30,436 posts across 39 completed scenarios.

The psychological idea

A sock-puppet audit uses programmed accounts to isolate one factor at a time. Each scenario paired an active bot, which followed, liked or watched longer, with a control bot that scrolled for the same time. Differences larger than the noise between identical controls were read as personalization. The authors varied language, location, likes, follows and how long a video was watched, from 25 to 400 percent of its length.

In practice

Every tested factor changed the feed. The authors rank following a creator as the strongest influence, then watching for longer, then liking, and say the gap between the last two is small. Location mattered more than language. Watching posts chosen to fit a persona changed feeds more than watching random posts, and watching 25 percent of videos changed them more than watching 75 percent, which the authors could not fully explain. Identical control accounts already saw feeds in which about two thirds of posts differed.

Why it mattered

It is an early test, run from outside the company, of which user actions change TikTok's feed. The authors say their data do not confirm the Wall Street Journal's finding, because following a creator influenced the feed more than the other factors tested. They argue that explicit actions such as following give users more control than lingering does.

Evidence behind the claims

• The authors ran a sock-puppet audit with a custom bot on the TikTok website in incognito mode, pairing an active user with a control user, with about 20 runs per scenario, between late June and mid-August 2021, collecting data on 30,436 posts across 39 successfully completed scenarios.

  Experimental findingChecked at full-text levelWhere: Sections 3.1, 3.2 and 4[1]

• In 11 control scenarios, where neither of two identical users interacted with the feed, the feeds still differed: roughly 66 to 70 percent of posts differed, depending on whether three or five batches were collected.

  Experimental findingChecked at full-text levelWhere: Section 4.1, Controlling Against Noise[1]

• The authors report that all tested factors (language, location, likes, follows, video view rate) influenced recommendations, with following the strongest, then video view rate, then likes, and location a stronger factor than language; the arXiv abstract orders likes before view rate.

  Experimental findingChecked at full-text levelWhere: Sections 4.2 and 4.6 and 6; arXiv abstract page[1][2]

• Video view rate was varied between 25 and 400 percent of video length; watching posts matching a set persona changed feeds more than watching random posts, and feeds of the user who watched only 25 percent of posts differed more than those of the user who watched 75 percent (average difference 0.85 percent against 0.56 percent), which the authors explain only tentatively.

  Experimental findingChecked at full-text levelWhere: Section 4.5, Video View Rate[1]

• The authors write that they cannot confirm the Wall Street Journal's findings because their data show following specific creators influences the feed more than the other tested factors.

  Experimental findingChecked at full-text levelWhere: Section 4.6, Concluding Results[1]

• The authors state their analysis is not exhaustive and is a first step, that findings may be accurate only for a small time window because TikTok changes its algorithm, that they used only a few agents on the web version, and that it includes only the most explicit factors, not commenting or sharing.

  Experimental findingChecked at full-text levelWhere: Introduction, Section 3.4 and Section 6, Conclusion[1]

• The arXiv page for the preprint shows submission on 28 January 2022 and the note that it was accepted for publication at the Web Conference 2022.

  DocumentedChecked at abstract levelWhere: arXiv abstract page, submission history and comments[2]

People involved

• Maximilian Boeker. Researcher, University of Zurich and Technical University of Munich
• Aleksandra Urman. Researcher, University of Zurich

Course connections

• OpenStax Psychology 2e, section 2.2: Approaches to ResearchMethod. The section discusses this topic.

  Opened the page: it says experiments give a high degree of control over variables, and that correlational data cannot show cause and effect. It does not mention control groups by name.

Course relevance is not evidence for the historical or corporate claims.

Sources and inspection

Inspection statements below preserve the original reading scope. A source marked as not verified supports no claim by itself.

1. [1]
   Boeker, Maximilian, Urman, & Aleksandra. (2022). An empirical investigation of personalization factors on TikTok. arXiv. https://arxiv.org/pdf/2201.12271Boeker, M; Urman, A. arXiv 2201.12271 (v1). 2022. Preprint. Primary.

   Relevant text checked. Reviewer opened the arXiv PDF (v1) and read pages 1 to 9 as page images: abstract, introduction, related work, methods, all experiment sections, discussion and conclusion. The earlier review did not inspect supplementary tables. A Codex follow-up found appendices A to C in the current 12-page v1 PDF; their availability is no longer described as missing.

2. [2]
   Boeker, Maximilian, Urman, & Aleksandra. (2022). An empirical investigation of personalization factors on TikTok. arXiv. https://arxiv.org/abs/2201.12271Boeker, M; Urman, A. arXiv. 2022. Preprint. Primary.

   Abstract or publisher record only. Reviewer read the arXiv abstract page: abstract, submission date and the accepted-at-Web-Conference-2022 comment. The abstract orders the factors follow, like, then video view rate, which differs from the paper body.

Access note: The Web Conference version was not opened. The arXiv v1 PDF currently includes appendices A to C; the earlier review covered the main text, not an independent recalculation from those tables.

Related cases and psychology

• TikTok explains how the For You feed recommends videos
• YouTube radicalisation audits and YouTube's borderline content post
• Bakshy, Messing and Adamic on news exposure on Facebook