Manipulation of the User

Category: When Analytics Works

A number of recent high-profile cases have raised the possibility of analytics being used to (illegitimately?) manipulate the thoughts, feelings and emotions of users. For example, one study experimented on Facebook users (without their knowledge or consent) to show that "emotional states can be transferred to others via emotional contagion, leading people to experience the same emotions without their awareness"(Kramer, Guillory & Hancock, 2014). And a study from Mozilla raises the question, "What happens when we can increase the number of people who believe something simply by changing the voice that it is read in?" (Kaye, 2020).

An article from RAND suggests, "Artificial intelligence–driven social bots that are now sending you car or skin care advertisements could start chatting with you—actually, experimenting on you—to test what content will elicit the strongest reactions.... Whoever is first to develop and employ such systems could easily prey on wide swaths of the public for years to come" (Paul and Posard, 2020).

Manipulation of the user can be used for beneficial purposes, as described above. However it becomes ethically problematic when the institution, rather than the user, benefits. As Kleber (2018) writes, "Casual applications like Microsoft's XiaoIce, Google Assistant, or Amazon's Alexa use social and emotional cues for a less altruistic purpose — their aim is to secure users' loyalty by acting like new AI BFFs. Futurist Richard van Hooijdonk quips: "If a marketer can get you to cry, he can get you to buy."

Moreover, Kleber continues, "The discussion around addictive technology is starting to examine the intentions behind voice assistants. What does it mean for users if personal assistants are hooked up to advertisers? In a leaked Facebook memo, for example, the social media company boasted to advertisers that it could detect, and subsequently target, teens' feelings of 'worthlessness' and 'insecurity,' among other emotions (Levin, 2017)".

Examples and Articles

AI can manipulate your emotions now!
"AI has entered a new level, where it has advanced towards emotional manipulation of humans by learning about a situation and controlling human reaction in it." Direct Link

Do you have another example of Manipulation of the User? Suggest it here

Force:yes