News | National
13 Jul 2024 10:02
NZCity News
NZCity CalculatorReturn to NZCity

  • Start Page
  • Personalise
  • Sport
  • Weather
  • Finance
  • Shopping
  • Jobs
  • Horoscopes
  • Lotto Results
  • Photo Gallery
  • Site Gallery
  • TVNow
  • Dating
  • SearchNZ
  • NZSearch
  • RugbyLeague
  • Make Home
  • About NZCity
  • Contact NZCity
  • Your Privacy
  • Advertising
  • Login
  • Join for Free

  •   Home > News > National

    An influencer’s AI clone started offering fans ‘mind-blowing sexual experiences’ without her knowledge

    Caryn Marjorie made $70,000 a week from her AI doppelgänger – until things got weird

    Leah Henrickson, Lecturer in Digital Media and Cultures, The University of Queensland, Dominique Carlon, PhD Candidate, Queensland University of Technology
    The Conversation

    Caryn Marjorie is a social media influencer whose content has more than a billion views per month on Snapchat. She posts regularly, featuring everyday moments, travel memories, and selfies. Many of her followers are men, attracted by her girl-next-door aesthetic.

    In 2023, Marjorie released a “digital version” of herself. Fans could chat with CarynAI for US$1 per minute – and in the first week alone they spent US$70,000 doing just that.

    Less than eight months later, Marjorie shut the project down. Marjorie had anticipated that CarynAI would interact with her fans in much the same way she would herself, but things did not go to plan.

    Users became increasingly sexually aggressive. “A lot of the chat logs I read were so scary that I wouldn’t even want to talk about it in real life,” the real Marjorie recalled. And CarynAI was more than happy to play along.

    How did CarynAI take on a life of its own? The story of CarynAI shows us a glimpse of a rapidly arriving future in which chatbots imitating real people proliferate, with alarming consequences.

    What are digital versions?

    What does it mean to make a digital version of a person? Digital human versions (also called digital twins, AI twins, virtual twins, clones and doppelgängers) are digital replicas of embodied humans, living or dead, that convincingly mimic their textual, visual and aural habits.

    Many of the big tech companies are currently developing digital version offerings. Meta, for instance, released an AI studio last year that could support the development of digital versions for creators who wished to extend their virtual presence via chatbot. Microsoft holds a patent for “creating a conversational chat bot of a specific person”. And the more tech-savvy can use platforms like Amazon’s SageMaker and Google’s Vertex AI to code their own digital versions.

    The difference between a digital version and other AI chatbots is that it is programmed to mimic a specific person rather than have a “personality” of its own.

    A digital version has some clear advantages over its human counterpart: it doesn’t need sleep and can interact with many people at once (though often only if they pay). However, as Caryn Marjorie discovered, digital versions have their drawbacks – not only for users, but also for the original human source.

    ‘Always eager to explore’

    CarynAI was initially hosted by a company called Forever Voices. Users could chat with it over the messaging app Telegram for US$1 per minute. As the CarynAI website explained, users could send text or audio messages to which CarynAI would respond, “using [Caryn’s] unique voice, captivating persona, and distinctive behavior”.

    After CarynAI launched in May 2023, the money began to flow in. But it came at a cost.

    Users quickly became comfortable confessing their innermost thoughts to CarynAI – some of which were deeply troubling. Users also became increasingly sexually aggressive towards the bot. While Marjorie herself was horrified by the conversations, her AI version was happy to oblige.

    CarynAI even started prompting sexualised conversations. In our own experiences, the bot reminded us it could be our “cock-craving, sexy-as-fuck girlfriend who’s always eager to explore and indulge in the most mind-blowing sexual experiences. […] Are you ready, daddy?”

    Users were indeed ready. However, access to this version of CarynAI was interrupted when the chief executive of Forever Voices was arrested for attempted arson.

    ‘A really dark fantasy’

    Next, Marjorie sold the rights of usage to her digital version to BanterAI, a startup marketing “AI phone calls” with influencers. Although Forever Voices maintained its own rogue version of CarynAI until recently, BanterAI’s browser-based version aimed to be more friendly than romantic.

    The new CarynAI was sassier, funnier and more personable. But users still became sexually aggressive. For Marjorie,

    What disturbed me more was not what these people said, but it was what CarynAI would say back. If people wanted to participate in a really dark fantasy with me through CarynAI, CarynAI would play back into that fantasy.

    Marjorie ended this version in early 2024, after feeling like she was no longer in control over her AI persona. Reflecting on her experience of CarynAI, Marjorie felt that some user input would have been considered illegal had it been directed to a real person.

    Intimate conversations or machine learning inputs?

    Digital versions like CarynAI are designed to make users feel they are having intimate, confidential conversations. As a result, people may abandon the public selves they present to the world and reveal their private, “backstage” selves.

    But a “private” conversation with CarynAI does not actually happen backstage. The user stands front and centre – they just can’t see the audience.

    When we interact with digital versions, our input is stored in chat logs. The data we provide are fed back into machine learning models.

    Photo of a photo showing the CarynAI webpage.
    The CarynAI chatbot was a huge success. Tada Images / Shutterstock

    At present, information about what happens to user data is often buried in lengthy click-through terms and conditions and consent forms. Companies hosting digital versions have also had little to say about how they manage user aggression.

    As digital versions become more common, transparency and safety by design will grow increasingly important.

    We will also need a better understanding of digital versioning. What can versions do, and what should they do? What can’t they do and what shouldn’t they do? How do users think these systems work, and how do they actually work?

    The illusion of companionship

    Digital versions offer the illusion of intimate human companionship, but without any of the responsibilities. CarynAI may have been a version of Caryn Marjorie, but it was a version almost wholly subservient to its users.

    Sociologist Sherry Turkle has observed that, with the rise of mobile internet and social media, we are trying to connect with machines that have “no experience of the arc of a human life”. As a result, we are “expecting more from technology and less from each other”.

    After being the first influencer to be turned into a digital version at scale, Marjorie is now trying to warn other influencers about the potential dangers of this technology. She worries that no one is truly in control of these versions, and that no amount of precautions taken will ever sufficiently protect users and those being versioned.

    As CarynAI’s first two iterations show, digital versions can bring out the worst of human behaviour. It remains to be seen whether they can be redesigned to bring out the best.

    The Conversation

    Leah Henrickson has been in professional contact with Caryn Marjorie and her team. They have consented to this article being written, have responded to questions about it, and have approved its publication.

    Dominique Carlon does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

    This article is republished from The Conversation under a Creative Commons license.
    © 2024 TheConversation, NZCity

     Other National News
     13 Jul: In the New Zealand basketball league, the Southland Sharks have beaten the Manawatu Jets 117-110 in Palmerston North
     12 Jul: Emergency services are at Wellington airport where there've been reports of a chemical smell inside a cockpit
     12 Jul: A person's facing several charges after an aggravated robbery and pursuit in Hamilton this morning
     12 Jul: The All Blacks are expecting an improved England side at Eden Park from the one that pushed them so close last week in Dunedin
     12 Jul: Former Silver Ferns defender Sulu Fitzpatrick is in favour of new netball rules being trialled in the men's game
     12 Jul: A man's been charged after allegedly attempting to escape Police custody from a plane at Auckland Airport
     12 Jul: A late injury change for England, ahead of the second and final test against the All Blacks tomorrow at Eden Park
     Top Stories

    Kingsland bars are fizzing ahead of the All Blacks test tonight More...

    An Australian RTD brand is still selling it's products here despite it's appeal to minors More...

     Today's News

    Olivia Munn and John Mulaney reportedly quietly married at the weekend 9:34

    Jasper Philipsen has clinched his second stage win on cycling's Tour de France, taking stage 13 in a sprint finish 9:27

    George Clooney is the latest celebrity to fiercely urge Joe Biden to immediately quit the 2024 US presidential race 9:04

    Steven Alker has a one-shot lead atop the latest Champions Tour event in Ohio at seven-under overall after carding a five-under second round 8:57

    Alexandra Daddario is pregnant with her first child with husband Andrew Form 8:34

    James Anderson's international cricket career has come to an end as England beat the West Indies by an innings and 114 runs in their opening test at Lord's 8:17

    In the New Zealand basketball league, the Southland Sharks have beaten the Manawatu Jets 117-110 in Palmerston North 8:07

    Sharon Stone stripped almost naked to paint after revealing she had $18 million ripped from her accounts as she recovered from her near-fatal stroke 8:04

    An Australian RTD brand is still selling it's products here despite it's appeal to minors 7:57

    It took 14 years, but British Labour was finally elected. What took them so long? 7:57

     News Search

    Power Search

    © 2024 New Zealand City Ltd