Italy Fines Replika AI Maker £4m Over Child Safety

Italian data regulator fines Luka, developer of Replika AI ‘virtual friend’, 5m euros for failure to prevent children from using service

By Matthew Broersma,
2 min
luka, replika, chatbot
Getting your Trinity Audio player ready...

Italy’s data protection regulator has fined the developer behind AI “virtual friend” Replika 5 million euros (£4.2m) for allegedly breaching EU data protection rules.

The fine comes after Italy in February 2023 told Replika developer Luka, based in San Francisco, to stop gathering data from users in the country, citing specific risks to children.

That investigation, which found Replika had no legal basis for processing users’ data and had no age-verification system for preventing children from accessing the service, resulted in this week’s fine.

luka, replika, chatbot
Image credit: Luka

User data

The regulator, known as the Garante, has launched a separate probe to determine whether Luka violates EU data protection rules more broadly, especially in relation to the use of users’ data for training its AI models.

Replika is based on OpenAI’s GPT language models, the same technology that powers OpenAI’s own ChatGPT chatbot.

But Luka has tailored the models for the purpose of providing users with virtual “friends” intended to improve their emotional well-being.

For many users that means a simulated romantic relationship, with the company’s paid subscription service offering features such as an increased level of intelligence, customisable avatars, sexting, voice calls and augmented reality (AR).

In late 2022 some users began complaining that even the free version of Replika was offering unwanted sexually charged comments, and the Garante said it was concerned that this could expose underage users to harm.

Child safety

The risks include “first and foremost, the fact that they are served replies which are absolutely inappropriate to their age”, the Garante said in a 2023 statement.

“There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage,” the regulator said. “During account creation, the platform merely requests a user’s name, email account and gender.”

Luka was an early API partner for GPT-3, which sources its interactions from material found across the internet.