Too good to be true? It probably is (AI)

A well-known TV presenter, Liz Bonnin, has been caught up in an elaborate AI scam when her likeness was used to deceive an insect-repellent company (Incognito) into thinking that she had agreed to be featured in ads for their products. Liz only realised that something was wrong after her management team saw the presenter in the online ads for the insect-repellent spray - something she had not agreed to.

The CEO of Incognito, Howard Carter, had previously sought Liz's endorsement before being contacted via a Facebook profile, purporting to be Liz Bonnin. Mr Carter and ‘Bonnin’ exchanged some messages before he was sent an email address and phone number. While Mr Carter initially thought the profile and approach were a “bit suspect”, he was convinced he was dealing with the real Ms Bonnin after being sent a series of voice notes from ‘her’. The deal was then hashed out over email and WhatsApp with Incognito eventually paying £20,000 to the fake Bonnin to secure the rights to feature her in the ads.

It turns out that the voice notes Mr Carter had been sent were part of an elaborate scam and were AI-generated to sound like Ms Bonnin. Experts who analysed the recordings note that there were “…gaps and recitation speed issues that are consistent with AI-generated speech. The dialogue is inconsistent in accent” and the speech quality sounded “unusually clear despite the noise”. The accent of the voice in the recordings also wasn't always consistent with Bonnin's well known Irish voice, with the voice in the recordings sometimes sounding Australian.

The case will serve as a warning to others to not only be wary of AI imagery, but also how AI may be used to generate ‘voices’ by scammers looking to deceive.

There is a growing trend of AI being used to impersonate celebrities or create ‘deepfakes’ (see here for how Stephen Fry's voice was manipulated in this way) which can lead to the use of their ‘likeness’ for certain purposes without their permission. The ASA has recently released guidance on the privacy rules in advertising, which, while they do not deal with AI-generated voice or imagery specifically, should be used as a guide in this area. In particular, those in the public eye still have a right to privacy and advertisers should get permission before referring to such individuals at all.

The government has made moves to tackle certain aspects of deepfakes. The Online Safety Act added a new section 66A to the Sexual Offences Act 2003 to create an offence of sharing a deepfake image or film depicting a person's genitals.  This came into force on 31 January 2024 and the offence is punishable by a fine or imprisonment. The UK government is also planning to criminalise making a sexually explicit deepfake image where the perpetrator wants to cause alarm, humiliation or distress to the victim.  This will be done by amending the Criminal Justice Bill and the offence will be punishable by an unlimited fine. 

These moves cover only a very limited element of the risks posed by deepfakes which raises the wider question as to whether the UK government or the ASA is likely to regulate the use of AI. At the moment the general rules under the CAP code can largely still be applied to the use of AI in advertising, but there is a broader issue around how AI can be used to deceive. In the past the UK government has taken a ‘hands off’ approach to regulation in this area (while the EU AI Act is due to come into force this summer) but it has recently suggested that AI regulation may be on the way after all (read more on that here). While it's likely to be a long way off and with a likely change of government in the meantime, let's see if those rumours develop into more than just that!