Meta recently unveiled Muse Image, an artificial intelligence image generator that allows anyone to manipulate pictures of any Instagram user with a public profile. Just as its Meta glasses — the facial recognition-equipped accessory that lets wearers record anyone without their consent — it is an extremely problematic feature because image-based abuse is one of the fastest-growing forms of technology-facilitated gender-based violence.
Everyday non-explicit images can easily be weaponized — as we’ve already seen with Grok — and the harm is spreading faster than any legislation or product safeguard. That gap leaves room not just for one-to-one exploitation but also organized markets that profit from it.
There is another gap: Big Tech policies around what constitutes image-based abuse are largely based on Western ideas of what makes an image “intimate,” usually explicit sexual content and nudity. The assumption that the only images that can be harmful are those that have explicit content, is far from the truth, leaving millions of women and others without protection.
AI can help solve some aspects of content moderation challenges, but it cannot solve all of them.”
Our recent research in Pakistan and among the Pakistani diaspora found that for millions of women, image-based abuse has little to do with the sharing of explicit or nude images. Instead, a photo of a woman who normally covers her hair caught without a headscarf, a video of her dancing at a wedding, or a picture of her standing beside a male classmate can have serious repercussions. For many women in South Asia and the Middle East or its diaspora, these everyday pictures can be weaponized and used to insinuate poor character or illicit affairs.
One of the victims we spoke to had selfies of herself — fully clothed, with her friends, admiring her new eyeliner, or showing off a new haircut — weaponized by her ex-husband in a long campaign of abuse. He shared the manipulated images with her WhatsApp contacts to damage her reputation at her workplace, turn her brothers against her, and jeopardize her relationship with her parents. WhatsApp and local officials said the images did not constitute explicit content, so they could not do anything about it. But the harm was real, even if the policies didn’t recognize it.
Many women in our report spoke of being sexually exploited or blackmailed over similar everyday pictures. In communities where going to the police is taboo, or family disapproval and violence are risks, victims look to Big Tech for accessible and effective processes to take down images and hold perpetrators to account. But Big Tech is failing them.
Tech companies say there is an inherent tension in keeping abuse off platforms and protecting freedom of speech, between believing reports of abuse and fearing malicious reports. These concerns are not without merit, but the balance of risk and harm is not calibrated to the ground reality.
The current state of trust and safety is fraught with contradictions. As countries act against online harms and bring specific legislation on image-based abuse like the Take It Down Act in the U.S. or the Online Safety Act in the U.K., companies are scrambling to ensure compliance. At the same time, investment in trust and safety is falling, and content moderation teams have been downsized, with Meta, Amazon, TikTok, and Alphabet undertaking significant layoffs. The Trump administration has declared that those involved in content moderation and online safety will face more scrutiny and visa denials, further deprioritizing these vital fields of work.
Big Tech needs to address context as much as content.”
This has hastened the shift to using AI to identify, flag, and moderate content, including child sexual abuse material and violent or extremist content. Human rights groups and United Nations representatives have raised concerns of discrimination, censorship, and removal of critical documentation because of AI in content moderation. Companies say AI is faster and more effective, and that it can become more accurate. It also spares human content moderators the trauma of having to review difficult content.
But AI cannot account for consent. For women and other marginalized communities, what matters is not the content of the image, but the lack of consent, the harm experienced, and the intention with which the abuse is perpetrated. Sharing an image in one context does not amount to consent for that image to be forwarded, screenshotted, or recontextualized. A woman posting a profile picture on Instagram isn’t consenting for it to be manipulated by AI and uploaded to Facebook groups and shared on WhatsApp groups. The harm is in the taking, manipulating, sharing, or threatening to share the image without the person’s knowledge or consent, rather than in what it shows.
This is why well-trained local and global human content moderation teams are needed. Companies need to move to a consent-based framework because it works across borders and cultures. It is an investment Big Tech needs to make, and governments need to regulate for. AI can help solve some aspects of content moderation challenges, but it cannot solve all of them.
Meta pulled Muse Image within 72 hours of its launch because of a backlash against its potential for abuse. Everyone should have a right to their own image, and this fundamental premise would help deal more effectively with deepfakes, celebrity impersonation romance scams, and the nonconsensual sharing of content. For content moderation to be meaningful and for users to feel safer — irrespective of age, location, or language — Big Tech needs to address context as much as content. Smart glasses and image generators can wait.
