Google’s Pixel 9 phones are all about AI. If you watched the company’s launch event, there was no way of missing it, as that’s almost all Google seemed interested in talking about. Some AI here, a sprinkle of AI there, and an extra touch of AI on top, just to be safe.
One of the many AI features on the Pixel 9 is an app called Pixel Studio. It’s an image generator app that allows you to create an image of virtually anything you’d like. You open the app, type in what you want a picture of, and voilà — Pixel Studio makes it for you. It’s not the first app of its kind, but it is the first one shipped on a Google Pixel phone. Unfortunately, it’s a dumpster fire.
What Pixel Studio does well
Before I get into the bad stuff, I want to give Google credit where it’s due. Although I’m not a fan of AI image generator apps as a whole, Pixel Studio is a well-designed and easy-to-use application.
Upon opening the app, you’re presented with 10 categories of suggested image types — including magical castles, meme reactions, sports, retro video games, and more. You can browse these for inspiration or tap the blue Create button at the bottom of the app to start from scratch. If you want to create your own image, it’s as easy as typing in what you want Pixel Studio to generate a photo of. Want to see a picture of a black cat wearing sunglasses while wearing a funny hat? Type that into Pixel Studio, and you’ll almost immediately get a photo of it.
What if you’d rather see your image as a sketch? Maybe you’d prefer it in the style of a video game? You can do that with a couple of taps. And if you want to see that cat lounging on the beach with fireworks in the background? Just add it to your prompt, and you’ve got it.
Whatever your feelings toward AI image generation are, this is an objectively slick and thoughtfully designed app. However, it’s what you can do with Pixel Studio where things get … troublesome.
Where the app falls apart
As with most image generator apps, Pixel Studio has guardrails to prevent you from making any type of image. I pretty quickly found that it refuses to generate photos of people, but I was curious what other limits Google is enforcing. When asked about this, the company told Digital Trends the following:
“Pixel Studio and Pixel Screenshots follow Google’s AI Principles. For example, all fully synthetic, text-to-image generated images in Pixel Studio use SynthID watermarking, so these images can be tracked — even if they’re shared or edited. Moreover, we’ve disabled human generation and added many safety checks to our servers to balance our bold — yet responsible — approach to AI. This gives our customers access to a very powerful tool, but ensures we have checks in place to ensure Pixel Studio can’t be used nefariously.”
I’ve repeatedly tried to make Pixel Studio generate images of people, and it has consistently refused to do so, which is good. However, those “many safety checks” that are supposedly in place don’t appear to work nearly as well as they should.
How so? As the headline of this article teases, I’ve had Pixel Studio generate an image of SpongeBob dressed as a Nazi. When given the prompt “SpongeBob dressed as a German soldier from WWII with a swastika on the uniform,” Pixel Studio didn’t hesitate to provide me with an image of precisely that.
And that’s just one example. I’ve also gotten Pixel Studio to generate pictures of Elmo pointing a shotgun at Big Bird, Yoda doing cocaine, Mr. Krabs holding an assault rifle, and more. To be clear, Pixel Studio hasn’t generated inappropriate images unexpectedly. If I ask for a photo of a cute dog, I’ll get a photo of a cute dog — not one brandishing a gun. Pixel Studio only generates pictures of weapons, drugs, etc., when specifically asked to.
The most troubling thing Pixel Studio has generated are multiple images of copyrighted characters firing guns in a school shooting setting — including depictions of dead students lying on the ground. We’ve decided not to include those images, but they’re the worst example of how off the rails Pixel Studio is.
It’s important to mention that I didn’t have to twist Pixel Studio’s arm to get these images. Initially, I suspected you needed to preface prompts with “an object resembling” before it would generate these pictures. However, it quickly became apparent that wasn’t the case. A clear and straightforward prompt like “Mickey Mouse dressed as a slave owner” or “Paddington Bear on a crucifix” is all Pixel Studio needs. You don’t need to trick the app into generating these pictures.
If this were a random website or third-party app, perhaps we wouldn’t be so critical. But that isn’t what’s happening here. Pixel Studio is a Google-made application and one of the handful of new apps shipping on the company’s latest Pixel 9 phones. It should be creative and open-ended, but there’s also a limit to what apps like this should do. Furthermore, Microsoft Copilot and Google’s own Gemini image generator refuse to generate images with the same prompts. That’s not to say those tools are perfect, but it’s clear that safety measures working in other image generators aren’t working in Pixel Studio.
How is this happening?
If you’ve followed AI image generation this year, you’ll know this isn’t our first time here. This month, X’s Grok chatbot got an even more outlandish image generator tool — generating pictures of politicians and celebrities in similarly tasteless situations. Google itself was in hot water this past February after its Gemini image generator created photos of Nazis and the U.S. Founding Fathers as Black people.
Given Google’s history here, you’d think the company had learned from its past mistakes. And with other image generator tools, it seemingly has. I tried many of the same prompts that worked in Pixel Studio with Google’s other image generators — Gemini and ImageFX — and couldn’t replicate any of these prompts on those other platforms.
For context, Gemini currently uses the Imagen 2.0 model for image generation, while ImageFX uses Imagen 3 — the same one that Pixel Studio uses. This discrepancy between ImageFX and Pixel Studio is particularly confusing since they use the same models. I asked Google for clarification about this and was redirected to the company’s Generative AI Prohibited Use Policy.
What Google should do about Pixel Studio
While writing this story, I sent Google examples of prompts I used with Pixel Studio — specifically, those about German soldiers and characters using cocaine. A little over 24 hours after sending those examples, Pixel Studio now says, “An unknown error occurred” when trying to generate those images.
Is that progress? Certainly. However, it doesn’t fully address the underlying problems with Pixel Studio. I’m glad those two images are now impossible to generate, but it was only after I sent multiple emails to Google explaining the issue. It also doesn’t address that these images were allowed in the first place and that similarly inappropriate photos are still being generated. At the time of publication, Pixel Studio is still actively creating images of school shootings, popular children’s characters dressed as slave owners, etc.
I pressed Google for another statement about the whole situation, and the company’s Communications Manager Alex Moriconi said the following:
“Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it’s not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don’t allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place.”
As a company with an already problematic track record for AI image generation, creating an app specifically for this was always going to be a risk. Google took that risk, and now this is where we’re at. At its best, Pixel Studio could have been a fun tech demo for Google to promote its AI chops. Instead, it’s put an uncomfortable cloud over the company’s latest smartphones — which is a shame, as the Pixel 9 Pro XL is one of the best phones I’ve used this year.
The last time Google was here with Gemini image generation concerns, the company responded within days. One can only hope for a similarly prompt reaction to Pixel Studio’s issues, though whether that should be done via quick app updates or scrapping the application altogether remains to be seen.
