When Users Aren’t Clued In.
Apple to clarify when AI summarization is at play after a few high profile mistakes
John Gruber (in response to story from Liv McMahon and Natalie Sherman, reporting for BBC News)
All of the following things are true:
All of Apple Intelligence is labelled “beta”.
People generally know what “beta” means.
Apple is promoting the hell out of Apple Intelligence to consumers, and its advertisements hide, rather than emphasize, its “beta” quality.
The promotion of a feature is an implicit encouragement to, you know, actually use it.
Apple Intelligence has to be opted into, and once enabled, can be turned off.
Apple Intelligence notification summaries are marked with an icon/glyph, sort of like the “↪︎” Unicode glyph with a few horizontal lines to suggest text encapsulated by the arrow — a clever icon to convey an abstract concept, to be sure.
The meaning of that icon/glyph is not at all obvious unless you know to look for it, and most users — even those who opted in to Apple Intelligence understanding that it was “beta” and might produce erroneous results — don’t know to look for that particular glyph.
Thus, even to a user well aware of what they opted into, when they get a notification summary from “BBC News” that claims, say, that Luigi Mangione shot himself, it is perfectly reasonable for that user to presume it was the BBC News reporting that Luigi Mangione shot himself.
I’ve been using the Apple Intelligence notification summary feature since it was made available in public release (I don’t use developer or public beta versions of Apple operating systems).
The very first time I saw a summarized notification with that little glyph Apple uses to indicate one as such, I was able to recognize immediately what it meant, that it was indicative of AI editorializing,1 and that I should, thus, take the message with a grain of salt.
Because I am extremely familiar with the iOS interface, new exactly what I was doing in enabling Apple Intelligence, and am very familiar with AI and the way it works to know not to take output at pure face value.
For the average user, I don’t think the interface Apple implemented creates enough of a difference in experience to highlight all of the above obviously.
Apple and the iPhone have spent over 15 years teaching us that the little messages that show up on your lock screen are verbatim notifications from the app indicated by the icon.
Even I, knowing everything I indicated above, have caught myself once or twice because the way summarized notifications are presented is so close to the way normal notifications are presented. This is especially true if quickly glancing a notification on an Apple Watch.
I think a better UX approach would be to lean into the rainbow border that Apple has added to Siri and other Apple Intelligence functions to indicate the use of AI. If any summarized notifications had that obvious multicolored visual cue, I think it would be easier both for users to differentiate and for many of us to raise an eyebrow and say “really?” To those who struggle to do so.
AI is new and, while those of us who use technology to its extreme have become intimately familiar with many of its facets over the last couple years, most non technical users of consumer electronics devices just accept automatic updates and press “yes” when their phone recommends turning a feature on. With something that changes the paradigm so much as do many AI features, the difference should be made more clear.
I use the word “editorializing” intentionally. I think any company employing AI to transform information being presented to a user is applying editorial to that information and should be responsible for it. (This isn’t a direct comment about the particular story here, simply a general statement).