Why Gemini Probably Isn’t as Good as Google Says It Is; A New Open-Source Security Threat: AIJacking

After months of build-up, developers were excited to finally see the long-awaited release of Google’s Gemini models yesterday. But their enthusiasm was tempered by the fact that the only model available right now is Gemini Nano, which you can guess means it’s small—so small it can run on Google’s Pixel phones. Developers have to wait until next week’s release of Gemini Pro, Google’s equivalent of GPT-3.5, to have a model worthy of testing, they told Jon Victor and me. 

And that’s not the only reason to be cautious about coming to any conclusion about Gemini. While many developers were impressed by the multimodal capabilities Gemini showcased in a splashy video demo, replicating that sort of performance is likely to be harder than the video made it seem. (Leave aside the fact that those multimodal capabilities aren’t available yet.) In making the video, Google researchers didn’t prompt Gemini with normal speech and video visuals like hand gestures. Instead, they seem to have used a carefully crafted combination of images and text, according to this blog post. Some of the actual prompts were more specific and detailed than the dialogue in the video.

Google staffers must be feeling the pressure because they also took some additional steps to make Gemini look extra good compared to its rivals. Front and center on Gemini’s release site is a chart claiming that Gemini’s most-advanced model, Ultra, outperforms OpenAI’s GPT-4 by a large margin on a commonly-used evaluation, MMLU.