Sign up for our daily and weekly newsletters, featuring the latest updates and exclusive content on industry-leading AI reporting. Learn more
Google continues to aggressively update Gemini as it moves toward its 2.0 model.
The company today announced a smaller variant of the Gemini 1.5, the Gemini 1.5 Flash-8B, as well as the “significantly improved” Gemini 1.5 Flash and the “more powerful” Gemini 1.5 Pro. The company says these offer performance improvements across many of its internal benchmarks, with the 1.5 Flash showing “huge gains” across the board and the 1.5 Pro performing even better in math, coding, and complex prompts.
“Gemini 1.5 Flash is currently the best in the world for developers,” Logan Kilpatrick, product manager for Google AI Studio, boasted in a post on X.
‘Latest experimental iteration’ of ‘unprecedented’ Gemini model
Google released Gemini 1.5 Flash (a lightweight version of Gemini 1.5) in May. The Gemini 1.5 model family is built to handle long contexts and can infer fine-grained information from 10M+ tokens. This allows the model to handle large multimodal inputs, including documents, videos, and audio.
Today, Google is shipping an “improved version” of the smaller, 8-billion-parameter variant of Gemini 1.5 Flash. Meanwhile, the new Gemini 1.5 Pro is a “drop-in replacement” for its predecessor, which was released in early August, and shows improved performance in coding and complex prompts.
Kilpatrick didn’t provide further details, saying only that Google “hopes to release it with a preview version in the coming weeks!”
He explained in the X thread that experimental models are a means to collect feedback and get the latest in-progress updates to developers as quickly as possible. “What we learn from experimental releases has implications for how we roll out models more broadly,” he posted.
The “latest experimental iterations” of Gemini 1.5 Flash and Pro have a 1 million token cap and are available for free testing via Google AI Studio and the Gemini API, and soon via the Vertex AI experimental endpoint. Both versions have free tiers, according to Kilpatrick, and the company plans to make future versions available for production use in the coming weeks.
Starting September 3, Google will automatically redirect requests to the new model and remove the old model from Google AI Studio and its APIs to “avoid the confusion of maintaining too many versions at once,” Kilpatrick said.
He posted to X, “Looking forward to seeing what you think and hearing how this model opens up more new multimodal use cases.”
Google DeepMind researchers call the scale of Gemini 1.5 “unprecedented” among modern LLMs.
“We’ve been overwhelmed by the enthusiasm for our first experimental model that we released earlier this month,” Kilpatrick wrote on X. “We’ve put a lot of effort into getting this out into the world at Google, and we can’t wait to see what you build!”
‘Solid Improvements’ Still Suffering from ‘Lazy Coding Disease’
Just hours after its launch today, the Large Model Systems Organization (LMSO) has posted a leaderboard update for the chatbot arena, based on 20,000 community votes. Gemini 1.5-Flash has made a “big leap” from 23rd place to 6th place, matching Llama’s level and surpassing Google’s Gemma open model.
Gemini 1.5-Pro also showed “strong performance” and “improvements” in coding and math.[d] considerably.”
LMSO praised the model, posting, “A huge congratulations to the Google DeepMind Gemini team for this amazing launch!”
As with any iterative model release, early feedback was mixed, ranging from flattering praise to ridicule and confusion.
Some X users have questioned why there are so many consecutive updates compared to version 2.0. One user posted, “Hey, this isn’t going to work anymore 😐 We need Gemini 2.0, the real upgrade.”
On the other hand, many self-proclaimed fanboys praised the rapid upgrade and fast shipping, reporting “solid improvements” in image analysis. “The speed is blazing,” one person posted, while another noted that Google keeps releasing while OpenAI is virtually silent. “The Google team is quietly, diligently, and relentlessly delivering,” one person even said.
But some critics have called it “awful,” “lazy” at handling tasks that require longer outputs, and said Google is “way behind” Claude, OpenAI, and Anthropic.
One X user lamented that this update, similar to GPT-4 Turbo, “unfortunately suffers from lazy coding disease.”
Another called the updated version “definitely not as good” and said it “often drives you crazy and you start doing things over and over again, which is what you often do with smaller models.” Another was excited to try it out, but agreed that Gemini was “by far the worst when it comes to coding.”
Some have mocked Google’s callous naming conventions and pointed to a huge mistake it made earlier this year.
“You guys have completely lost the ability to name things,” one user joked, while another agreed: “You guys seriously need someone to help you with your naming.”
And one person asked dryly: “Does Gemini 1.5 still hate white people?”
