Table of Contents
Can Google’s long-awaited Gemini AI model overthrow OpenAI’s GPT-4?
Google’s introduction of Gemini AI has been highly anticipated, especially since the company has been vocal about its capabilities surpassing those of OpenAI’s GPT-4, the engine behind ChatGPT Plus. This launch provides an opportunity to evaluate how Gemini stacks up against GPT-4.
The debut of Google’s Bard in March 2023 sparked significant enthusiasm, as it signaled an end to OpenAI’s sole dominance in the ChatGPT space, introducing a formidable competitor.
However, Bard didn’t quite live up to the lofty expectations set for it, leaving GPT-4 as the leading force in the generative AI chatbot arena. Now, with the arrival of Google’s Gemini, the question arises: does this new AI model outshine ChatGPT?
What Is Google’s Gemini AI Model?
Google’s most advanced generative AI model, Gemini, can comprehend and function with a variety of data types, including text, audio, images, and videos. With the help of its most potent AI technologies, Google is attempting to develop a single AI model. There will be three versions of Gemini available:
- Gemini Ultra: The biggest and most powerful version, made to tackle really difficult jobs.
- Gemini Pro: Less capable than Ultra, but the ideal model for scaling and providing great performance across a wide range of activities.
- Gemini Nano: The most effective model created for task deployment on devices. Gemini Nano, for instance, allows developers to create integrated systems or mobile apps, introducing potent AI to the mobile market.
In a recent post on its official blog, The Keyword, Google announced that its Gemini Ultra model has surpassed current industry standards in multiple benchmarks. According to Google, Gemini Ultra has outperformed the industry’s frontrunner, GPT-4, in various critical benchmarks.
Google highlighted Gemini Ultra’s exceptional performance, particularly its achievement of a 90.0% score on the challenging MMLU benchmark. This score signifies a groundbreaking moment, as Gemini Ultra becomes the first model to exceed human-level performance in this comprehensive test, which covers 57 diverse subjects.
Gemini Ultra is designed to excel in comprehending, elucidating, and generating high-quality code in several widely-used programming languages, such as Go, JavaScript, Python, Java, and C++. These achievements are impressive on paper. However, it’s important to remember that these are benchmark results, which might not fully capture a model’s effectiveness in practical scenarios. Therefore, the key question remains: how effectively does Gemini perform in real-world applications?
How to Use Google Gemini AI
Among the three versions of Google’s Gemini AI model, Gemini Pro is readily accessible for use. Presently, you can experience Gemini Pro through Google’s Bard chatbot. To engage with Gemini Pro via Bard, simply visit bard.google.com and log in using your Google account.
Google has scheduled the release of Gemini Ultra for January 2024, so for the time being, comparisons are being made between Gemini Pro and ChatGPT.
How Gemini Compares to GPT-3.5 and GPT-4
Whenever a new AI model is introduced, it’s common practice to benchmark it against OpenAI’s GPT models, which are widely regarded as the gold standard in the field. To evaluate the capabilities of Google’s Gemini, we conducted tests involving Bard and ChatGPT, focusing on their proficiency in areas like mathematics, creative writing, code generation, and processing image inputs.
Our first test involved a basic math problem: calculating the result of -1 x -1 x -1. We began with Bard. Despite asking the question twice, Bard initially provided incorrect responses. It was only on the third try that we received the correct answer, though for a fair assessment, this delayed accuracy is not ideal.
We tried ChatGPT running on GPT-3.5. The first trial got it right.
In our evaluation of Gemini’s capabilities in image interpretation, we assigned it the task of analyzing several well-known memes. However, Gemini declined the task, stating it was unable to process images featuring people. On the other hand, ChatGPT, powered by GPT-4V, accepted the challenge and executed the task without any issues, demonstrating its proficiency in this area.
We made another attempt to test its coding and problem-solving skills while having it read an image. We provided a screenshot to Bard, which was using Gemini Pro, and instructed it to decipher and generate HTML and CSS code in order to recreate the screenshot.
Here’s the source screenshot.
Below is Gemini Pro’s attempt to interpret and replicate the screenshot using HTML and CSS.
Here is an attempt by GPT-4 to mimic the screenshot. The outcome is not unexpected given that GPT-4 has a long history of being a proficient coder. We’ve already shown how to create a web application from scratch using GPT-4.
We asked Gemini Pro to create a poem about Tesla (the electric vehicle car brand). It showed marginal improvements from previous tests we’ve done in the past. Here’s the result:
At this point, we believed it would be more acceptable to compare the findings against GPT-3.5 rather than the supercharged GPT-4. So we requested ChatGPT to write a poem akin to GPT-3.5.
It may be a personal choice, but Gemini Pro’s take on this seems better. But we’ll let you be the judge.
Is Gemini Better than ChatGPT?
Prior to the release of Bard, expectations were high that it would be a strong contender against ChatGPT. However, Bard did not live up to these expectations. Now, with the arrival of Gemini, the performance of its Gemini Pro variant has yet to make a significant impact in challenging ChatGPT’s dominance.
Google has made promising claims about the superior capabilities of Gemini Ultra, and there is anticipation that it could potentially meet or even surpass the expectations set in its announcement. Yet, the true test will come when we can fully evaluate the most advanced version of Google’s generative AI tool. Until then, the question of whether it can dethrone existing AI models remains unanswered. As of now, GPT-4 continues to hold its position as the leading AI model in the field.
