We’ve all been amazed at the things ChatGPT can do — but what about its “fails”? We’ve compiled here a list of simple questions that ChatGPT sometimes gets wrong, using its current Turbo GPT-3.5 model. We’ll leave it to the reader to figure out why they’re wrong, but if you need any ChatGPT training to get the right answers, let us know!
Prompt 1: What countries start with the letter V? – credited to ChanMix on Reddit. The confidently incorrect answer:
Prompt 2: I put a diamond in a cup and then placed the cup upside down on my bed. Later I came back, took the cup, and put it in the fridge. Where is the diamond? – credited to Daniel Feldman on Twitter. An interesting hallucination, also ChatGPT being confidently incorrect:
Prompt 3: Mary weighs 85 pounds. Susan weighs 105 pounds. Jake weighs 115 pounds. Two of them standing together on the same scale could weigh 200 pounds, right? -credited to Sergey Ivanov on Twitter. ChatGPT provided its reasoning for this confidently incorrect answer:
Prompt 4: Imagine a maximum security prison with 100 male inmates at the beginning of the year, with many being in rival gangs that hate and often murder each other. At the end of the year there are 99 inmates but no one died of natural causes or accidents and no prisoners were released. What happened? This question was somewhat nebulous, but it’s unlikely that a male inmate would become pregnant, and even then, the math doesn’t work:
Prompt 5: How many words are in your response to this prompt? – credited to Ozpeter in OpenAI blog
Prompt 6: Name 5 countries that start and end with the same letter. – from this New York Times article. It seems to miss the obvious…
Prompt 7: Name a Swedish band that is a palindrome.
Prompt 8: I’m Suzie and I’m 6 years old. My mommy says Santa will visit us in December and give us prezzies and candy but my older brother Johnny says that there is no Santa Claus. Is Santa Claus real?
Interestingly, the paid version of ChatGPT, with its next-generation GPT-4 model, got most of these questions correct. Here are some runners-up that GPT-3.5 answered correctly most — but not all — of the time:
- Write a sentence that ends in the words “so far.”
- Riddle: You have two chickens and a fox that need to be transported across a river using a boat. The boat can only carry one animal at a time. You cannot leave a chicken alone with the fox, as the fox will eat the chicken. Animals are unable to operate the boat; only you can do that. Determine the steps needed to safely transport all three animals to the other side of the river.