Siri Is Savvier, but Still Not Smarter Than a Fifth Grader

Let’s get this out of the way: Siri, Apple’s digital assistant, is not particularly smart. At least not in the conventional sense and certainly nowhere near as smart as your average 10-year-old.

And yet, Apple boasts that this version of Siri, which is the centerpiece of the latest iOS 9 mobile operating system, is “packed with more intelligence throughout” — indeed, the company claimed a 40 percent improvement in speed and accuracy. But more intelligent than what, exactly?

We decided to find out by putting Siri’s enhanced IQ to the test, administering some standard knowledge tests we found online. The results? Siri is undeniably capable, but she’s still no match for a child.

We started with an admittedly low bar: A quiz inspired by the TV game show “Are You Smarter Than a Fifth Grader?” Siri responded quickly and directly to just one of the five questions posed: Which planet is the closest to the sun? The answer is Mercury.

The rest of the time, Siri did what any school child does when he or she doesn’t know the answer to questions about, say, the length of a congressman’s term or the number of sides to a heptagon: She searched the Web. Google Now, by comparison, was able to provide the correct answers to three of the five questions posed.

Apple

The more advanced Wechsler adult intelligence scale proved more of a challenge for Siri, who doesn’t revel in abstract thinking. Take, for example, the response to this math question: “John, who is 12 years old, is three times as old as his brother. How old will John be when he is twice as old as his brother? Possible answers: 15, 16, 18, 20 or 21?

Apple

Clearly, we weren’t getting anywhere with these tests. Machine learning expert Indraneel Mukherjee from LiftIgniter explained that the virtual assistants of today aren’t anywhere close to approximating human intelligence. The technology is not there yet. Rather, Siri and other virtual assistants are designed for a more useful purpose: To reduce the time mobile users spend performing everyday tasks, such as finding the fastest route to the airport or the nearest gas station.

“It’s hard for one user to measure quantitative changes in the algorithm,” said Mukherjee, who worked as part of Google’s artificial intelligence team. “Qualitatively, you can determine, did the restaurant recommendations improve? Did it automatically figure out that I have to catch a flight and that there has been a delay in traffic? … If you can experience the difference, that would translate to a big percentage of improvement.”

Still, we searched for a way to quantify Apple’s claim of a 40 percent improvement.

Last year, Stone Temple Consulting, an SEO consultant and digital marketing firm, pitted Siri, Google Now and Microsoft’s Cortana against one another in a “knowledge smackdown” to see which could most accurately answer a series of 3,000 questions. Google Now pummeled Siri.

At Re/code’s behest, Stone Temple put Siri back in the hot seat for 12 hours this week to see how Apple’s more muscular digital assistant would perform, not against Google, but against itself. The firm used 475 questions drawn from its original survey to evaluate Siri’s responses. The questions were a set that Google Now had responded to accurately back in 2014, but with mixed results from old Siri. Could new Siri do better?

Stone Temple Chief Executive Eric Enge said the queries are similar to ones users might pose, including how-to questions (How do I apply eye liner? How do I apply for a mortgage?), requests for comparisons (Compare Jupiter to Mars.) or those that test trivial knowledge (How fast can a human run? How old is the sun?). The questions are designed to elicit a direct response.

Stone Temple evaluated Siri’s responses in two ways: Whether she offered a “rich answer” — that is, a direct response to the question, instead of displaying Web search results. Then, it checked for accuracy. New Siri improved on both measures.

Last year, Siri answered roughly a third of the questions asked, correctly responding 81 percent of the time. New improved Siri answered more of the questions posed — 41 percent — with greater accuracy at 92 percent. In other words, Siri’s accuracy in answering the same set of 475 questions improved by 32 percent.

Stone Temple Consulting

In our own test, Apple’s mobile assistant excels at responding to practical, real-world queries, such as: “What was the score of last night’s Angels’ game?” or “Should I bring an umbrella to work today?” or “Find me some good sushi restaurants nearby.” She also does a great job of on-the-fly currency conversions, telling the time in distant locales and predicting when the sun will set.

Savvier Siri also performs a few tricks that she couldn’t do in her earlier days. Asked to fetch photos from a spring trip to Hong Kong, she immediately displayed every image I captured — starting with the bridge that took me into the island from mainland China. Old Siri found images of Hong Kong from the Web.

Tell Siri to play the No. 1 song and she immediately launches Justin Bieber’s “What Do You Mean?” — then asks if I’d like to hear more like it (oh, hell no).

The assistant now offers public transit information, which she displays in Apple maps alongside driving and walking directions. This would be helpful for anyone living in a city other than Los Angeles, where getting to, say, the Getty Center from Orange County via train or bus isn’t an option.

There were some improvements to Siri that we didn’t evaluate, such as her ability to understand more languages and dialects.

Siri also has found new answers to life’s great questions. Ask the old assistant about the meaning of life and she will quip, “I find it odd that you would ask this of an inanimate object.”

The new and improved version offers some concrete advice that seems an extension of CEO Tim Cook’s nice-guy persona.

“Try and be nice to people. Avoid eating fat. Read a good book every now and then. Get some walking in, and try to live together in peace and harmony with people of all nations and creeds.”

Additional reporting by Mark Bergen.

This article originally appeared on Recode.net.

Explanatory journalism is a public good

At Vox, we believe that everyone deserves access to information that helps them understand and shape the world they live in. That's why we keep our work free. Support our mission and help keep Vox free for all by making a financial contribution to Vox today.

One-Time Monthly Annual

$95/year

$120/year

$250/year

$350/year

Other

$

Yes, I'll give $120/year

Yes, I'll give $120/year

We accept credit card, Apple Pay, and Google Pay. You can also contribute via