Watson vs. Humans

IBM’s Watson computer, which recently made an impressive showing on Jeopardy!, will next be tasked with helping clinicians in the healthcare environment. In a deal with Nuance Communications, Inc., Watson’s Deep Question Answering, Natural Language Processing, and Machine Learning capabilities will be combined with Nuance’s speech recognition and Clinical Language Understanding to address the increasingly complex demands of healthcare diagnostics. The IBM press release has more details (and lots of capitalized words) on the deal.

One of the hotter topics surrounding Watson’s dominant performance over the course of the three day demo was its (his?) spectacular failure in Final Jeopardy!. The category: US Cities; the answer: “Its largest airport was named for a World War II hero; its second largest, for a World War II battle.” Watson answered “What is Toronto?” IBM’s explanation suggests that Watson relies on statistical analysis for its reasoning and that it downgrades the category description in importance because in this game, they can be vague or downright tricky. But you and I would immediately recognize “US Cities” as a constrained set in which the answer must reside. Watson was confused further, IBM explained, because there are several “Torontos” in the US.

Oh-Oh! If you look at the map they provide showing the various cities in the US named Toronto, it is clear poor Watson must depend on what others tell him is true. As of this writing, the map (click on “What is Toronto?”)¬†labels “Toronto, MO” as being in Illinois, and “Toronto, IL” as being in Missouri. I thought Watson was impressive, and I think IBM has made great strides. As Watson demonstrated, machines can make mistakes. As the map at IBM’s site demonstrates, humans make mistakes too. We just have to remember that at least for now, the only knowledge computers can learn, is what we offer them…

[Update February 21, 2011: Alas, IBM appears to have removed the map above from its website. Cached versions don’t link to the map anymore either…]

© 2011 Rod Piechowski, Inc. Consulting

Leave a Comment