Suzanne Tracy, Editor-in-ChiefA computer program is running 24 hours a day at Carnegie Mellon University, searching the Web for images, doing its best to understand them on its own and, as it builds a growing visual database, gathering common sense on a massive scale.

The Never Ending Image Learner, or NEIL, leverages recent advances in computer vision that enable computer programs to identify and label objects in images, to characterize scenes and to recognize attributes, such as colors, lighting and materials, all with a minimum of human supervision. In turn, the data it generates will further enhance the ability of computers to understand the visual world.

But NEIL also makes associations between these things to obtain common sense information that people just seem to know without ever saying — that cars often are found on roads, that buildings tend to be vertical, and that ducks look sort-of like geese. Based on text references, it might seem that the color associated with sheep is black, but people — and NEIL — nevertheless know that sheep typically are white.

According to Abhinav Gupta, assistant research professor in Carnegie Mellon's Robotics Institute, images are the best way to learn visual properties. Images also include a lot of common-sense information about the world. People learn this by themselves and, with NEIL, he hopes that computers will do so as well. The research is supported by the Office of Naval Research and Google.

A computer cluster has been running the NEIL program since late July and already has analyzed three million images, identifying 1,500 types of objects in half a million images and 1,200 types of scenes in hundreds of thousands of images. It has connected the dots to learn 2,500 associations from thousands of instances.

One motivation for the NEIL project is to create the world's largest visual-structured knowledge base, where objects, scenes, actions, attributes and contextual relationships are labeled and cataloged. Some projects have tried to compile this structured data with human assistance. But the scale of the Internet is so vast — Facebook alone holds more than 200 billion images — that the only hope to analyze it all is to teach computers to do it largely by themselves.

People also tell NEIL what categories of objects, scenes, etcetera, to search and analyze. But sometimes, what NEIL finds can surprise even the researchers. It can be anticipated, for instance, that a search for "apple" might return images of fruit, as well as laptop computers. But Gupta and his landlubbing team had no idea that a search for F-18 would identify not only images of a fighter jet, but also of F18-class catamarans.

Read the full article, entitled “New Research Aims to Teach Computers Common Sense,” and more coverage of the high performance computing industry on our Web site at You can view NEIL's findings at the project Web site,