Articles
Validation: Assessing the Legitimacy of Computational Results
Fri, 05/15/2009 - 8:08am
Validation: Assessing the Legitimacy of Computational Results
Evaluating the truth and justification of scientific beliefs is an essential part of computation-based science
Validation is a critical part of the scientific process and of scientific computation. While not sexy, the validation process ensures that responsible research is being performed and that legitimate results are being produced, published and promoted. An essential part of the scientific method, it seeks to evaluate the truth and justification of a scientific belief. For computation-based science, this includes open and impartial scrutiny of the computational model by peers in the field, and the creation of unbiased and reproducible tests to justify a belief by others that the model (and its instantiation in both software and hardware) actually corresponds to some truth or physical reality.
An investor in a biotechnology company once summarized these concepts both succinctly and pointedly with the question: How do I know the computer will do something that actually represents what happens in a test tube as opposed to running some expensive computer game with my money? The answer, in this case, was to perform a blind experiment to answer a pre-determined and specific question. However, the investor’s general question represents a conundrum with which scientists, managers, investors and customers must all wrestle and answer as computational methods are proposed for ever more general problems.
With the advent of inexpensive low-power yet high-performance computing hardware, computer vision appears to be a hot topic of research that may someday permit computers to “see” well enough to perform recognition and visual association tasks. A surprising challenge with computer vision research is that people quite easily perform visual association and recognition tasks themselves and, hence, can easily fool themselves into mistakenly believing an algorithm or method has general efficacy. This challenge applies to both researchers as well as customers. An unfortunate consequence can be the loss of both time and money, as inadequate or inadvertently biased testing and/or demonstrations appear to provide justification for further investment in some purported solution that fails to live up to expectation. Caveat emptor applies.
With a myriad of computer vision research projects and companies making assertions and claims, the noise level in computer vision research appears to be increasing. Happily, open source computer vision projects, such as OpenCV, OpenVidia and many others provide a source of comparison. Of course, test data sets and real-world data also are excellent sources of comparison, but ensure that the test sets do not have some form of bias that can make a particular demonstration look good. More generally, having both a “show me” and “code talks” attitude can be one of the safest starting points. Most generally, if you cannot test it, and it sounds too good to be true, consider that it may actually be too good to be real.
Unfortunately, novelty is also a challenge in a hot topic area such as computer vision, as the same concepts can be re-invented and touted as new. For years, even popular media publications such as Popular Mechanics have discussed “face detection” technology, including consumer cameras that can autofocus on faces, tag pictures as “mom” or zoom in on faces with the LCD display after the picture is taken to see whether someone had their eyes closed. A quick Internet search shows that computer vision has been actively researched for years and even decades. Again, many of the open source projects can be used as resources to check for novelty. Some projects contain libraries that implement hundreds of known methods. Other Internet resources include search engines and portals, such as CVonline. The point being that appropriate due diligence — along with appropriate disclosure and validation — are needed while making the decision to work upon or fund any project.
Some areas of research, such as the social sciences, use a technique called triangulation to validate a scientific belief. Triangulation is the application and combination of several research methodologies in the study of the same phenomenon. It is an alternative to the previous discussion and is used to overcome the weakness or intrinsic biases in single method, single-observer, single-theory studies in the hope that the accuracy and credibility of the result will be increased. The idea is that one can be more confident with a result if different methods lead to the same result. If an investigator uses only one method, the temptation is strong to believe in the findings. If an investigator uses two methods, the results may well clash. By using three methods to get at the answer to one question, the hope is that
• two of the three will produce similar answers, or
• if three clashing answers are produced, the investigator knows that the question needs to be reframed, methods reconsidered, or both.
In vision research, for example, it is generally possible to identify a face or object in a picture (e.g. “this is a picture of my mother”) and to measure how well a method compares against a validation data set. Many social scientists, on the other hand, are confronted with the challenge of modeling social interactions and human systems based on incomplete and subjective data based in the interpretations of multiple subject matter experts. For these areas of research, triangulation has been accepted as a valid and powerful method for establishing and justifying the truth of a scientific belief.
Rob Farber is a senior research scientist in the Molecular Science Computing Facility at the William R. Wiley Environmental Molecular Sciences Laboratory, a Department of Energy national scientific user facility located at Pacific Northwest National Laboratory in Richland, WA. He may be reached at editor@ScientificComputing.com.
Related Resources
CVonline homepages.inf.ed.ac.uk/rbf/CVonline/SUPPORT/overview.htm
OpenCV en.wikipedia.org/wiki/OpenCV
OpenVidia openvidia.sourceforge.net/index.php/OpenVIDIA
Triangulation en.wikipedia.org/wiki/Triangulation_(social_science)
Evaluating the truth and justification of scientific beliefs is an essential part of computation-based science
Validation is a critical part of the scientific process and of scientific computation. While not sexy, the validation process ensures that responsible research is being performed and that legitimate results are being produced, published and promoted. An essential part of the scientific method, it seeks to evaluate the truth and justification of a scientific belief. For computation-based science, this includes open and impartial scrutiny of the computational model by peers in the field, and the creation of unbiased and reproducible tests to justify a belief by others that the model (and its instantiation in both software and hardware) actually corresponds to some truth or physical reality.
An investor in a biotechnology company once summarized these concepts both succinctly and pointedly with the question: How do I know the computer will do something that actually represents what happens in a test tube as opposed to running some expensive computer game with my money? The answer, in this case, was to perform a blind experiment to answer a pre-determined and specific question. However, the investor’s general question represents a conundrum with which scientists, managers, investors and customers must all wrestle and answer as computational methods are proposed for ever more general problems.
With the advent of inexpensive low-power yet high-performance computing hardware, computer vision appears to be a hot topic of research that may someday permit computers to “see” well enough to perform recognition and visual association tasks. A surprising challenge with computer vision research is that people quite easily perform visual association and recognition tasks themselves and, hence, can easily fool themselves into mistakenly believing an algorithm or method has general efficacy. This challenge applies to both researchers as well as customers. An unfortunate consequence can be the loss of both time and money, as inadequate or inadvertently biased testing and/or demonstrations appear to provide justification for further investment in some purported solution that fails to live up to expectation. Caveat emptor applies.
With a myriad of computer vision research projects and companies making assertions and claims, the noise level in computer vision research appears to be increasing. Happily, open source computer vision projects, such as OpenCV, OpenVidia and many others provide a source of comparison. Of course, test data sets and real-world data also are excellent sources of comparison, but ensure that the test sets do not have some form of bias that can make a particular demonstration look good. More generally, having both a “show me” and “code talks” attitude can be one of the safest starting points. Most generally, if you cannot test it, and it sounds too good to be true, consider that it may actually be too good to be real.
Unfortunately, novelty is also a challenge in a hot topic area such as computer vision, as the same concepts can be re-invented and touted as new. For years, even popular media publications such as Popular Mechanics have discussed “face detection” technology, including consumer cameras that can autofocus on faces, tag pictures as “mom” or zoom in on faces with the LCD display after the picture is taken to see whether someone had their eyes closed. A quick Internet search shows that computer vision has been actively researched for years and even decades. Again, many of the open source projects can be used as resources to check for novelty. Some projects contain libraries that implement hundreds of known methods. Other Internet resources include search engines and portals, such as CVonline. The point being that appropriate due diligence — along with appropriate disclosure and validation — are needed while making the decision to work upon or fund any project.
Some areas of research, such as the social sciences, use a technique called triangulation to validate a scientific belief. Triangulation is the application and combination of several research methodologies in the study of the same phenomenon. It is an alternative to the previous discussion and is used to overcome the weakness or intrinsic biases in single method, single-observer, single-theory studies in the hope that the accuracy and credibility of the result will be increased. The idea is that one can be more confident with a result if different methods lead to the same result. If an investigator uses only one method, the temptation is strong to believe in the findings. If an investigator uses two methods, the results may well clash. By using three methods to get at the answer to one question, the hope is that
• two of the three will produce similar answers, or
• if three clashing answers are produced, the investigator knows that the question needs to be reframed, methods reconsidered, or both.
In vision research, for example, it is generally possible to identify a face or object in a picture (e.g. “this is a picture of my mother”) and to measure how well a method compares against a validation data set. Many social scientists, on the other hand, are confronted with the challenge of modeling social interactions and human systems based on incomplete and subjective data based in the interpretations of multiple subject matter experts. For these areas of research, triangulation has been accepted as a valid and powerful method for establishing and justifying the truth of a scientific belief.
Rob Farber is a senior research scientist in the Molecular Science Computing Facility at the William R. Wiley Environmental Molecular Sciences Laboratory, a Department of Energy national scientific user facility located at Pacific Northwest National Laboratory in Richland, WA. He may be reached at editor@ScientificComputing.com.
Related Resources
CVonline homepages.inf.ed.ac.uk/rbf/CVonline/SUPPORT/overview.htm
OpenCV en.wikipedia.org/wiki/OpenCV
OpenVidia openvidia.sourceforge.net/index.php/OpenVIDIA
Triangulation en.wikipedia.org/wiki/Triangulation_(social_science)
