Utility of Experimental Design in Automatic Target Recognition Performance Evaluation
James M Higdon
Synopsis "Utility of Experimental Design in Automatic Target Recognition Performance Evaluation"
This research investigates current practices in test and evaluation of classification algorithms, and recommends improvements. We scrutinize the evaluation of automatic target recognition algorithms and rationalize the potential for improvements in the accepted methodology. We propose improvements through the use of an experimental design approach to testing. We demonstrate the benefits of improvements by simulating algorithm performance data and using both methodologies to generate evaluation results. The simulated data is varied to test the sensitivity of the benefits to a broad set of outcomes.The opportunities for improvement are threefold. First, the current practice of "one-at-a-time" factor variation (only one factor is varied in each test condition) fails to capture the effect of multiple factors. Next, the coarse characterization of data misses the opportunity to reduce the estimate of noise in test through the observation of uncontrolled factors. Finally, the lack of advanced data reduction and analysis tools renders analysis and reporting tedious and inefficient. This research addresses these shortcomings and recommends specific remedies through factorial testing, detailed data characterization, and logistic regression. We show how these innovations improve the accuracy and efficiency of automatic target recognition performance evaluation.This work has been selected by scholars as being culturally important, and is part of the knowledge base of civilization as we know it. This work was reproduced from the original artifact, and remains as true to the original work as possible. Therefore, you will see the original copyright references, library stamps (as most of these works have been housed in our most important libraries around the world), and other notations in the work.This work is in the public domain in the United States of America, and possibly other nations. Within the United States, you may freely copy and distribute this work, as no entity (individual or corporate) has a copyright on the body of the work.As a reproduction of a historical artifact, this work may contain missing or blurred pages, poor pictures, errant marks, etc. Scholars believe, and we concur, that this work is important enough to be preserved, reproduced, and made generally available to the public. We appreciate your support of the preservation process, and thank you for being an important part of keeping this knowledge alive and relevant.
