Research for Validation and Validity

This memorandum reports the procedures used for the development and scoring of hands-on performance tests for the mechanical maintenance phase of the Job Performance Measurement project.

All large-scale data collection efforts must contend with the issue of data quality. This research memorandum examines the quality of data collected for the automotive maintenance portion of the Marine Corps Job Performance Measurement project, and describes measures taken to minimize the effect of questionable or missing cases. Particular attention is focused on data inconsistencies and problems associated with operational fielding of the Computer Adaptive Test-Armed Services Vocational Aptitude Battery and the Enhanced Computer Administered Test.

A surrogate measure is an expedient indicator of job proficiency that is reliable and valid and that yields results similar to a Hands On Performance Test (HOPT). HOPTs are the benchmark by which surrogates should be compared but HOPTs are too expensive, dangerous, or time intensive to use for most purposes. This paper uses Job Performance Measurement (JPM) data to analyze the usefulness of six surrogates (proficiency marks, training grade point average, conduct marks, job knowledge tests, video firing tasks, and supervisor ratings) for infantry tests. The paper addresses how useful each proxy would be for setting infantry enlistment standards and assessing Marine Corps training needs.

New kinds of tests are being evaluated as potential additions to the Armed Services Vocational Aptitude Battery (ASVAB). They are compared on the basis of the criterion variance they explain when added to the ASVAB. The evaluation may use scores on the ASVAB given during enlistment processing, or a new ASVAB may be administered concurrently with the new tests. This paper compares these two research designs in terms of their effect on evaluation of new tests. The analysis uses Infantry data from the Marine Corps' Job Performance Measurement project, in which concurrent as well as enlistment ASVAB scores are available. While lower increments in explained variances are obtained when the ASVAB is concurrent, the difference between the administrations is small compared to variations across criterion variables and occupational areas.