Research for Reliability

As part of the Tricare- Tidewater evaluation, CNA fielded a military beneficiary health care survey in the fall of 1992 to collect information on access, satisfaction, health status, and utilization. To supplement these data, and support the Tricare evaluation, we developed an administrative data set using claims data from the Civilian Health and Medical Program for the Uniformed Services (CHAMPUS), Biometrics records, and Medical Expense and Performance Reporting System (MEPRS) data. This administrative data set provides not only cost information but also detailed utilization records, including information about specific diagnoses and treatments. While working with the survey data, we observed internal inconsistencies in how people responded to the utilization section of the survey. These inconsistencies raised concerns about the veracity of the self-reported use measures. Given these concerns and the unique opportunity of having both survey and administrative measures of use for the same population, we thought it was appropriate to attempt to validate the self-reported utilization data against administrative records. In this paper, we compare the self-reported utilization data to the corresponding administrative utilization data, created from the CHAMPUS Quick Response Data File and Biometrics records, for a subset of our survey respondents. We look at incidences of overreporting and underreporting use in the survey data, and we attempt to explain these occurrences and the impact of this misreporting on aggre

The Armed Services Vocational Aptitude Battery (ASVAB), used to select and classify enlisted personnel, is highly correlated to math and verbal content areas. New computerized predictor tests that are sensitive to traits not measured by the current ASVAB subtests may be able to improve predictive validity. This research memorandum investigates the potential of one such group of tests, the Enhanced Computer-Administered Tests, to predict performance in the mechanical maintenance specialties.

A fundamental requirement in the development and administration of performance measures is that such assessments should result in reliable scores that accurately indicate a person's level of proficiency. This research memorandum examines the reliability of two performance measures of mechanical maintenance developed for the Marine Corps Job Performance Measurement project: hands-on performance tests and job knowledge tests. Multiple estimates of reliability were computed, and the consistency of test administrators in scoring hands-on performance was specifically examined.

An experimental computerized adaptive testing (CAT) version of the Armed Services Vocational Aptitude Battery (ASVAB) has been developed and administered, and a new version is under preparation. It is important that each CAT-ASVAB subtest be at least as reliable as its paper-pencil counterpart. This report presents two methods for estimating subtest reliabilities of the CAT version of the ASVAB, and illustrates them using data from the experimental version. These methods can be used with later versions.