Program Update

1. Operational Psychology Department Naval Aerospace Medical Institute DET Naval Operational Medicine Institute Program Update

2. ASTB Background ASTB is primary tool used to select SNAs for the USN, USMC, & USCG and SNFOs for the USN & USMC Different sections of the ASTB are also used to select OCS candidates and Reserve Intel officers ASTB consists of six subtests: Math Skills Test (MST) Reading Comprehension Test (RCT) Mechanical Comprehension Test (MCT) Spatial Apperception Test (SAT) Aviation and Nautical Information Test (ANIT) Aviation Supplemental Test (AST) Current test battery primarily measures general cognitive ability (“g”) and relevant job knowledge (ANIT)

3. Score Components Weighted subtest score combinations yield three stanine scores (ranging from 1 to 9) : Academic Qualification Rating (AQR) – predicts academic performance in API and ground school Pilot Flight Aptitude Rating (PFAR) & Flight Officer Flight Aptitude Rating (FOFAR) – both predict flight grades in primary flight training AQR, PFAR, and FOFAR also predict attrition from flight training Also yields the Officer Aptitude Rating (OAR) – which is used in Officer Candidate School selection (ranging from 20-80) Composed of MST, RCT, and MCT Can be merged with remaining subtests for complete ASTB

4. Administration Available in traditional paper and pencil format and online via APEX.NET; moving to primarily web based delivery in 2007 150 permanent CONUS and OCONUS custody locations: OSOs, NRDs, NROTCs, Military INSTs, and selected NAS/MCAS APEX 2.0 currently in use at over 100 CONUS & OCONUS sites Sites average about 10,000 ASTBs administered every year, approaching 50% of all ASTB being delivered on-line via APEX Completed paper and pencil ASTBs are returned to NOMI for scoring, reporting, & database inclusion; APEX delivered tests are automatically scored, stored, and reports can be downloaded

5. Administrations by Format and FY

9. Predictive Relationships with Training Outcomes Validity data presented for FY98 to FY06JAN Test Components: AQR: Predictor of Academic NSS PFAR and FOFAR: Predictors of Primary Flight NSS Attrition

11. Administrative Reasons for Attrition

12. Training Task Analysis

13. BUMED Sponsored 5-YR Redevelopment Plan Goals: Improve predictive validity of the ASTB with emphasis on flight performance Reduce attrition, specifically focusing on flight-related and motivational components Increase test security Enhance measurement precision of cognitive ability portion of ASTB Specific Projects: Addition of Performance-Based Measures (PBMs) Development of a Forced Choice Aviator Personality Measure Development of a Aviation-Related Biographical Inventory Transition to computer adaptive testing (CAT) Currently, in year 3 of redevelopment process

14. Performance-Based Measures Assesses spatial and psychomotor abilities Provides for multi-tasking, divided attention, and multi-channel processing Leveraging USAF TBAS with modifications Modifications due to be completed by June 06 Data collection underway (N=72) on USAF version Practice Effects Relationships with current ASTB subtests Scoring algorithm Plan to examine relationships with stress-related personality measures

15. Personality Instrument Goal: minimize socially desirable responding (“faking”) by concealing the personality traits of interest from the examinee Multidimensional forced-choice adaptive measure What if you could only answer YES to one of the following statements: “I am highly motivated to succeed” “I am not easily rattled” Task analysis identified 9 personality traits relevant to aviation performance:

16. Personality Instrument Item development (200-300/scale) completed for all scales 20Mar06 Completed rater evaluations of item extremity and impression management for Dependability, Adaptability, and Stress Tolerance ICC for extremity ratings = .94 Items with between-rater SDs > 1.0 omitted Selection of anchor items (20 from each scale) 3 Low Extremity items (M < 1.90) 11 Moderate Extremity items (M = 4.00 – 4.90) 3 High Extremity items (M = M > 6.50) 3 items at M = 2.00, 3.00, and 5.00 Scales will assess three traits at a time: 3 different forms (60 anchor items + 180 assortment of items from three traits = 240 total items per form) Initiating data collection for Dependability, Adaptability, and Stress Tolerance Jun06 Parameter estimation Dimensionality assessment Rater evaluations for all scales scheduled to be completed by July06 Possible collaboration with NPRST utilizing NCAPS Ratings conducted on a 7-point scale To reduce the number of items presented to participants, it was necessary to minimize the number of anchor items each scale would contribute to an anticipated three pilot scales used in data collection. These anchor items, because of the large number of empirically observed responses, should provide the most stable item estimates in the pilot scale and could be tested on their own merit as stand-alone pen and paper scales that provide some glimpse of scale unidimensionality, a principle assumption for IRT scale development. Ratings conducted on a 7-point scale To reduce the number of items presented to participants, it was necessary to minimize the number of anchor items each scale would contribute to an anticipated three pilot scales used in data collection. These anchor items, because of the large number of empirically observed responses, should provide the most stable item estimates in the pilot scale and could be tested on their own merit as stand-alone pen and paper scales that provide some glimpse of scale unidimensionality, a principle assumption for IRT scale development.

17. Biographical Inventory Empirical literature reviewed to identify biographical variables related to attrition from aviation training Five factors identified concerning interest and involvement in: Science and engineering Athletics Military Orientation Outdoor Recreation Technical/Manual Activities Items for the 5 factors were derived from existing biographical measures, accomplishments measures, and focus groups with flight students Items are factual Items are verifiable (conclusive evidence) or supportable (convincing evidence) Items concern behavior under test taker’s control and involve opportunities and resources available to everyone Items use a combined fixed-response and open-ended format Items are not experimentally dependent on each other

18. Biographical Inventory Items pilot tested for content, clarity, and to determine whether any items were insensitive or unfair towards protected groups Prototype inventory 100 items for the five scales 3 supplementary items 7 demographic items Converted to computer-based administration Data collection initiated 04May06

19. CAT-ASTB In traditional testing, every examinee gets the same or parallel forms of test items However, not all test items are created equal Think of a test item as having 3 defining characteristics: Difficulty Discriminability Susceptibility to guessing If these characteristics for each test item are know, can essentially “create” a unique test for each examinee If you miss Q#1, Q#2 is easier If you get Q#1 right, Q#2 is harder This pattern continues until the test narrows in on true ability level Advantages: Greater measurement precision Shorter testing time Enhanced test security

20. CAT-ASTB CAT program in beta testing to ensure item selection and scoring algorithms are functioning appropriately with existing item pool (N = 75) Item bank development Contract with external testing company to develop 1,500 new test items Enhance graphics for 200 existing mechanical and spatial apperception items Analysis of existing parameter estimates resulted in more appropriate distribution of new items across difficulty levels Final delivery 30Jun06 Data collection to obtain parameter estimates Comparability studies (online, P&P, CAT) Validation analyses

22. Testing Labs Sponsored by CNATRA Establish in high-volume IFS locations for data collection to support ASTB redevelopment efforts Pensacola, FL** (25 seats) TBS (Quantico, VA) (40 seats) Naval Academy (Annapolis, MD) (40 seats) Computer equipment and peripherals Tests administered in IFS/NIFS