Big-Data M e t h o ds Measurement

Big-Data M e t h o ds Measurement

Big-Data M e t h o ds Measurement Statistical Innovations for Health and Educational Research Steven Andrew Culpepper Department of Statistics University of Illinois at Urbana-Champaign Educational Testing Service May 6, 2016

Big-Data M e t h o ds Measurement Methodological Areas for Innovation Statistical inference for large-scale surveys involving latent variables. Collaboration with Trevor Park. AERA Grant #2014-03610-00 Fine-grained measurement of cognitive skills and attributes. Collaboration with Jeff Douglas, Yuguo Chen, and Yinghan Chen.

Big-Data M e t h o ds Measurement NAEP Overview NAEP provides data on the status of what Americas students know and can do in various subject areas. To reduce test time large-scale assessments administer a random sample of items in a given content area. Individual score estimates are more variable, so NAEP measures achievement for groups rather than individuals. A Multiple Imputation (MI) procedure (e.g., Rubin, 1987; Thomas, 1997) is employed to predict group proficiency with student, teacher, and school administrator survey responses.

Researchers should instead use statistical methods that are designed for high-dimensional inference. Big-Data M e t h o ds Measurement Why Model Selection in Large-Scale Testing? Large-scale surveys include 100s of variables. Identifying the most relevant predictors of achievement is more challenging. Applied researchers need high-dimensional regression procedures for testing hypotheses. The developed model could be used to support the redesign of

background questionnaires. Big-Data M e t h o ds Measurement Bivariate Normal and Generalized Laplace Priors Big-Data M e t h o ds Measurement NAEP Application

The GAL, MVN, and AM software were applied to the 2011 NAEP mathematics data. NAEP administered J = 155 items to N = 175,200 8th grade students to assess mathematics achievement in K = 5 subject areas: algebra (J1 = 49); data analysis, statistics, and probability (J2 = 23); geometry (J3 = 30); measurement (J4 = 26); and number properties and operations (J5 = 27) The model included G = 148 groups with a total of V = 262 variables. Big-Data

M e t h o ds Measurement Comparison of Methods for 2011 NAEP Mathematics Big-Data M e t h o ds Measurement Comparison of GAL and AM Software Table: Race-Based Achievement Gaps using the GAL model and Plausible Values (PV) Race

African Am. Content Area 1 2 3 4 5 GAL EST SE -0.498 0.027 -0.515 0.034

-0.559 0.031 -0.595 0.030 -0.573 0.029 PV EST SE -0.440 0.031 -0.383 0.030 -0.471 0.025 -0.478 0.035 -0.489 0.025 Note. Results are unweighted. 1 = algebra; 2 = data analysis, statistics, and probability; 3 = geometry; 4 = measurement; and

5 = number properties and operations. Big-Data M e t h o ds Measurement Implications for Test Developers 83 out of 148 groups of variables statistically related with achievement. The GAL prior could be used to decide which background questions to retain, modify, or delete for subsequent data collections. Such efforts could optimize the time students, teachers, and school administrators dedicate to completing surveys.

Big-Data M e t h o ds Measurement Implications for Researchers The GAL prior and AM software produced different estimates of the achievement gap. The GAL also yielded a more parsimonious model in Monte Carlo studies and the application. We would like to disseminate the methodology as an R package. Big-Data

M e t h o ds Measurement School in the Year 2000 - Postcard from the 1900 World Exhibition in Paris Big-Data M e t h o ds Measurement Latent Variable Models Latent variable models assume that a collection of unobserved traits or attributes underlie observed test or survey responses.

Most studies consider broadly defined, continuous latent variables. Broadly defined continuous latent variables are useful for correlational research and ranking individuals on traits. Cognitive Diagnosis Models (CDMs) instead consider a set of discrete binary attributes/skills. Big-Data M e t h o ds Measurement Cognitive Diagnosis Models CDMs provide more detailed diagnostic information regarding student skills/attributes than is available with more broadly

defined continuous traits in item response models. CDMs have been applied in several areas: Education Pathological gambling Anxiety disorders. The application of CDMs is dependent upon the availability of cognitive theory that specifies the skills and/or attributes necessary for success on a collection of tasks. Big-Data M e t h o ds Measurement Purdue Spatial Visualization Test Rotation (PSVT-R): Item #1

Big-Data M e t h o ds RSVT-R: Item #2 Measurement Big-Data M e t h o ds Measurement Continuous vs. Discrete Latent Variables A continuous IRT model would assume test-takers broadly defined spatial abilities can be mapped to random variable i

A cognitive diagnosis model (CDM) would classify students into attribute classes, Ii = (i1, . . . , iK ) where ik0 =i does not have attribute k 1 i has attribute k We could classify individuals based upon four rotation attributes: i1 = 90o x-axis, i2 = 90o y-axis, i3 = 180o x-axis, and i4 = 180o y-axis. Skills could form a hierarchy such that i1 must be mastered before i3 and i2 before i4 . Big-Data M e t h o ds

Measurement Challenges CDMs require robust cognitive theory that clearly specifies the underlying attributes/skills. Cognitive theory is catalogued in the Q matrix. The unavailability of cognitive theory to specify Q limits widespread application of CDMs. Big-Data M e t h o ds Measurement Statistical Advances

Developed statistical methodology to estimate Q. The new procedure can be employed to assess existing cognitive theory or develop new theory. CDMs can be accurately applied to understand learning progressions.

Recently Viewed Presentations

  • Rights and Responsibilities - John A. Ferguson High School

    Rights and Responsibilities - John A. Ferguson High School

    You finished student teaching and the school district you want to teach in has called you for an interview. Mr. Thomas, from the personnel office, seems impressed with your credentials and the interview is going well. He explains that the...
  • Nuclear Size  Different experiments give different results  Radius

    Nuclear Size Different experiments give different results Radius

    Nuclear Size Different experiments give different results Radius not well defined. Depends on probe and relevant physics. Probes should be close to the order of the size of the nucleus ~ 10-14 m.
  • This is a test of this font THIS IS A TEST OF THIS FONT

    This is a test of this font THIS IS A TEST OF THIS FONT

    If you select a flight and it has this yellow exclamation point icon, the system is notifying you that your flight is not within policy. The system will still allow you to book your flight, however, it will require you...
  • Cursing the enemies of the Ahl al-Bayt

    Cursing the enemies of the Ahl al-Bayt

    Prepared by Tablígh Sub-Committee of ISIJ of Toronto Ziyárat of Condolence for the Day of `Áshúra ãÐáäÃÂB ãTáÒå·ã{ áÅájB áZãnBÑ CÖ áÀå×áÃá® âÅÚáätÂB Peace be on you, O inheritor of Ádam, the chosen one of Alláh. Ziyárat of Condolence for...
  • Concrete Steps into 3G Mobility & Margin

    Concrete Steps into 3G Mobility & Margin

    These applications are primarily for business markets, although several also serve consumer markets, such as education and online healthcare • mobile office - horizontal business services common to a wide range of industries and individuals, such as corporate database access,...
  • Compliant Mechanisms Presented By: Ravi Agrawal, Binoy Shah,

    Compliant Mechanisms Presented By: Ravi Agrawal, Binoy Shah,

    Compliant Mechanisms Presented By: Ravi Agrawal, Binoy Shah, and Eric Zimney Outline Working Principal Advantages and Disadvantages Compliance in MEMS devices Design and Optimization Analysis: Static and Dynamic Example Devices Conclusion Working Principle Deflection of flexible members to store energy...
  • The Names of God

    The Names of God

    12/25 Christmas. 1/1 New Year's Day. 1/8 Bible Study resumes in 2019. Outline . Names of God in the Hebrew Bible. ... Metaphors. Use what we know to describe what we don't know. First word, comes usually from the world...
  • Objective: To develop a tool box of passes

    Objective: To develop a tool box of passes

    Goal 1/3 of the netball court Equipment: 1 ball per pair Pairs stand facing each other in a line. Do 10 each way of the following passes Chest (arms by side, hands make a W with thumb tips touching, ball...