Charter Schools and Student Achievement in Florida

Charter Schools and Student Achievement in Florida

VALUE-ADDED MODELS AND THE MEASUREMENT OF TEACHER QUALITY Douglas Harris Tim R. Sass Dept. of Ed. LeadershipDept. of Economics and Policy Studies Florida State University Florida State University ([email protected]) ([email protected]) IES Research Conference June 2006 -- Preliminary, Do Not Quote Without P ermission -- Evaluating Value-Added Methodology The recent availability of panel data has produced a flood of research studies using various value-added approaches Research Questions Are assumptions underlying the valueadded approach valid? Are some methods more likely to produce reliable estimates than others? What data are most important to obtaining consistent estimates? -- Preliminary, Do Not Quote Without P ermission -- Evaluating Value-Added Methodology Basic Model Types Cumulative Model Unrestricted Value-Added Model Value-Added Models with Persistence Restrictions Restricted Value-Added or Gain-Score Model Contemporaneous Model Specification Issues for Value-Added Models

Treatment of teacher heterogeneity Measures of classroom/school inputs Treatment of student heterogeneity Aggregation -- Preliminary, Do Not Quote Without P ermission -- General Cumulative Model of Student Achievement A it A t X i ( t ), Fi ( t ), Ei ( t ), i 0 , it where A it Xi(t) Fi(t) Ei(t) i0 it = achievement level for individual i at the end of their tth year of life = entire history of individual inputs = entire history of family inputs = entire history of school-based educational inputs = a composite variable representing time-invariant characteristics an individual is endowed with at birth (such as innate ability) = a normally distributed, mean-zero error. -- Preliminary, Do Not Quote Without P ermission -- Basic Assumptions of Value-Added Models Cumulative achievement function does not vary with age and is additively separable. Family inputs are time invariant. Parents do not compensate for poor school inputs or poor outcomes Todd and Wolpin (2005) reject exogeneity of parental inputs at 90 percent, but not at 95 percent confidence level The marginal inputs of all school-based inputs, parental inputs, and the initial student endowment each decline geometrically (at potentially different rates) over time. Lagged achievement serves as a sufficient statistic for prior inputs

We find twice-lagged inputs do not provide additional information -- Preliminary, Do Not Quote Without P ermission -- Unrestricted Value-Added Model A it 1X it 1Eit A it 1 i it where i = an individual student effect representing time-invariant student/ family characteristics it = it - it-1 is a random error. Given it is a function of the lagged error in achievement, it is correlated with Y t-1 and ordinary least squares is inconsistent. Rarely consistently estimated with large-scale data sets. -- Ding and Lehrer (2005), Sass (2006). -- Preliminary, Do Not Quote Without P ermission -- Persistence Restrictions Restricted Value-Added or Gain-Score Model A A it A it 1 1X it 1Eit i it is assumed to equal 1 (no decay in effect of past inputs) Alternatively, can interpret as an achievement growth model where growth is independent of past school inputs. Contemporaneous Model A it 1X it 1Eit i it is assumed to equal 0 (complete decay in effect of past inputs). -- Preliminary, Do Not Quote Without P ermission -- Decomposition of Schoolbased Inputs in ValueAdded Model A it 1X it 2 P ijmt 3Tkt A it 1 i k m it Eit is decomposed into four components: classroom peer inputs, P-ijmt race, gender, mobility and age of peers; class size time-varying classroom teacher inputs, Tkt experience, professional development, advanced degrees time-invariant classroom teacher inputs, k race/ ethnicity, gender, pre-service education, pre-college ability school-wide time-invariant inputs, m

Assume non-teaching inputs are constant across classroom within a school and school-wide inputs do not vary over sample period. -- Preliminary, Do Not Quote Without P ermission -- Modeling Teacher Heterogeneity Substituting teacher time-invariant measured characteristics for teacher fixed effects A it 1X it 2 P ijmt 3Tkt A it 1 i YYk m it where it = (k-Yk) + it. This approach will produce biased estimates if unmeasured timeinvariant teacher characteristics, (k-Yk), which are now part of the error term, are correlated with observed time-varying student, peer or teacher variables in the model (ie. Xit, P-ijmt, Tkt). -- Preliminary, Do Not Quote Without P ermission -- Classroom and School Inputs A it 1X it 2 P ijmt 3Tkt A it 1 i k m it Exclusion of peer variables (P-ijmt) Number of peers (class size) and peer characteristics (gender, race, mobility, age) If peer variables are correlated with student and teacher characteristics (Xit and Tkt), omission will produce inconsistent estimates Exclusion of school fixed effects (m) Given that teachers do not frequently change schools, omission of school effects will mean that teacher fixed effects will capture both teacher effects and some of the school effect, leading to inconsistent estimates -- Preliminary, Do Not Quote Without P ermission -- Modeling Student Heterogeneity A it 1X it 2 P ijmt 3Tkt A it 1 i k m it Substituting measured time-invariant student characteristics for student fixed effects

Race/ethnicity, foreign/native born, language parent speak at home, free-lunch status As with teachers, if unmeasured time-invariant student characteristics are correlated with independent variables, will get inconsistent estimates -- Preliminary, Do Not Quote Without P ermission -- Modeling Student Heterogeneity A it 1X it 2 P ijmt 3Tkt A it 1 i k m it Fixed vs. random student effects Fixed effects allow for a separate intercept parameter for each student (equal to the mean error for that student) whereas random effects assume that the student-specific intercepts are drawn from a known distribution (typically normal) Since random effects are part of the error structure, they must be orthogonal to the model variables (Xit, P-ijmt, Tkt) in order to yield consistent estimates Given that fixed effects estimates are always consistent (whether or not unobserved student heterogeneity is correlated with other variables in the model), can test orthogonality assumption by applying a Hausman test Multilevel fixed effects models have been computationally burdensome -- Preliminary, Do Not Quote Without P ermission -- Aggregation A it 1X it 2 P ijmt 3Tkt A it 1 i k m it Measuring characteristics of specific teachers vs. grade-level-within-school averages Since Texas data does not identify specific teacher, work by Rivkin, Hanushek and Kain (2005) relies on average characteristics of teachers within a grade Advantages/Disadvantages of aggregation

Eliminates problems associated with non-random assignment of students to teachers within a school May reduce problem of measurement error since individual errors may cancel out at grade level May upwardly bias estimated impacts of school resources in the presence of omitted variables Tends to reduce precision of estimates -- Preliminary, Do Not Quote Without P ermission -- Data Floridas K-20 Education Data Warehouse Census of all children attending public schools in Florida Student records linked over time Covers 1995/1996 2003/2004 school years Includes student test scores and student demographic data, plus enrollment, attendance, disciplinary actions and participation in special education and limited English proficiency programs Includes all employee records including individual teacher characteristics and means of linking students and teachers to specific classrooms -- Preliminary, Do Not Quote Without P ermission -- Sample for Analysis Middle school students (grades 6-8) who took SSS-NRT (Stanford-9) math test in three consecutive years during 1999/2000 2003/2004 Enrolled in a single math course in the Fall Up to 4 years of achievement gains 4 cohorts of students Includes a variety of math courses, from remedial to advanced and gifted classes Use random sample of 100 middle schools Reduces computational burden of estimating fixed

effects Represents about 12% of middle schools in state -- Preliminary, Do Not Quote Without P ermission -- Value-Added Model Estimates With Varying Degrees of Persistence Explanatory Variable Restricted Value Added =1 Value-Added (Partial Decay) =.8 =.6 =.4 =.2 Contemporaneous =0 0 Years of Experience -2.7495 (0.91) -2.2506 (0.83) -1.7517 (0.72) -1.2529 (0.57) -0.7540 (0.38) -0.2551 (0.14) 1 Year of Experience 1.7714 (0.82) 1.8542 (0.96)

1.9369 (1.11) 2.0197 (1.29) 2.1025 (1.48) 2.1852* (1.66) 2-4 Years of Experience -0.4575 (0.29) -0.1030 (0.07) 0.2515 (0.20) 0.6061 (0.52) 0.9606 (0.91) 1.3152 (1.36) Content In-service Hourst -0.0177 (0.92) -0.0187 (1.08) -0.1097 (1.28) -0.0207 (1.50) -0.0217* (1.74) -0.0227** (1.99) Content In-service Hourst-1 0.0216 (1.07) 0.0183 (1.01)

0.0150 (0.92) 0.0117 (0.80) 0.0084 (0.63) 0.0051 (0.41) Content In-service Hourst-2 0.0517** (2.37) 0.0434** (2.21) 0.0352** (2.00) 0.0269* (1.70) 0.0186 (1.30) 0.0104 (0.78) Content In-service Hourst-3 0.0100 (0.42) 0.0084 (0.39) 0.0068 (0.35) 0.0052 (0.30) 0.0035 (0.23) 0.0019 (0.13) Advanced Degree 0.4133 (0.24) 0.5187 (0.34)

0.6241 (0.45) 0.7295 (0.59) 0.8350 (0.75) 0.9404 (0.93) ______________________________________________________________________________ Student Fixed Effects Teacher Fixed Effects School Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes ______________________________________________________________________________ Number of Students Number of Observations 47,442 74,196 47,442 74,196 47,442 74,196 47,442 74,196 47,442 74,196

47,442 74,196 -- Preliminary, Do Not Quote Without P ______________________________________________________________________________ ermission -- Correlation of Estimated Teacher Effects From Models with Varying Degrees of Persistence Restricted Value Added =1 Value-Added (Partial Decay) =.8 =.6 =.4 =.2 Contemporaneous =0 =1.0 1.0000 =0.8 0.9967 1.0000 =0.6 0.9828 0.9946 1.0000 =0.4 0.9492 0.9717 0.9910

1.0000 =0.2 0.8813 0.9169 0.9534 0.9852 1.0000 =0.0 0.7612 0.8115 0.8679 0.9266 0.9773 1.0000 ______________________________________________________________________________ -- Preliminary, Do Not Quote Without P ermission -- Restricted Value-Added Model Estimates With Differing Controls for Teacher Heterogeneity Time-Invariant Teacher Characteristics Explanatory Variable Teacher Fixed Effects 0 Years of Experience -3.8087*** (3.62) -1.5874 (0.49) 1 Year of Experience

-0.4017 (0.58) 2.4501 (1.07) 2-4 Years of Experience -2.3244*** (3.56) 0.1362 (0.08) Content In-service Hourst -0.0104 (0.91) -0.0153 (0.76) Content In-service Hourst-1 0.0151 (1.32) 0.0210 (1.00) Content In-service Hourst-2 0.0116 (0.93) 0.0511** (2.16) Content In-service Hourst-3 0.0018 (0.11) 0.0127 (0.47) Advanced Degree 1.2414** (2.45) 0.0812 (0.04) Student Fixed Effects Teacher Fixed Effects School Fixed Effects

Yes No Yes Yes Yes Yes Number of Students Number of Observations 45,914 70,437 45,914 70,437 Do Without P -- Preliminary, Not Quote ermission -- Restricted Value-Added Model Estimates With Differing Classroom/School Controls Explanatory Variable Peers,Class Size, School F.E. Included No. Peer Var. No Class Size No School F.E. Peers, Class Size, School F.E. Excluded

0 Years of Experience 1 Year of Experience 2-4 Years of Experience Content In-service Hourst Content In-service Hourst-1 Content In-service Hourst-2 Content In-service Hourst-3 Advanced Degree -2.7495 (0.91) 1.7714 (0.82) -0.4575 (0.29) -0.0177 (0.92) 0.0216 (1.07) 0.0517** (2.37) 0.0100 (0.42) 0.4133 (0.24) -2.6024 (0.86) 1.8920 (0.88) -0.4340 (0.27) -0.0171 (0.89) 0.0190 (0.94) 0.0501** (2.30) 0.0127 (0.53) 0.3020 (0.18) -2.7354 (0.90) 1.7763 (0.83) -0.4518 (0.28) -0.0176 (0.92) 0.0218 (1.09) 0.0521** (2.40) 0.0101 (0.42) 0.3965 (0.23)

-2.2279 (0.75) 1.8644 (0.88) -0.4871 (0.31) -0.0185 (0.98) 0.0209 (1.05) 0.0529** (2.46) 0.0097 (0.41) 0.4384 (0.26) -1.9576 (0.66) 2.1128 (1.01) -0.3328 (0.22) -0.0173 (0.92) 0.0182 (0.92) 0.0516** (2.41) 0.0122 (0.52) 0.3039 (0.18) ______________________________________________________________________________ F-Test on Constraints 2.53** 0.90 0.79 0.83 ______________________________________________________________________________ Student Fixed Effects Teacher Fixed Effects School Fixed Effects Yes Yes Yes Yes Yes Yes Yes

Yes Yes Yes Yes No Yes Yes No ______________________________________________________________________________ Number of Students Number of Observations 47,442 47,442 47,442 74,196 74,196 -- Preliminary, Do Not Quote Without74,196 P 47,442 74,196 47,442 74,196 ______________________________________________________________________________ ermission -- Correlation of Estimated Teacher Effects From Models with Differing Classroom/School Controls Peer Var., Class Size, School F.E. Included No. Peer Var. No Class Size No School F.E. Peer Var., Class Size, School F.E. Excluded Peer Var., Class Size, School F.E.

1.0000 No Peer Variables 0.9540 1.0000 No Class Size Variable 0.9134 0.9355 1.0000 No School Fixed Effects 0.5143 0.5365 0.5050 1.0000 No Peer, No Class Size, No School F.E. 0.5126 0.5318 0.5016 0.9880 1.0000 ______________________________________________________________________________ -- Preliminary, Do Not Quote Without P ermission -- Restricted Value-Added Model Estimates with Differing Controls for Student Heterogeneity Explanatory Variable Time-Invariant Student Characteristics Student Fixed Effects Student

Random Effects Diff. Between Fixed and Random Effects 0 Years of Experience -2.8612*** (2.70) -2.7346 (0.90) -2.8788*** (2.71) 0.1442 (0.07) 1 Year of Experience -1.9354** (2.54) 1.7988 (0.84) -1.9571** (2.56) 3.7560** (2.48) 2-4 Years of Experience -1.8139*** (3.12) -0.4381 (0.28) -1.8316*** (3.15) 1.3936 (1.28) Content In-service Hourst -0.0057 (0.77) -0.0181 (0.95) -0.0064

(0.87) -0.0117 (0.88) Content In-service Hourst-1 -0.0044 (0.58) 0.0215 (1.07) -0.0048 (0.64) 0.0264* (1.85) Content In-service Hourst-2 0.0139* (1.77) 0.0515** (2.36) 0.0131* (1.68) 0.0384** (2.43) Content In-service Hourst-3 -0.0050 (0.58) 0.0102 (0.42) -0.0060 (0.69) 0.0161 (0.96) Advanced Degree 0.1584 (0.24) 0.4168 (0.24) 0.0575 (0.09) 0.3592 (0.30)

______________________________________________________________________________ Student Fixed Effects Teacher Fixed Effects School Fixed Effects No Yes Yes No Yes Yes Yes Yes Yes ______________________________________________________________________________ Number of Students Number of Observations 47,435 74,187 47,435 74,187 47,435 74,187 -- Preliminary, Do Not Quote Without P ______________________________________________________________________________ ermission -- Correlation of Estimated Teacher Effects From Models With Differing Controls for Student Heterogeneity Student Fixed Effects Student Random Effects Time-Invariant Student Characteristics Student Fixed Effects 1.0000 Student Random Effects

0.4514 1.0000 Time-Invariant Student Characteristics 0.3926 0.9498 1.0000 ______________________________________________________________________________ -- Preliminary, Do Not Quote Without P ermission -- Restricted Value-Added Model Estimates -Teacher-Specific vs. Within-School GradeLevel Averages Explanatory Variable Teacher-Specific Characteristics Within-School GradeLevel Average Teacher Characteristics 0 Years of Experience -2.7304 (0.90) -0.0499 (0.01) 1 Year of Experience 1.7819 (0.83) 4.3829 (1.23) 2-4 Years of Experience -0.5060 (0.32) -2.3871 (0.71) Content In-service Hourst -0.0185 (0.96) -0.0454

(0.75) Content In-service Hourst-1 0.0214 (1.06) -0.0761 (1.37) Content In-service Hourst-2 0.0534** (2.44) 0.0251 (0.38) Content In-service Hourst-3 0.0112 (0.47) 0.0679 (0.87) Advanced Degree 0.4881 (0.28) 1.0405 (0.36) Student Fixed Effects Teacher Fixed Effects School Fixed Effects School-by-Year Fixed Effects Grade-by-School Fixed Effects Yes Yes Yes No No Yes No No Yes Yes Number of Students Number of Observations 47,404

74,013 -- Preliminary, Do Not Quote Without P 47,404 74,013 ermission -- Summary of Findings Model Selection Restricted value-added model seems to be a good approximation of the full cumulative model Specification Use of student and teacher fixed effects (rather than covariates) important Random effects may yield inconsistent estimates Important to include school fixed effects, but classroom peer variables relatively unimportant Aggregation to the grade level has some effect, though estimates not radically different from estimates with teacher-level data -- Preliminary, Do Not Quote Without P ermission --

Recently Viewed Presentations

  • CervidTB Stat Pak Use for TB Testing - USDA

    CervidTB Stat Pak Use for TB Testing - USDA

    The CervidTB Stat-Pak® was licensed by the Center for Veterinary Biologics for elk and red deer in 2009 and for white-tailed and fallow deer in 2010-2011. USAHA asked USDA to evaluate this test for use as an official Cervid TB...
  • The Last Meow: Learning from Student Writing

    The Last Meow: Learning from Student Writing

    Clearly, our love affair with our pets has gotten out-of-control. Adapted from Burkhard Bilger, "The Last Meow" (The New Yorker) Explain Bilger's argument and discuss the extent to which you agree or disagree with his analysis. Support your position, providing...
  • Taking Money from Friends and Family; Avoiding the Pitfalls

    Taking Money from Friends and Family; Avoiding the Pitfalls

    Target Old Ladies, Pensions and Thanksgiving dinner companions. Poison the Company and make it unattractive to Future Investors. Not enough stock for management. Give away rights of first refusal or exclusivity. Severely underestimate the cost and time. Leave the numbers...
  • Diapositiva 1 - American Concrete Institute

    Diapositiva 1 - American Concrete Institute

    Meetings with Dick ACI Spring 2005 Convention New York City International Session: "Seismic Engineering for Concrete Structures - Italian Perspective" Workshop OICE - ACI ITALY CHAPTER, Rome, October 2009 Design of new Structures and Retrofitting of Existing ones: the Approach...
  • 2013 Spokane Vendor Training Vendor View of State

    2013 Spokane Vendor Training Vendor View of State

    "To maximize the benefits to the public, state government should be operated in an efficient and effective manner. The Department of Enterprise Services is created to provide centralized leadership in efficiently and cost-effectively managing resources necessary to support the delivery...
  • Attacks Using Stack Buffer Overflow

    Attacks Using Stack Buffer Overflow

    Arial Bitstream Vera Sans Wingdings Symbol Times New Roman Verdana 宋体 Office Theme Attacks Using Stack Buffer Overflow Reference Outline Background Background-Buffer overflow Background-layout of the Virtual Space of a Process Cont. Layout Of Stack Example Slide 10 Slide 11...
  • Dropped Object Awareness Over 130 Members Worldwide Dropped

    Dropped Object Awareness Over 130 Members Worldwide Dropped

    (Change Management) Preventive and Mitigating Controls Preventive Controls PREVENTING AN INCIDENT BY REDUCING THE LIKELIHOOD THAT AN INCIDENT WILL OCCUR Good Planning (Site Inspection TBRA, PTW) Primary Fixings / Secondary Retention / Maintenance Surveys and Inspections Tools and Equipment Aloft...
  • Revolution and Reaction - WordPress.com

    Revolution and Reaction - WordPress.com

    Favored drastic change to extend democracy to all people. ... But Czars were not willing to free the serfs because it would make the nobles angry and they needed the landowners to stay in power. ... Culturally distinct groups resist...