Benjamin Kelcey

Benjamin M Kelcey

Associate Professor of Quantitative Research Methodologies

Teachers College

3311B

CECH Research Methods - 0022

office 638-o

Professional Summary

My research focuses on causal inference and measurement methods within the context of multilevel and multidimensional settings such as classrooms and schools. With respect to causal inference, I develop and apply methods for statistical adjustment that enable researchers to mount focused, specific analogies of their observational studies to randomized experiments with emphasis on multilevel or clustered settings. In this way, I view statistical methods not as substitutes for random assignment but rather as tools to uncover objective information that will lead to well specified and designed randomized studies. My measurement work is also generally situated within multilevel contexts (e.g., measurement of teaching) and develops and applies methods to address the multilevel and multidimensional nature of teaching and learning. Statistically, these research areas unfold within the context of multidimensional, multilevel and cross-classified explanatory item response models, mixed effect models, propensity score methods, and analogues and extensions of each of these. The confluence of these statistical foci take substantive root in understanding the extent to which teacher and school factors can intervene to promote more effective and contextually responsive classroom and school environments. This work has included the measurement and analysis of teachers' knowledge, instructional quality and strategies, and student achievement to identify and develop methods of effective teaching and to establish links among these constructs.

Education

Ph.D., University of Michigan 2009 (Quantitative Research Methods)

M.A., University of Michigan 2006 (Statistics)

B.S., University at Albany 2001 (Mathematics)

Research Support

2018 -2021 National Science Foundation Multisite Designs for Developmental Processes in Mathematics

2016 -2021 National Science Foundation Multilevel Mediation Models to Study the Impact of Teacher Development on Student Achievement in Mathematics

2017 -2021 Institute of Education Sciences Project Coordinate: Increasing Coordination and Use of Evidence-based Practices (with M. Brownell)

2016 -2020 Institute of Education Sciences Exploration of Writing Instruction for Kindergarten Children (with Ying Guo, Allison Breit-Smith, and Chris Swoboda)

2016 -2020 Institute of Education Sciences Optimizing Learning Opportunities for Students' Early Learning Observation System (with Carol Connor)

2016 -2020 Institute of Education Sciences The Day Reconstruction Method: A New Tool for Measuring the Work of K-12 Teachers, and the Contexts of Their Work (with E. Camburn)

2014 -2018 National Science Foundation Power Analyses for Moderator and Mediator Effects in Cluster Randomized Trials (with Jessaca Spybrook & Nianbo Dong)

.

2013 -2014 William T. Grant Foundation Measuring Teaching with Cross-Classified Random Item Effects Item Response Models

2013 -2016 Institute of Education Sciences Exploring Effective Reading Comprehension Instruction: Classroom Practice, Teacher, and Student Characteristics (with Carol Connor and Joanne Carlisle)

2013 -2014 Ohio Education Research Center A data-driven approach to kindergarten readiness and the importance of the preschool years: A partnership between researchers, practitioners, and stakeholders (with Jessica Logan)

2013 -2014 William T. Grant Foundation and the Spencer Foundation Measuring Teaching with Cross-classified Random Item Effects Item Response Models

2012 -2016 National Science Foundation Empirical Benchmarks of Design Parameters for Group Randomized Trials (with Geoffrey Phelps & Nathan Jones)

2012 -2013 National Center for Teacher Effectiveness The Sensitivity of Classroom Observation Assessments of Teacher Quality to Scoring Model (with Heather Hill)

2012 -2016 National Science Foundation Empirical Benchmarks of Design Parameters for Group Randomized Trials Active

2011 -2012 Multilevel Measurement Models for Understanding Teacher Quality (with Brian Rowan)

2010 -2013 Institute of Education Sciences Development of an Interactive, Multimedia Assessment of Teachers' Knowledge of Early Reading (with Joanne Carlisle)

2009 -2012 National Science Foundation Investigating the Measurement and Development of Mathematical Knowledge for Teaching (with Geoffrey Phelps)

2009 -2011 Institute of Education Sciences Assessment of Pedagogical Content Knowledge of Teachers of Reading (with Joanne Carlisle)

2009 -2010 State of Michigan Dept of Energy Mission Possible: Graudation (MPG) King-Chavez Parks (KCP) Initiative (with S. Kanoyton)

2009 -2010 State of Michigan Dept of Energy Measuring Teacher Instruction (with S. Kanoyton)

Investigators: . .

Investigators:...

Investigators:. .

.

2020 -2024 Building Preschool Children's Complex Language Using Informational Texts, Institute of Education Sciences (with Breit-Smith, A & Guo, Y.)

2020 -2023 A Statistical Framework and Tools for Planning Multilevel Randomized Cost-Effectiveness Trials, National Science Foundation (with N. Dong)

Grant: #2020-0565-01 / R305A200283 Investigators:Kelcey, Benjamin 08-01-2020 -07-31-2021 Institute of Education Sciences IES Sub fron NCSU Role:PI $31,277.00 Active Level:Federal

Publications

Peer Reviewed Publications

Dong, N., Spybrook, J., & Kelcey, B. (2020. )Power analyses for moderator effects in with (Non)Random Slopes in cluster randomized trials .Methodology, ,

Kelcey, B., Cox, K., & Dong, N. (2020. )Croon’s Bias-Corrected Factor Score Path Analysis for Small to Moderate Sample Multilevel Structural Equation Models .Organizational Research Methods, ,

Kelcey, B., Spybrook, J., Dong, N., & Bai, F. (2020. )Experimental Power for Cross-Level Mediation in School-Randomized Studies of Teacher Development .Journal of Research on Educational Effectiveness, ,

Kelcey, B., Xie, Y., Dong, N., & Spybrook, J. (2020. )Power and Sample Size Determination for Three-Level Cluster-Randomized Designs Probing Mediation .Multivariate Behavior Research, ,

Kelcey, B., Bai, F. & Xie, Y. (2020. )Statistical Power for Multilevel Mediation Under Partially Nested Designs .Psychotherapy Research Journal, ,

Spybrook, J., Zhang, Q, Kelcey, B., & Dong, N. (2020. )Learning from Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works, For Whom, and Under What Conditions .Educational Evaluation and Policy Analysis, ,

Guo, Y., Kelcey, B., & Puranik, C. (2020. )The Role of Home Literacy Practices in Kindergarten Children’s Early Writing Development: A One-year Longitudinal Study .Early Education and Development, ,

Shen, Z. & Kelcey, B. (2020. )Optimal sample allocation under unequal costs in cluster-randomized trials .Journal of Educational and Behavioral Statistics, ,

Connor, C., Kelcey, B., Sparapani, N., Petscher, Y., Siegal, S., Adams, A., Hwang, J., & Carlisle, J. (2020. )Predicting Second and Third Graders’ Reading Comprehension Gains: Observing Students’ and Classmates Talk During Literacy Instruction using COLT. Scientific Studies of Reading, ,

Kim, Y., Petscher, Y., Uccelli, P., & Kelcey, B. (2020. )Academic Language and Listening Comprehension – Two Sides of the Same Coin? An Empirical Examination of Their Dimensionality, Relations to Reading Comprehension, and Assessment Modality. Journal of Educational Psychology, ,

Kelcey, B., Bai, F., & Xie, Y. (2020. )Micro-Macro and Macro-Micro Effect Estimation in Small Scale Latent Variable Models with Croon's Method .Testing, Psychometrics & Methodology in Applied Psychology, ,

Jensen, B., LeBaron, T., Covay-Minor, E., Davis, D., Dietiker, L., Gabriel, R., Halpin, P., Kelcey, B., Rui, N., & Steinberg, M. (2019. )Complexity and Scale in Research on Teaching Effectiveness .

Kelleher, M., Kinnear, B., Sall, D., Schumacher, D., Schauer, D., Warm, E., & Kelcey, B. (2019. )A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments .Academic Medicine, ,

Dong, N. & Kelcey, B. (2019. )A Review of Causality in a Social World: Moderation, Mediation and Spill-over .Journal of Educational and Behavioral Statistics, ,

Cox, K. & Kelcey, B. (2019. )Robustness of Statistical Power in Group-Randomized Studies of Mediation Under an Optimal Sampling Framework .Methodology, ,

Cox, K. & Kelcey, B. (2019. )Optimal Design of Multisite and Cluster-Randomized Studies Using Fallible Outcome Measures .Evaluation Review, ,

Kelcey, B. (2019. )A Robust Alternative Estimator for Small to Moderate Sample Structural Equation Models .Addictive Behaviors, ,

Cox, K. & Kelcey, B. (2019. )Optimal Sample Allocation in Group-randomized Studies of Multilevel Mediation with a Group-level Mediator .Journal of Experimental Education, ,

Kelcey, B. & Shen, Z. (2019. )Strategies for Efficient Experimental Design in Studies Probing 2-1-1 Mediation .Journal of Experimental Education, ,

Kelcey, B., Hill, H., & Chin, M. (2019. )Teacher Mathematical Knowledge, Instructional Quality, and Student Outcomes: A Multilevel Mediation Analysis .School Effectiveness & School Improvement, ,

Kelcey, B., Spybrook, J., & Dong, N. (2018. )Sample Size Planning in Cluster-randomized Studies of Multilevel Mediation .Prevention Science, ,

Kelcey, B. & Shen, Z. (2017. )Planning community-based assessments of HIV educational intervention programs in sub-Saharan Africa .Health Education Research, ,

Kelcey, B. & Shen, Z. (2017. )Experimental evaluations of school-based HIV programs in sub-Saharan Africa .Studies in Educational Evaluation, ,

Zaidi, Nikki L Bibler; Swoboda, Christopher M; Kelcey, Ben; Manuel, R Stephen (2017. )Hidden item variance in multiple mini-interview scores. Advances in health sciences education : theory and practice, ,22 (2 ),337-363

Kelcey, B., Dong, N., Spybrook, J. & Shen, Z. (2017. )Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators .Multivariate Behavioral Research, ,

Kelcey,B. Dong, N., Spybrook, J. & Cox, K. (2017. )Statistical Power for Causally-defined Indirect Effects in Group-randomized Trials with Individual-level Mediators .Journal of Educational and Behavioral Statistics, ,

Shen, Z., Escue-Simon, C., & Kelcey, B. (2017. )The potential consequences of using value-added models to evaluate teachers .eJournal of Education Policy, ,

Kelcey, B., Phelps, G., Spybrook, J., Jones, N., & Zhang, J. (2017. )Designing Large-scale Multisite and Cluster-randomized Studies of Professional Development .Journal of Experimental Education, ,

Dong, N., Kelcey, B., & Spybrook, J. (2017. )Power analyses for moderator effects in three-level cluster randomized trials .Journal of Experimental Education, ,

Wallace, T., Kelcey, B., & Ruzek, E. (2016. )What Can Student Perception Surveys Tell Us About Teaching?: Empirically Testing Theoretically-Proposed Dimensions of Teaching .American Educational Research Journal, ,

Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016. )Performance Trajectories of Teacher Knowledge Growth Throughout Professional Development Programs. Evaluation Review, ,

Spybrook, J., Kelcey, B., & Dong, N. (2016. )Power Analyses for Detecting Treatment by Moderator Effects in Cluster Randomized Trials .Journal of Educational and Behavioral Statistics, ,

Kelcey, B, Shen, Z., Spybrook, J. (2016. )Intraclass Correlation Coefficients for Designing School Randomized Trials in Education in Sub-Saharan Africa .Evaluation Review, ,

Spybrook, J. & Kelcey, B. (2016. )Introduction to Three Special Issues on Design Parameters Values for Planning Cluster Randomized Trials in the Social Sciences .Evaluation Review, ,40 ,491-499

Kelcey, B. & Shen, Z. (2016. )Multilevel Design of School Effectiveness Studies in Sub-Saharan Africa .School Effectiveness and School Improvement, ,

Spybrook, J., & Shi, R., & Kelcey, B. (2016. )Progress in the Past Decade: An Examination of the Precision of Cluster Randomized Trials Funded by the U.S. Institute of Education Sciences .International Journal of Research & Method in Education, ,

Dwyer, J. Kelcey, B., Berebitsky, D., & Carlisle, J (2016. )A Study of Teachers’ Discourse Moves That Support Text-based Discussions .Elementary School Journal., ,

Adelson, J. & Kelcey, B. (2016. )Bringing a perspective from outside the field: A commentary on Davis et al.’s (2010) use of a modified regression discontinuity design to evaluate a gifted program. Journal of Advanced Academics, ,

Kelcey,B. (2014. )Measuring Teachers’ Instruction with Multilevel Item Response Theory. International Journal of Research in Education Methodology, ,

Moss, B., Kelcey, B., & Showers, N. (2014. )Classrooms as Moderators of Developmental Education Effectiveness and Student Achievement .Community College Review, ,42 ,201-220

Kelcey, B. & Swoboda, C. (2014. )Prognostic Scores in Clustered Settings .In Propensity Score Analysis: Fundamentals, Developments, and Extensions Edited by Wei Pan & Haiyan Bai, ,

Kelcey, B., & Phelps, G. (2014. )Strategies for Improving Power in School Randomized Studies of Professional Development. .Evaluation Review, ,37 ,520-554

Kelcey, B., McGinn, D., & Hill, H. (2014. )Approximate Measurement Invariance in Cross-classified Rater-mediated Assessments. Frontiers in Quantitative Psychology and Measurement, ,

Carlisle, J., Kelcey, B., & Berebitsky, D. (2013. )Teachers' Support of Students' Vocabulary Learning During Literacy Instruction in High Poverty Elementary Schools .American Educational Research Journal, ,

Kelcey, B., & Carlisle, J. (2013. )Learning About Teachers’ Literacy Instruction From Classroom Observations .Reading Research Quarterly, 48, 301-317, ,

Kelcey, B., & Phelps, G. (2013. )Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes .Educational Evaluation and Policy Analysis, ,35 ,370-390

Frank, K., Maurolis, S., Duong, M., & Kelcey, B. (2013. )What would it take to Change an Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences .Educational Evaluation and Policy Analysis, 35, 437-460, ,

Carlisle, J., Kelcey, B., Rosaen, C., Phelps, G., & Vereb, A. (2013. )A Framework for Analysis of Case Studies of Reading Lessons .Journal of Education and Training Studies , ,224-238

Kelcey, B. (2011. )Assessing the Effects of Teachers’ Reading Knowledge on Students’ Achievement Using Multilevel Propensity Score Stratification .Educational Evaluation and Policy Analysis, ,33 ,458-482

Kelcey, B. (2011. )Covariate Selection in Propensity Scores Using Outcome Proxies .Multivariate Behavioral Research, ,46 ,453-476

Carlisle, J., Kelcey, B., Berebitsky, D., & Phelps, G. (2011. )Embracing the Complexity of Instruction: A Study of the Effects of Teachers' Instruction on Students' Reading Comprehension .Scientific Studies of Reading, ,15 ,409-439

Carlisle, J., Kelcey, B., Rowan, B., & Phelps, G. (2011. )Teachers' Knowledge About Early Reading: Effects on Students' Gains in Reading Achievement .Journal for Research on Educational Effectiveness, ,4 ,289-321

Songer, N. B., Kelcey, B., & Gotwals, A. (2009. )Empirically Driven Development of a Learning Progression Focused in Complex Reasoning about Biodiversity .Journal of Research in Science Teaching, 46, 610-631, ,

Cox, K. & Kelcey, B. (in review) A Partial Posterior Predictive Distribution Test for Multilevel Mediation .Journal of Modern Applied Statistical Methods, ,

Bulus, M., Dong, N., Kelcey, B., & Spybrook, J. (in development) Designing cluster-randomized trials to detect the main, moderation and mediation effects in the R environment: An exposition via PowerUpR library with examples from STEM education .Journal of Statistical Software, ,

Spybrook, J., Kelcey, B., & Dong, N. (in review) Statistical Power in Multilevel Models. In Multilevel Modeling Methods with Introductory and Advanced Applications edited by Ann O’Connell, Betsy McCoach, Bethany Bell .

Cox, K. & Kelcey, B. (in review) Comparing Estimation Approaches for Multilevel Structural Equation Models with Limited Sample Sizes and Model Misspecifications .Methodology, ,

Other Publications

PowerUpR Shiny App for Designing Multilevel Studies Probing Mediation .

PowerUpR Shiny App for Designing Multilevel Studies .

PowerUpR Shiny App for Designing Partially Nested Studies Probing Mediation. .

Presentations

Paper Presentations

Kelcey, B. (2014. )Noninvariant Measurement in Rater-Mediated Assessments of Teaching Quality .

Kelcey, B. (2014. )Power Analysis for Cross Level Mediation in Cluster Randomized Trials .

Kelcey, B. (2014. )Approximate Measurement Invariance in Rater‐mediated Assessments: Measuring Teaching Quality with Classroom Observations .

Kelcey,B., Phelps, G., & Spybrook, J. (2014. )School Randomized Designs of Professional Development using Teacher Knowledge Outcomes .

Kelcey, B. & Spybrook, J. (2014. )Designing Adequately Powered Cluster Randomized Trials using Optimal Design Plus .

Spybrook, J. & Kelcey, B. (2014. )Power Calculations for Binary Moderators in Cluster Randomized Trials .

Kelcey, B. (2014. )Noninvariant Measurement in Rater-mediated Assessments .

Kelcey, B. McGinn, D., Hill, H., & Charalambous, C. (2014. )Dimensionality and Generalizability of the Mathematical Quality of Instruction Instrument .

Kelcey, B., & Carlisle, J. (2014. )Teachers’ Knowledge about Teaching Reading, Their Instructional Practices, and Their Students’ Reading Achievement: Evidence from Quantile Mediation .

Carlisle, J., Connor, C., Kelcey, B., Petscher,Y., Sparapani, N., Ingebrand, S. (2014. )Development of an Observation System: Creating Opportunities to Learn From Text .

Kelcey, B., Frank, K., & Seltzer, M. (2014. )Parametric Sensitivity Analysis Thresholds for Mediation Effects .

McGinn, D., Kelcey, B., Hill, H., & Chin, M. (2014. )Using Item Response Theory to Learn about Observational Instruments .

Kelcey, B. (2014. )Multilevel Mediation: Design and Analysis .

Shen, Z. & Kelcey, B. (2014. )Power Analysis for Multilevel Mediation .

Swoboda, C., Kelcey, B., & Zhang, J. (2014. )Estimation of cross-classified random effects with missing data using multiple imputation .

Shen, Z. & Kelcey, B. (2014. )Power Analysis for Cross-level Mediation .

Kelcey, B. & Phelps, G. (2013. )Parameters for the Design of Group Randomized Studies for Teacher Professional Development .

Kelcey, B. McGinn, D., & Hill, H. (2013. )Approximate Measurement Invariance in Rater-mediated Assessments: A Random Item Effects Item Response Model for Measuring Teaching Quality with Classroom Observations .

Kelcey, B. McGinn, D., & Hill, H. (2013. )Measurement of Classroom Teaching Quality with Item Response Theory .

Kelcey, B. McGinn, D., & Hill, H. (2013. )Measurement Issues in Teacher Observations and the Estimation of Teacher Effects .

Spybrook, J., Westine, C., Taylor, J., & Kelcey, B. (2013. )Designing Adequately Powered Cluster Randomized Trials Using Optimal Design Plus .

Kelcey, B. (2013. )Introduction to the Measures of Effective Teaching Longitudinal Database .

Kelcey, B. (2013. )Psychometric Models for Measuring Individualized Instruction .

Kelcey, B. (2013. )Propensity Score Matching within Prognostic Strata .

Kelcey, B. (2012. )Propensity Score Matching within Prognostic Strata .

Kelcey, B., & Carlisle, J. (2012. )Patterns of Instruction in Early Reading Lessons .

Kelcey, B., & Carlisle, J. (2012. )A Multilevel Bifactor Framework for the Measurement of Multidimensional Instruction .

Frank, K., Maurolis, S., Duong, M., & Kelcey, B. (2012. )A Quality Threshold for Statistical Controls in Observational Studies .

Moss, B., Kelcey, B., & Showers, N. (2012. )Classrooms as Moderators of Developmental Education Effectiveness and Student Achievement .

Kelcey, B. (2012. )A Multilevel Bifactor Framework for the Measurement of Teaching Across Subject Areas .

Kelcey, B. (2011. )Inferences on Instructional Practice Measured With Multiple Sources of Variation .

Kelcey, B. (2011. )Measurement with Cross-classified Item Response Models .

Kelcey, B. (2011. )The Threshold of Embedded M Collider Bias and Confounding Bias .

Kelcey, B. (2011. )Matching Across Schools with Multilevel Propensity Scores .

Gersh, A. & Kelcey, B. (2011. )Improving New Teacher Effectiveness: An Analysis of Collective Teacher Knowledge and Possible Contextual Effects .

Carlisle, J., Kelcey, B., & Berebitsky, D. (2011. )Teachers' Vocabulary Instruction and Word Meaning Interactions in High Poverty Schools .

Kelcey, B. (2010. )Applications of Multilevel Propensity Scores .

Kelcey, B. (2010. )A Multivariate, Multilevel Rasch Model for Characterizing Observed Instruction .

Kelcey, B., & Carlisle, J. (2010. )A Triangulation of Teachers' Effectiveness via Value Added Models, Teacher Knowledge and Observed Instruction .

Kelcey, B. (2010. )A Multivariate, Multilevel Rasch Model for Characterizing Teaching and Learning .

Kelcey, B. (2009. )Propensity Score Variable Selection and Model Type in Multilevel Settings in which Treatment Assignment Varies by Schools .

Carlisle, J., Kelcey, B., Phelps, G., Johnson, D., & Berebitsky, D. (2009. )Do teachers’ instructional practices during reading comprehension lessons contribute to their students’ progress in reading? .

Frank, K., Duong, M., Maurolis, S., & Kelcey, B. (2009. )The Robustness of Inferences from Quasi- and Randomized Experiments .

Songer, N. B., Kelcey, B., & Gotwals, A. (2009. )How and When Does Complex Reasoning Occur? Empirically Driven Development of a Learning Progression Focused on Complex Reasoning about Biodiversity .

Swoboda, C. & Kelcey, B. (2014. )Combining propensity and prognostic scores in clustered settings .

Kelcey, B. (2015. )Degenerate Power in Multilevel Mediation: The Non-monotonic Relationship Between Power & Effect Size .

Kelcey, B., Spybrook, J., Zhang, J., Phelps, G., & Jones, N. (2015. )Strategies for Improving Power in Cluster Randomized Studies of Professional Development .

Contact Information

office 638-o