Rubric Research Outline

Summary

Here at 80,000 Hours we’re currently working to make a rubric to rate our research/blog quality. My first step was to do a minor literature survey, and outline of which is given below. You can also read and contribute to it on Workflowy.

Main Take-Aways

  • One of the main benefits of rubrics is for writing and learning, using them as a guideline, as opposed to only ranking things that have been done ~c3. As such, a rubric could provide a lot of value to help us decide and find a consistent style of article ~c1i1.
  • Often numbers are less important than written feedback, especially for improvement ~c1i1.
  • It’s really important to have the “students” or “writers” help build the rubric ~c1i1.
  • It’s possible to use rubric to decrease variance in scores. However, it’s also possible that rubrics will increase variance in scores. While there does not seem to be a ton of research in what kinds of rubrics improve precision, it seems like this should be considered and efforts should be made to make sure that the rubric is very clear ~c1i1.

Outline

  • Books
  • Academic Articles
    • The use of scoring rubrics: Reliability, validity and educational consequences
      • Meta study on reviews of metrics for performance assessment, 2007
      • Overview:
        • 1) Rubrics successfully enhance scoring reliability
        • 2) Rubrics improve learning / instruction when known & understood.
      • Conclusions
        • This paper aimed to review empirical research and illuminate the questions of how the use of rubrics can (1) enhance the reliability of scoring, (2) facilitate valid judgment of performance assessments, and (3) give positive educational consequences, such as promoting learning and/or improve instruction.
        • A first conclusion is that the reliable scoring of performance assessments can be enhanced by the use of rubrics. In relation to reliability issues, rubrics should be analytic, topic-specific, and complemented with exemplars and/or rater training. Since performance assessments are more or less open ended per definition, it is not always possible to restrict the assessment format to achieve high levels of reliability without sacrificing the validity.
        • Another conclusion is that rubrics do not facilitate valid judgment of performance assessments per se. Valid assessment could be facilitated by using a more comprehensive framework of validity when validating the rubric, instead of focusing on only one or two aspects of validity. In relation to learning and instruction, consequential validity is an aspect of validity that might need further attention.
        • Furthermore, it has been concluded that rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment. It is thus argued that assessment quality criteria should emphasize dimensions like transparency and fitness for self-assessment to a greater extent than is done through the traditional reliability and validity criteria. This could be achieved through a framework of quality criteria that acknowledges the importance of trustworthiness in assessment as well as supports a more comprehensive view on validity issues (including educational consequences).
    • Quantitative analysis of the rubric as an assessment tool: an empirical study of student peer‐group rating
      • http://www.tandfonline.com/doi/pdf/10.1080/0950069022000038268
      • “Taken together, these data indicate that use of the rubric for self-evaluation affords the students valid assessments of their own performances.”
      • “Research has shown that group work and peer assessment have resulted in increased student comprehension and, ultimately, in higher grades (for a review, see Falchikov 1991, Gatfield 1999).”
    • A review of rubric use in higher education
      • 2010 Meta-study
      • http://class.web.nthu.edu.tw/ezfiles/669/1669/img/1381/6.Areviewofrubricuseinhighereducation.pdf
      • “Two studies suggested that rubric use was associated with improved academic performance, while one did not.”
      • “Studies of the validity of rubrics have shown that clarity and appropriateness of language is a central concern. Studies of rater reliability tend to show that rubrics can lead to a relatively common interpretation of student performance”
      • “Little attention has been paid to the validity of rubrics. Most of the work has been on reliability - a necessary but insufficient condition of validity.
      • “The language used in rubrics is considered to be one of the most challenging aspects of its design (Tierney and Simon 2004; Moni, Beswick, and Moni 2005). As with any form of assessment, the clarity of the language in a rubric is a matter of validity because an ambiguous rubric cannot be accurately or consistently interpreted by instructors, students or scorers (Payne 2003). T”
      • “raters must be sufficiently trained in order to achieve acceptable levels of reliability (typically 70% agreement or higher).”

Blog Efficiency

It took me 3 +- 1 hours to write this, plus another 30 minutes edit it and post it here.

Comments