Review and Critique of National Center for 
Evaluation, Standards and Student Testing
(CRESST) Technical Report:

"Issues in Portfolio Assessment: Providing Evidence of Writing
Competency Part I: The Purposes and Processes of Writing"

Home | Up

This 82 page report discusses issues that impact on the validity of the Language Arts "Dimensions of Learning" standard for grading writing portfolios. The Dimensions of Learning standard was set forth by the California Learning Assessment System/Educational Testing Service Research and Development Portfolio Assessment Project.

This report was published in August of 1995. During the time this study took place the CLAS/ETS standards were in an embryonic stage of development. However the methodologies used in this study and the overall approach to the issue of authentic/valid/reliable/consistent large scale assessment of student portfolios is still extremely relevant. This report is a valuable contribution to the areas of how portfolio writing, student selection of pieces to include, rubrics, peer review, student revision and editing, use of resources, etc. effect the assessment process and in turn effect what is actually taught in the classroom. The report attempts to synthesize, compare and contrast the (1) Actual teaching/writing curriculum and approaches in four classrooms in grades 2,4,7-8 and 8 (2) Individual student responses to questions from investigators about their writing and the writing process that are designed to "elicit" the dimensions rubric and (3) Analysis of the student's portfolios.

The study takes a close look at the two fundamental parts of the Dimensions of Learning standard as applied to writing, namely: (1) Writing for Multiple Purposes, Genres, and Audiences and (2) Writing with Resources, Processes, and Reflection. Pages 4-8 provide the complete rubric description. It then analyzes these two parts of the rubric from the point of view of: (1) Opportunities to Learn (what is actually taught in the classroom) and (2) Student's Understandings/Choices for Portfolios, Resources, Processes and Reflection. The report then presents an overall summary and conclusion.

The conclusion stresses the need for sharing of information between teachers, assessment professional and students. According to the report's authors, "Too often the voices of assessment experts, teachers and children are separated- expressing themselves in distant contexts and rarely meeting in conversation…The purpose of this report is to bring these voices together in dialogue that will inform all concerned…to find points of analytical congruence that will help writers learn their craft." The goal is standards that are feasible, productive and developmentally appropriate. There is a need to balance "the vision of student choice as a desirable goal for students with what is needed to (a) benefit their growth as writers and (b) to ensure portfolio raters are provided appropriate evidence." One of the more interesting conclusions of the report was that "Using portfolio scores to measure individual student achievement seems particularly problematic, in that portfolio writing contains writing composed in complex social contexts with highly variable support from teachers, peers and parents." However, the report concludes or implies that a coherent assessment system is possible. The report concludes that teachers and students would benefit from knowing in advance how raters read and assess portfolios. Case studies of individual rater assessments and "think-alouds" may be useful vehicles. Teachers must themselves provide their own think-alouds to model for their students the process of portfolio selection, peer review of writing, etc.

What I found more interesting than the study's conclusions were the recorded observations of teachers and students in the classroom as they attempted to teach and learn writing as a process and adapt to the new methods of portfolio assessment. Of particular interest was the feedback from students based on researchers' questions designed to assess student understanding and how that understanding related to classroom instruction.

Student Feedback

bullet Students who were taught with the more "romantic/personal" philosophy of composition showed less ability to effectively use genre and form. The romantic philosophy stresses writing from what is personally meaningful for the student and not writing to achieve the objectives of a specific rubric or writing to conform to the requirements of a specific genre or type of writing. In general students taught using the romantic approach took more personal ownership of their writing and there was more evidence that their writing was used to construct personal meaning. The children taught with the more classical approach that stressed genre and more structured rubrics, exhibited greater versatility in the use of genres, however their writing showed significantly less personal ownership and the students' responses/peer review tended more towards regurgitation of teacher questions rather than original thought.
bullet Student peer review and revision and editing where difficult to assess in the portfolios because in most cases only the final draft/end product was contained in the portfolio. When students were asked about the writing process itself, in general they showed a poor understanding of revising for content, writing to the needs of a specific audience (other than the teacher herself) and easily confused editing (grammar, spelling corrections, etc.) with continual revision for content. Most students felt that revision was a punishment for ineffective writing, not a way to ferret out and crystallize new ideas. Interestingly, it appeared that many students were being ill served by peer review. Many of the peer responses were overly general or did not address fundamental issues in their writing. The more advanced students, in particular, accurately assessed many of their peer reviews as inadequate and not really aiding them in building their writing skills.
bullet Students, according to the researchers, are more often than naught, unable to make choices on works to include in a portfolio for assessment that dovetail best with the writing rubric used by CLAS/ETS. They required very careful teacher guidance to select the best material.
bullet To underscore further, students had a very poor understanding of the entire need to treat writing as a process. In one case the teacher actually tried to incorporate writing about the writing process and the act of revision itself as part of the grading rubric. For example, points were awarded if students paid close attention to peer response and incorporated the advice into the next draft. This helped only a little in improving student's understanding of the nature and benefits of the revision process. Finally, in one case, to ease the "writing burden" for the students, oral, not written peer review was encouraged.

The report's authors tended to recommend a balance between the classical and romantic approaches to teaching composition in the classroom. However, though all teachers to some degree exhibited "balance" they clearly leaned to one or the other and this was reflected in student's responses and progress. The report's authors felt that whether the instructional approach was classical or romantic would affect the CLAS/ETS assessment, as raters themselves would be teachers who might favor one approach over the other.

Conclusion

The CLAS/ETS assessment for writing and accompanying rubric has a strong impact on what is actually taught in the classroom. Teachers will inevitably help to "prepare" their students to do well on these assessments. Obviously the reverse is true as well -the CLAS/ETS rubrics reflect what is being taught in the classroom. However there is significant variation in what is actually being taught to students and this will effect the validity of the actual test results in assessing individual student performance. In fact, since writing is contextual in nature and the role of the teacher is so pivotal -portfolio assessments are not a truly reliable evaluation of individual student performance as much as an evaluator of teacher /student /class /societal influences on student's writing.

Finally, it is clear from student feedback that they are having great difficulty in achieving the objectives of the CLAS "Dimensions in Learning" rubric for writing. Teachers need to carefully assess whether or not they, as teachers, are merely "wrapping" tried and true "classical" approaches in a "prettier" wrapper or whether they are actually teaching writing as a process for constructing meaning and developing "higher order" thinking skills. Further, a careful assessment must be made of whether and how teaching writing as a process is actually benefiting students. That is, are students able to comprehend the rubric's objectives and goals and become forceful, independent, versatile writers who use writing to construct meaning for themselves and their audience. The study suggests that teachers have a long way to go to actually achieve this. Students, to often seem either to be mimicking the process without real understanding or are attuning their effort more to teacher grades and classroom time constraints than to achieving true understanding. Are we expecting to much or are we simply not teaching well enough and in either case -can we fairly test the end results on a large scale. The study poses the right questions but offers no firm answers. Perhaps there are no answers that can be broadly applied to the diversity of California classrooms.

 

Home • Up • Number the Stars • The Devil in Vienna • Strategies for Readers • Reasons to Teach • Language Acquisition • Bilingual Assessment • Portfolios • Amazon Rainforest & Web Thematic Unit • Teach with the Internet • All Kinds of Minds