ENGL110 ASSESSMENT PROJECT 2018-2019
BACKGROUND
English 110 grades suggest that, by and large, students are meeting first-year college writing objectives. Yet, faculty express concern over the quality of students’ writing. To address these concerns, the First-Year Writing Program collaborated with the Office of Institutional Effectiveness (OIE) to answer the following questions:
- How well are students meeting ENGL 110 learning goals?
- How well do students’ ENGL 110 course grades correlate with rubric-based research paper scores?
- How can we best support ENGL 110 students and faculty? (“closing the loop”)
ASSESSMENT MEASURES & METHODS
This assessment of ENGL 110 writing outcomes is based on faculty ratings of students’ ENGL 110 research papers from the fall 2018 semester employing a rubric developed by the First-Year Writing program director and English faculty (see Appendix A). Using a stratified random sample, 55 ENGL 110 sections were selected for participation in the assessment. Three to five students were randomly selected from each section; non-graded papers from these students were collected from instructors and de-identified (to assure a blind assessment) prior to evaluation against the rubric.
ENGL 110 Learning Goals
Dr. Amy Wan (Co-director of First-Year Writing in spring 2018) recruited a team of ten English faculty to formalize the following ENGL 110 learning goals, which are based on the goals established by the National Writing Program Administrators Council and the National Council of Teachers of English:
Students will:
- Produce writing that responds appropriately to a variety of rhetorical situations with a particular focus on academic argumentation.
- Learn reading strategies to summarize, synthesize, analyze, and critique other people’s arguments and ideas fairly.
- Learn research practices that will help strengthen their writing and thinking.
- Produce writing that shows how writers may navigate the diverse processes of composing including revision and collaboration.
- Produce writing that strategically employs appropriate language conventions in different writing situations.
- Take ownership of their work and gain an understanding of their own voice, style, and strengths.
Pathways EC1 Learning Outcomes
All incoming students at Queens College are required to take ENGL 110, which satisfies the English Composition 1 (EC1) general education requirement at Queens College (EC1 is a requirement of the University’s General Education curriculum known as Pathways). To be considered an EC1 course, the curriculum must be designed such that students meet each of the following learning outcomes:
Courses must meet ALL of the following learning outcomes:
- Read and listen critically and analytically, including identifying an argument’s major assumptions and assertions and evaluating its supporting evidence.
- Write clearly and coherently in varied, academic formats (such as formal essays, research papers, and reports) using Standard English and appropriate technology to critique and improve one’s own and others’ texts.
- Demonstrate research skills using appropriate technology, including gathering, evaluating, and synthesizing primary and secondary sources.
- Support a thesis with well-reasoned arguments, and communicate persuasively across a variety of contexts, purposes, audiences, and media.
- Formulate original ideas and relate them to the ideas of others by employing the conventions of ethical attribution and citation.
ENGL 110 Student Work
No single common assignment would allow the reviewing team to assess each of the learning outcomes above for ENGL 110 and Pathways EC1. Since it would be too labor intensive to assess multiple types of assignments, the team selected the common assignment of a research paper, which allowed for the assessment of a selected set of outcomes: ENGL110 learning goals (1), (5) and (6) and Pathways EC1 learning outcomes (2), (4) and (5).
ENGL 110 Assessment Rubric
Dr. Wan coordinated meetings with her team to develop and test a rubric for ENGL 110 student work (Appendix A).
Each rubric criterion is aligned to at least one of the ENGL 110 learning goals or Pathways EC1 learning outcomes assessed by this project:
- Several criteria from Criteria Groups A, B and C align with ENGL 110 learning goal (1).
- Criteria Group E aligns with ENGL 110 learning goal (6).
- Criteria Group F aligns with ENGL 110 learning goal (5).
- Criteria Groups A through D align with Pathways EC1 learning outcomes (2), (4) and (5).
PROJECT IMPLEMENTATION
Dr. Wan notified ENGL 110 instructors about the assessment project prior to implementation to inform them about the goals of the project and the potential request to submit student work. Faculty were informed that only OIE staff members would be privy to the names of faculty participating in the project and were assured that any student work contributed to the project would be de-identified by OIE before faculty rated the papers.
OIE created a stratified random sample of ENGL 110 sections and randomly selected three to five students from each section, whose papers would be solicited for the project. Selected students who had withdrawn from their ENGL 110 class were replaced by another randomly selected student in that section. At the end of the fall 2018 semester, instructors of sampled sections submitted the selected ungraded student papers to OIE for document de-identification and assignment to raters to avoid rater bias. Appendix B shows the communication to faculty.
All identifying information, including student name, instructor name and section, was removed from each paper to ensure a blind assessment. Each paper was assigned a unique identifier to facilitate analysis after the papers were rated.
Dr. Wan’s team of faculty reviewers were each paid a small stipend via CUE funding to compensate them for their time, and each faculty reviewer was given access to a specific online folder that contained a set of student research papers to review and rate. Reviewers were not assigned papers from their own students. The communication to faculty reviewers can be found in Appendix C.
Faculty reviewers scored each paper against the ENGL 110 rubric, assigning a score to each rubric criterion and entered their scores into a Google Form. OIE then downloaded, sorted and analyzed the ratings data. For each of the 19 criteria, papers received a score between 1 and 4, with 1 indicating minimal proficiency, and 4 indicating excellence. A paper’s total score across all 19 criteria ranged between 19 and 76.
LIMITATIONS AND CAVEATS
- Each paper was rated by only one reviewer to limit the amount of work undertaken by reviewers. For future assessment projects, if time and resources allow, papers should be rated by more than one reviewer to improve inter-rater reliability.
- We did not control for tutoring or other types of academic support.
- Both Honors and SEEK sections were included in the study. Since few papers from these types of sections were submitted to the project, differences in ratings cannot be attributed to section type. However, across all section types (general, Honors, and SEEK) students’ strengths and weaknesses, according to the ENGL 110 rubric, remained the same.
RESULTS
Descriptive Statistics
Faculty reviewers rated 121 research papers (79% of papers requested) from 45 sections of ENGL 110 across 40 faculty, representing 42% of fall 2018 ENGL 110 sections and 6% of fall 2018 ENGL 110 students. Reviewers each rated 11 papers.
The descriptive statistics presented below show that the study sample was largely representative of the fall 2018 ENGL 110 population.
Rater Reliability
The average rating received for a criterion was 2.62. Average ratings by reader suggest that, across raters, the rubric was used consistently with the exception of one (Rater 8).
Ratings
The graphs below show the distribution of scores for each criteria group/criterion of the rubric. Each paper received a score of 1, 2, 3, or 4 on each rubric criterion. A score of 1 indicated that the paper either minimally or did not meet the learning outcome; a score of 4 indicated that the paper fully met the learning outcome. Scores of 1 and 2 are depicted in orange at the left-hand side of the graphs. Scores of 3 and 4 are depicted in blue on the right-hand side of the graphs. Each graph shows how the proportion of scores 1 and 2 compare to the proportion of scores 3 and 4 for a criterion or criteria group.
While students are generally meeting outcomes related to language use and fluency of writing (Criteria Groups E and F), the majority of students are not meeting outcomes related to context, idea development, sources, or organization (Criteria Groups A through D).
The most positive results were for Fluency of Writing (Criteria Group E), specifically, for Adherence to disciplinary formatting conventions, Writer-reader engagement, and Effectiveness of diction to produce voice, tone, and style (criteria 13, 16 and 15, respectively):
-
75% of papers adhered to most formatting conventions: title, first page header, double-spaced, 12-size font, 1-inch margins and page numbers;
-
75% of papers mostly or consistently reflected the genuine interest and engagement of the writer;
-
63% of papers adequately or skillfully used diction to produce voice, tone, and style.
The most concerning results were for Content and Idea Development (Criteria Group B), specifically for Ability to sustain and develop an argument:
-
36% of papers adequately or thoroughly developed their central argument;
-
44% of papers presented a thesis that is understandable, focused, and responds to the stated problem;
-
46% of papers mostly framed claims in ways that makes their relevance clear to the essay’s larger argument.
Also of concern are students’ ability to provide context, citation use and coherence between paragraphs (criteria 3, 9 and 12):
-
40% of papers successfully produced insight by deriving and differentiating from the ideas and perspectives of others;
-
43% of papers integrated sources unambiguously and ethically using appropriate convention;
-
47% of papers presented ideas logically such that a reader is never confused about why one paragraph follows another.
These findings echo Dr. Julie George’s 2013-2014 Writing at Queens College-Wide Assessment Project.
In 2013, Dr. George embarked on a project to assess writing in capstone courses.
Table 9 below shows that the learning outcomes with the lowest mean scores were related to genre and content development. Most notable is the result for “Employing logical structures”, which relates to the ENGL110 rubric item “Ability to develop and sustain an argument”.
Table 10 below shows the sample means for Pathways learning outcomes, two of which were assessed by Dr. George’s project: Writing clearly and coherently and Support a thesis with well reasoned arguments. These outcomes saw the lowest mean scores.
Given the findings from the assessment of ENGL 110 and the findings from 2013 study of capstone courses, it is likely safe to assume that the majority of students are graduating without skills related to academic argumentation.
Ratings by Instructor Experience
First-Year Writing program directors were curious as to whether instructor experience influences how well students meet the learning goals of ENGL 110. We found that differences in ratings by instructor experience are very small: Papers from instructors who have taught 1-2 sections of ENGL 110 received an average rating of 2.7, while papers from instructors who taught more than 6 sections of ENGL 110 received an average rating of 2.9. Importantly, across these instructor experience, students’ strengths and weaknesses remain consistent. In students’ weakest area (Criteria Group B. Content and Idea Development), there is little difference in average ratings by instructor experience: the average rating for papers from instructors who taught 1 or 2 sections of ENGL 110 is 2.4 and the average rating for papers from instructors who taught more than 6 sections of ENGL 110 is 2.6.
CONCLUSIONS
Assessment Question 1. How well are students meeting ENGL 110 learning goals?
Overall, students are meeting ENGL 110 learning goals (5) and (6), which are aligned to Rubric Criteria groups E and F:
-
(5) Produce writing that strategically employs appropriate language conventions in different writing situations.
-
(6) Produce writing that demonstrates an understanding of their own voice, style, and strengths.
However, students are not meeting ENGL 110 learning goal (1), which is aligned to Rubric Criteria group B:
-
(1) Produce writing that responds appropriately to a variety of rhetorical situations with a particular focus on academic argumentation.
Additionally, students are not meeting the Pathways EC1 learning outcomes assessed by this project; ratings are lower for the criteria that align with these learning outcomes (Criteria A-D):
-
(2) Write clearly and coherently
-
(4) Support a thesis with well-reasoned arguments, and communicate persuasively
-
(5) Formulate original ideas and relate them to the ideas of others by employing the conventions of ethical attribution and citation.
Assessment Question 2. How well do students’ ENGL 110 course grades correlate with their rubric-based research paper scores?
The scatter plot below shows there is a weak but positive correlation between students’ total scores across all 19 criteria (a range between 19 and 76) and the grades they received in their ENGL 110 classes (grade point value).
Overall, students’ ENGL 110 course grades were more positive than their research paper scores on a writing assessment rubric.
Assessment Question 3. How can we best support ENGL 110 students and faculty?
Dr. Wan and OIE coordinated meetings with the reviewing team, the Center for Teaching and Learning (CTL), and the Academic Support Centers to discuss the findings of the assessment project in Spring 2019. The reviewing team reported that though the findings were not surprising, the collaborative experience they had working on the project together was just as meaningful and important as the assessment results. In pausing to think about how the challenges they face as instructors relate to the assessment findings, the faculty were able to reflect on their experiences, share best practices, and collaborate on possible solutions.
OIE & REVIEWING TEAM RECOMMENDATIONS
-
Facilitate departmental discussions about student writing.
-
Offer professional development opportunities for faculty.
-
Re-assess first year writing in 2-5 years.
FIRST-YEAR WRITING NEXT STEPS
-
Provide a pared down version of the ENGL 110 rubric to ENGL 110 students
-
Provide workshops for faculty on (1) alignment between ENGL 110 learning activities and ENGL 110 learning goals, (2) pedagogical tools, and (3) content development.
-
Assess the final assignment of ENGL 130 (Pathways EC2) using the ENGL 110 rubric.
In Fall 2020, co-directors Dr. Drury and Dr. Williams facilitated workshops for ENGL 110 faculty focusing on discussions around the results of the assessment project.
OIE hopes to facilitate more assessment projects like this in the future – projects that not only assist faculty in evidence-based decision making, but which also foster collaboration, community, and peer learning.
APPENDICES
Appendix A
Appendix B
From the Queens College English Department
Dear XXX,
I write to request your participation in an English Department project to assess College Writing in English 110 courses. Through this project, we aim to determine (1) How well students are meeting ENGL 110 course goals, (2) How the department and college can better support faculty who teach College Writing, and (3) In what ways the curriculum might be improved so that more students learn the skills and practices we’d like them to develop in our English 110 courses.
As part of this project, a team of faculty reviewers will assess students’ written work using a rubric designed and tested for this purpose. Assessment findings will be used to generate thoughtful discussions about pedagogy, professional development for faculty, and workshops for students. If you are interested in contributing to this conversation by serving as a faculty reviewer, please contact Amy Wan.
Students from your ENGL110 section(s) have been randomly selected for this project. Submitting student work is simple. The Office of Institutional Effectiveness (OIE) has created a secure Google repository for instructors to upload electronic copies of student work. Only you and select staff from OIE have access to your folder(s). Student work will be de-identified by OIE staff before faculty reviewers have access to these documents. This way, faculty reviewers will not be able to trace papers back to a student or to an instructor.
NOTE: To access the repository using the link(s) below, you must be logged into Google with your short-form QC email address (xxx@qc.cuny.edu). If you are logged in with another account, you will need to logout of Google first. At the Queens College login screen, enter your QC username (xxx) and the associated password (credentials for QC Mail and MyQC). If you encounter any issues, or if you have questions about the assessment project, please contact Lizandra Friedland at lfriedland@qc.cuny.edu.
By Dec. 15th: Please upload to the folder ENGL110-XXXXX,
(A) The research paper of each student listed below:
XXXX
XXXX
(B) And the first and last drafts of the research paper from:
XXXX
Please note: Student papers must be without teacher commentary. Students need not be notified. And documents may be uploaded to the repository in either Word or PDF format.
Thank you for your participation in this important assessment project! We look forward to sharing what we learn with you.
Kind Regards,
Lizandra Friedland, Asst. Director of Survey Research and Assessment
Amy Wan, Associate Professor and Co-Director of First Year Writing
Karen Weingarten, Associate Professor and Acting Chair
Appendix C
Dear XXX,
Thank you for volunteering as a reader for this project! Each reader has been randomly assigned a set of anonymized student papers to review and rate. Your set of papers is located in this Google Folder:
URL-XXXXX.
This folder contains all of the papers that you are being asked to rate using the rubric attached.
Please rate each paper using the attached rubric, and record your ratings into this Google Form:
URL-XXXXX.
NOTE:
● Submit a form for each paper in your folder.
● Please complete your review and rating of all papers by 2/20/2019.
● To access your folder and the form, you must be logged into Google with “xxx@qc.cuny.edu”.
● If you contributed student papers to the project, other members of the team will be reading them. Raters will not review work from their own students.
When the team is ready to assess outcomes related to editing and revisions, a second set of papers will be added to your folder. These will be assessed against a separate rubric. Look forward to emails from Amy Wan regarding this second part of the project.
Through your efforts, the department will gain a better sense of how well students are meeting ENGL110 and general education College Writing goals, as well as how the department and college can better support faculty who teach English 110. Findings will be analyzed and shared to help you and your team generate thoughtful discussions about pedagogy, professional development, and workshops for students.
Again, thank you for your participation in this important project.
Kind regards,
Lizandra Friedland, Asst. Director of Survey Research and Assessment
Amy Wan, Associate Professor and Co-Director of First Year Writing
Karen Weingarten, Associate Professor and Acting Chair of English