Writing exams: Paper vs Online

Up until a few years ago, our BC Social Studies 11 students wrote a provincial examination to wrap up their course.  This was a standardized test featuring 55 multiple choice questions and two essay questions.  The exam was the same for all students in the province and was sometimes used by the school, district, or province to provide data on how students were doing.  Teachers occasionally used the exam results to give them feedback on whether what they are teaching has had the desired outcome.  In many cases the exam was ignored.  Not precisely a high-stakes exam (although it was worth 40% of the students' overall mark), although for some teachers it drove their course planning and was often seen as a barrier to more creative ways to teacher the course.  In my opinion it was a reasonable assessment; it drew from all areas of the course curriculum -- 20th century Canadian History, Human Development and Environmental Issues, Canadian Politics and Government -- and included a balance of straight-forward and higher order questions.  The exam also assured that students across BC more-or-less got an introduction to common topics of citizenship and Canadian identity, the state of the developed vs developing world, why it was worth voting (and who the parties were), and about Canada's involvement in world affairs. It was perhaps too focused on content and less on broader thinking concepts and subject-specific skills (other than interpreting population pyramids).  The essay questions could be hard for students, but they really showed whether students could synthesize learning from a big chunk of the course, and also whether they could write at a level that could be expected from a Grade 11 student.

In January 2013, a group of Social Studies teachers in Prince George conducted an informal experiment to compare the results of students who wrote their provincial exams using either the paper format or an online format in a computer lab. Who would do better on the written section?

BACKGROUND
At the first school, the teachers insisted on paper copies of the exam.  At a second school, they decided to try having all of the students write online.  The written section on the exam -- an essay on a historical topic and a second essay on a topic related to human geography or the environment -- is marked by teachers.  Our schools have similar demographics, the exam is the same, and the teachers who taught the course have similar styles and roughly the same attention to content, division of curriculum, and review strategies.  We did notice that the school with the online writers did not seem to emphasize human geography to the same extent as the other school, and this showed up in the responses to the second essay.  There were two classes in each school writing the exam, so we had about 50 exams at each site (thus 100 essays) to provide data.  The students did not get to choose paper vs online, so this perhaps removed the element of preferred styles and comfort-based selection.  Students with special adaptations required (e.g. scribes) wrote the exam in a separate sitting and were not included in our experiment.  The exam session is 2 hours, although most students require and hour or a little more to complete it, so "exam" fatigue" is rarely a concern.  The essays are all scored with the same 6-point grading rubric (see image above), and the teachers marked the exams together with two teachers marking each paper and agreeing on a score.  The discussion of results and conclusions (this blog post) are the result of the conversation between one of the markers from each school -- myself and a colleague. We participated in the marking but did not actually teach any of the classes involved.

RESULTS
While we weren't completely impressed by their achievement, the Paper group won this contest with ease. There was a higher overall average score, with less 0s and 1s and almost no NRs. The student students provided more detail, used more complex sentences, and had fewer lapses with grammar and punctuation. Interestingly, they related more "stories" from class; that is, more anecdotes that sounded like direct quotes from the teacher (for better or worse) or lines of thinking that were the result of activities that likely involved writing or speaking in class (as opposed to something studied before the exam).  They also had more repetition -- cycling back through an idea to fill the space.

The online writers had a lower average, with considerably more NRs, 0s, and 1s, and no 6s. They had shorter sentences and paragraphs, and used more informal grammar and less punctuation. These students had a higher prevalence of poor diction (word choice or vocabulary) but the syntax was fine (arrangement of words and phrases). We concluded that they knew most of the same facts and possessed similar opinions as the paper group,  but simply referenced them without expansion, kind of a "I know this stuff -- just read my mind" approach.  

DISCUSSION
With all of their writing contained in a textbox, then online writers had no annotation of their text, no circling or evidence of revision (eraser marks) or any other evidence of the "struggle" to capture their thoughts. We admitted that we felt a bias about this -- writing that came from (and had) a "personality" seemed to be more authentic than the digital text. We also found the digital essays easier to mark -- without the "personality," e.g. the peculiarity handwriting (particularly neat vs messy) we spent less time second guessing whether we were assessing a visual quality that was not necessarily tied to their level of understanding. With digital writers, it was simply a matter of how does this piece of writing place on the rubric?

Our interpretation of these results was that students writing essays online fall into a pattern of digital communication that is informal, truncated, and full of insinuation rather than exposition. Their writing has a quality of expedience and we imagined they were written much faster than their paper counterparts. The students writing essays on paper exhibited more care and attention to their work, but also included more material meant to fill up the page.  Perhaps the online writers had no expectation of how "big" their essay should look on the screen, whereas the paper writers looked at three lined pages for each essay and had a feeling that they should at least get to page two before wrapping it up.

We agreed that students are very comfortable writing in digital spaces, but this does not necessarily serve them well for formal tasks such as essay writing.  This conclusion goes against what many experts would suggest -- even based on our own experience, it would seem that the digital format would serve one better: it is easy to go back, fix errors, cut & paste from one section to another for a better flow, change one's mind about various parts including paragraph and so on.  But this is our adult sensibility, we are teachers who have spent years writing, first on paper and later with the "magic" of computers and word processors.  For some of our students, using a word processor is like accessing a heritage skill.  They are more finely attuned to the gestured inputs of digital devices, and are slower to type and less likely to take advantage of editing tools than the generation of students who used computers but did not have smartphones.

We carried these observations forward to other schools in 2013 and resisted them after subsequent SS11 provincial exams; our conclusion were generally reinforced by what we heard from every school.  The school that used the online exam in our experiment in 2013 started giving students a choice about online vs paper, and they also put a renewed emphasis on writing skills in their Social Studies courses.  These particular provincial exams are no longer with us, but I thought this would be an interesting set of thoughts to consider as we anticipate a new round of standardized tests in BC -- the upcoming Grade 10 numeracy assessment and the Grade 10 and 12 literacy assessments.

Collaborating for Inclusion

This is a story about the integration of Outdoor Ed and Social Studies, but also a story about what inclusion looks like.

Ian Leitch is teacher in Prince George and a member of the Pacific Slope Consortium (as am I!). Over the last few years, we have been working on a project we've called TTSP -- you can read about that here.

Many of Ian's contributions to the Pacific Slope and dialogue with TTSP members revolve around his ongoing efforts to integrate outdoor and experiential learning and identity-building curriculum with Social Studies and Outdoor Ed classes. This has translated into unique courses, such as a version of Social Studies 11 Explorations that centers on the Canoe in Canadian history, geography, and culture, with student learning about traditional ecological knowledge and experimenting with "pioneer" skills.

These course trials have led to powerful learning at his school. His wilderness expeditions are legendary, and have taught students about their own limits and strengths, and their capacity for inclusion.

​In 2017, Ian invited Miranda, a remarkable student who lives with severe Cerebral Palsy and spends most of her day in a wheelchair, to join his Outdoor Ed class. With community resources, family and classmates' support, Miranda was able to participate, right up to setting out on a canoe expedition. On a second outing, Miranda had another first -- sitting by a fire and carving wood. This class experience allowed her to act on her love of nature.

Aidan, a young man living with challenging autism, was able to construct his own shelter and spend a – 20 degree winter night under the stars. He did so because the place was made safe for him by his teacher and classmates. One of the key learning outcomes for this course was how to co-design field excursions to involve every student. For Ian's class, inclusion is a team effort.

 

Photos (Ian Leitch) & names, and stories shared with student & parent permission

Introducing the Capacities

Or, How to Think Like a River.

In British Columbia, we are about 6 years into 8+ year process to implement new curriculum in our K-12 schools.  The "redesigned curriculum" seeks to make some of the implicit goals of education -- communication, critical and creative thinking, personal and social responsibility -- explicit by identifying them and calling them core competencies.

Within each course, the curriculum is framed by Big Ideas. Curricular Competencies, and Content.  Depending on who you talk to, or which Ministry of Education document you read, or video clip from an expert you watch, these frames (the whole new curriculum really) are quite fluid...

• Big Ideas can be rolled in or out, curricular competencies can be swapped out for others, and the content is merely a suggestion.
• There are no standard assessments to measure success.
• There are multiple opportunities to embed indigenous perspectives, but no detailed prescriptions for what this should look like.
• There are entry points for community and place-responsive education, and a greater emphasis on holistic interdisciplinary learning.
• There are implications for pedagogy, but no actual dictates about what that looks like or what paradigms should guide the "new teacher."
• Having common course outlines within schools will be elusive -- choice and flexibility is where it's at.
• The implementation was underfunded and lacked clarity, and the whole process has had political undertones related to government funding and control of educational agendas (i.e. as opposed to teachers' agendas).
• Parts of the process were too slow or experienced punishing delays (e.g. piloting Grade 10 curriculum for three years in a row).
• Some teachers are organizing their course and assessment using the Big Ideas, others are using the curricular competencies for this purpose, while others are sticking with content to structure units and guide assessment.
• Most teachers will, of course, pay attention to all three and aim at some kind of synthesis.

For better or worse, this is the plan for the next long while in BC. I am still rather excited to be part of this change, or at least parts of it, but very much aware of its shortcomings, its unintended consequences, and the challenges faced by teachers in making sense of it. I often work with new teachers, both in the local UNBC teacher training program and early career teachers in my school district. These are the ones who are thought to have been "trained in the new curriculum" but in reality they are more uncertain than the "vets" about what it all means. They realize that the "fluid curriculum" gives them creative reach and freedom to experiment, but wow would they ever like some modelling and guidelines.

Speaking of fluidity, I have given some thought to how to assess students in the this brave new curricular world. It is a competency-based system, and yet assessing competencies on their own is great for formative work (try, evaluate, reflect, revise, re-try) but not so great for summative (final standing, marks, and advancement).  For example, I don't think we want to start having report cards that say Johnny got an A in establishing significance but a C- in perspective taking. The curricular competencies work well for individual tasks, for taking apart problems and developing skills.

In the study of stream dynamics, we have a couple of terms to describe how rivers move sediment -- competence and capacity. Steam competence refers to the size of particles that can be carried; the higher the competence, the bigger the particle size. This is mainly the job of young rivers in steep terrain, rolling and dragging big stones and carving away at the valley walls. Stream capacity refers to the total volume of sediment that can be carried.  Rivers with high capacity have already seen the big particles broken down, and carry a big load of sediment in suspension and solution, and lay it down beside the river or cary it out to sea.

I love these terms as a metaphor for what happens in classrooms. At first we take on big problems, one by one, and start to see how it gets easier when the problems start coming apart with small, repeated tasks and strategies. We erode the barriers, and build a unique channel through a challenging landscape. Later, we have the capacity to make broad connections between problems, to communicate what we have done, and take responsibility for the impact of our knowledge and understanding. This process is cyclical, happening over and over again in a class, in a course, and in K-12 education. Thus, I am interested in using my classes to develop both student competence (an explicit goal of the curriculum) and capacity (my understanding of an implicit goal of the curriculum).  While any individual task may practice and assess competence, when it comes to overall marking categories and summative assessment, as well as readiness to advance to the next grade, my focus is on capacity. For the context of Social Studies,  I have settled on four: Foundations -- this is knowledge and understanding of core content, Skills -- both hard and soft, like the ability to read maps, determine bias in sources, or organize an argument, Thinking -- application of concepts (mainly the curricular competencies) and cognitive skills to problems of history and geography, and Connections -- inquiry, synthesis, and activation of learning. Here is the framework I use to position "The Capacities" within the new Social Studies curriculum (and here is the pdf link):

This is my latest stab at trying to reconcile the various parts of the hidden and revealed aspects of the curriculum, and to provide a topography for student assessment. As always, feedback welcome via @gthielmann, in a comment below, or by email gthielmann (AT) gmail (DOT) com.

Teachers as Advocates for Public Education

Like many school districts, the one in which I work has policies on employee rights & responsibilities that includes language on how teachers can speak out:

3.14 is an interesting one and the focus of my post. It has been used as a bugbear by both employees and the employer to suggest that teachers have a gag order on public commentary and should avoid bringing concerns about the education system to the media. This sentiment was not the intention behind the policy nor should it be the basis of interpretation. This was made clear during a policy revision process in 2016-17 that included Policy 1170.3. Both sides affirmed the idea that employees are welcome to use reasonable tools to raise awareness and make improvements to public education.  The employer did not dispute the almost ubiquitous association between responsible use of social media and public commentary. The Prince George Teachers (PGDTA) lobbied to bring changes to this policy, to affirm the idea that healthy, responsible, public comment by teachers was both of benefit to the education system and also respectful of tradition and diversity, not to mention Charter rights. Alas, the feedback and proposed changes from the PGDTA were rejected by the trustees on the advice of senior management.

The wording of this policy as it stands does, however, provide a starting point for a reasonable interpretation of the extent to which teachers can use public commentary to advocate for a quality education system.

The key word here is "irresponsible" -- e.g. a public comment making a critique personal (e.g. directed toward a district employee), or using false information, inflammatory language, breech of confidentiality, slander, careless generalizations, etc. "Responsible" comment should be welcome. There is no gag order (or Charter or Rights exception) on teachers using media to improve the education system, nor does a critique of a particular policy or initiative undermine the public education system -- it is intended to bring about effective change.

We have ample precedents of practising teachers in Prince George and BC responsibly offering public comment in order to identify educational issues and suggest solutions. I have done this myself many times since 2010 on radio, newspaper, TV, public board meetings, and on social media. I have spoken out on technology plans, school closures, program implementation, district budget cuts and budgeting process, superfluous spending, management of student data, student data security, school achievement contracts, asset disposal, collaboration models, labour negotiation, bad faith bargaining, Labour Relations Rulings (one of my faves), school ground greening, school repair, response to mental health services, and many aspects of leadership and district planning process.

As a rule of thumb, the more the public comment is directed towards general trends and local or provincial phenomena that we can all own as a education system, the better. "Public commentary" should be about "public issues" and not about private concerns and personal grudges. A good example is the current revision to the curriculum in BC -- hundreds of BC teachers have chimed in via twitter, facebook, and blogs about how this process has unfolded and about the mistakes made along the way. These folks are not just complaining or hanging on to the past, they consistently offer solutions or alternatives, and speak from a position of authority and/or experience.

There are special circumstances that call for actual whistleblowing, e.g. a response to corruption, gross negligence, child protection, workplace harassment, or serious safety violations.  Sadly, our district policies do not include protection for whistleblowers, and no doubt there have been issues in the past that were allowed to fester unchecked because no one wanted to confront the problems publicly.  I believe that situations calling for whistleblowing should be dealt with separately from Policy 1170.3 -- advocacy for public education is one thing, coming forward with an allegation of harassment or a dangerous workplace hazard is another.

Of course, many teachers are comfortable leaving advocacy on serious or sensitive issues to their union leadership -- that's ok. Full-time released union officers are not bound by Policy 1170.3 or its equivalent elsewhere, and often do spend some of their time preparing briefs for the media and engaging in public comment through social media and public presentations to local school boards. We can rely on the BCTF, too, for providing public education advocacy -- they have devoted considerable resources to this end and have made it a core value of the organization.

Ordinary teachers, however, should be comfortable with the idea that part of their responsibility as a professional is to be an advocate for public education. This means that they should not shy away from responsible public comment and, where appropriate, political action and social justice projects. Our job does not end in the classroom, although that is the most important part. Our job includes diverse roles within overlapping educational communities on many scales, from the classroom all the way through to international solidarity. In between are school, district, and provincial issues that absolutely require both private and public comment from teachers. Thankfully, each of us does not have to attend to all issues at all scales -- do what you are good at, do it when you are ready, and do it responsibly.

New job

After 23 years of teaching Social Studies, English, Geography, and other fun stuff, sometimes with various pull-outs blocks for leadership or support and coordination, I'll be out of the classroom next year, for at least one year.

I will be splitting my time between Pro-D Coordination (managing the local teacher Pro-D fund and organizing Pro-D events and conferences) and Curriculum Support for senior Humanities teachers.  The first job I've had for 5 years, but the second one is new for me.  I'm thrilled to start this -- in some ways it looks like the work I've done off the side of my desk for years, but it will also involve some new roles.  Here is the concept map that I used to prepare for the interview:

I would like to focus some of this "Curriculum Coach" time on our early career and new assignment teachers as they grow into their roles, even those who may not have senior courses next year.  I will be available for mentoring, curriculum & resource suggestions, inquiry & assessment design, co-teaching or classroom visits, and whatever else may be of use. I'm also envisioning a new addition ro our mentoring series in our district where we connect early career secondary teachers with experienced teachers in an interactive seminar setting -- something like a carousel with hands-on activities. The one-to-one and small cohort models have worked quite well for the elementary teachers but we have not drawn out the secondary numbers we hoped for. The goal here is to impact the development of a vibrant classroom, purposeful teaching & learning, and authentic assessment.

Of course, the other side of this is that I won't be at D.P. Todd next year, perhaps never again as I imagine landing at a new school or situation when my current seconded assignment ends.  I have been at this school for 15 years -- two-thirds of my career -- and I leave with mixed emotions. As I survey the vast hoard of books, lesson material, artifacts, and remainders of student projects that have accumulated in my classroom over the years, I am reminded, mainly, of the things I love about teaching. About teaching high school Social Studies students in particular.  There have been frustrating parts, too, but I've disposed of that evidence and generally suppress those memories because, hey, when you're in in for the long haul it has to be about the passion and positive stuff, otherwise it is time to get out.  I have been really fortunate to have some special students in the last few years, students who may not have been at the top academically, but really stepped up to conduct meaningful research and find creative ways to express their learning.  That's the group I have enjoyed teaching the most.