11 June 2013

A Week in the Life of an AP Physics Reader


What happens between the time the AP Physics Exam is given and when scores are posted? While I can't walk you through all the steps, I can tell you how they are graded.

2 Weeks before

About 48 hours after the exam is given, the questions are posted online and shortly after that, the initial question assignments are sent to the invited Readers. Most then solve the appropriate question and develop a preliminary rubric. Some Readers may also join online discussions about these questions.

Day -3 to Day -1

Three days before the Readers arrive at the grading site (currently Kansas City for the physics exam), the leadership team gathers. The Table Leaders (the ones directly supervising the readers) and the Question Leaders (supervising the TL's) start working on the scoring rubrics. Starting with an initial one, the TL's start looking through random samples to develop a sense of how typical students  answered the questions and how well the rubric matches those responses. The TL's and QL's go through a few rounds of discussing and making small tweaks to the rubric until all concerned are satisfied that it is a fair way of assessing student knowledge of the material covered by the question. Once rubric version 0.99 has been developed, the TL's start developing the training material for the graders. As the TL looks through more and more student responses, the rubric may undergo some final adjustments, however it is locked down before the graders arrive.


Day 1

After an 8 am initial group meeting, the graders go to their assigned locations for training. The Table Leader introduces the Readers to Rubric Version 1.0. That initial discussion includes what responses get what point values and why the leadership team decided that. Common mistakes also get mentioned.

Once the TL is convinced that the Table has a good initial understanding of the rubric, the team breaks into small groups which then reviews a small set of common student responses and scores them as a group. Invariably, there is a spread of scores. The whole table then discusses the individual responses and what the correct scoring should be. The next round consists of graders scoring a sample set by themselves and then comparing scoring with a partner. Again, a group discussion follows, but usually the scoring is becoming more consistent.

Another round of partner discussions follows but with a set of more uncommon responses. The idea is that while everyone will score correctly the perfect or nearly perfect responses and the completely wrong responses, everyone needs to grade the strange and unusual responses the same way. As before, a round of individual scoring follows the partner scoring with discussions subsequent. If warranted, another round may follow either with the entire group or with individuals, but usually this two rounds of training is sufficient.

This Blog has been Back-read

By the afternoon of the first day, the readers are grading on their own, but with one important check, the back-read. Back-reading is when the Table Leader scores exams already scored by a reader. If the TL gets a different score, there is a quick discussion with that reader where the TL helps the reader to hone in on the correct adherence to the rubric. The problem is usually is a matter of some subtlety that the TL has seen with 3 days of looking at exams but that the reader has not seen with only a few hours.  If there are consistent and substantial differences between  the scoring by the reader and by the TL, that reader may be pulled aside for some focused retraining, however, that rarely happens. The TL back-reads not every exam. but a random sample from each reader the next couple of days. With most graders, as they become more experienced, the number of differences falls. The TL then back-reads fewer of those graders exams and spends more time with the graders who are having a difficult time with the more subtle scoring decisions they have to make. The back-reading process is an important one in ensuring that each exam is scored according to the rubric and that an exam that is scored a 9 in the first day would be scored a 9 on the last day.

Days 2-7

Now the slog starts. Each day starts with some quick announcements, but then it is heads down and grading. This is not a time where you grade a few exams, go get a cup of coffee, exchange gossip, and then go back to grade a few more. The expectation of graders is that they are actively scoring exams from 8:00 until the mid-morning break is called. Then, precisely 15 minutes later, you are back to scoring until the lunch break. One hour later, your head is down and you are scoring until the mid-afternoon break and then until the "End of Day" call comes. At first, it seems to be brutal and demeaning to highly-training professionals. But, when you personally have to go through several thousand exams, and the group has to score all, discipline is key.

Once the rubric is "internalized", a typical Reader develops a rhythm and gets into a zone. While the day is long, the repetition seems to make it shorter.

One quirk with the Reading is that you are not expected to start scoring until the actual start time. In fact, readers who want to get an early start are teased and chided (gently). While tardiness is frowned upon and will have a detrimental effect on your review, trying to impress the bosses by starting a half-hour early in the morning will not be noticed (officially).

Once a couple of days have passed, the Chief Reader may determine that certain questions are not being read as fast as anticipated, and so some Readers may be reassigned to other questions. For those Readers, the training process starts anew, but the slog continues after at most half a day. Should a Reader be on a question that finishes early, that Reader is then put in the "Bubble Room". That Reader then bubbles in the scores that another Reader has recorded on a question that has a lot of exams to grade. The goal is that everybody finishes together.

 So, why do it?

Some Readers come to one year's reading, hate the regimentation, and are never seen again. But most are eager to come back. Why? One reason is that some see this as summer camp for physics teachers with many of your favorite "bunkmates" returning year after year. Another is that you learn so much about how students can see a problem and what misconceptions they may have. It helps your teaching.

The Reading is held in a section of Kansas City that has a small but vibrant urban core, so there are many evening activities (the picture above is the home of the KC Symphony which is a couple of blocks away from our hotel). The College Board also hosts a couple of evening forums to let people know about the future direction of the AP Exams. And not to be missed is the Physics Professional/Unprofessional Night. So when the Readers are waiting outside the hotel early in the morning to catch flights home, many are already talking about the next year.

No comments:

Post a Comment