Sunday, October 30, 2011

The Two-Questions Method of Word Meaning Assessment: A Group Procedure for Use with Preliterate Children (Kearns & Biemiller)

This post is courtesy of Gail Kearns and Andrew Biemiller. Gail Kearns is a reading specialist and former teacher who founded a diagnostic and remedial reading clinic in Concord, Massachusetts. She worked for Jeanne Chall at Harvard University as a research assistant on readability projects and on the first edition of the Diagnostic Assessments of Reading (DAR); more recently she co-authored the Second Edition of the DAR. Gail also co-authored Trial Teaching Strategies (TTS), a useful tool for planning tutorials and interventions.  Andrew Biemiller, Professor Emeritus, University of Toronto, is a vocabulary researcher and the author of Words Worth Teaching: Closing the Vocabulary GapAndrew is advisor to the WGBH television program for young children, Martha Speaks. Please see  Andrew's prior post describing this educational and yet entertaining PBS program. Andrew is a national expert in reading and language comprehension, and has also investigated self-direction in schools. Gail and Andrew have collaborated together on vocabulary research, described below.


For too long, classroom teachers and researchers have been unable to assess vocabulary reliably and efficiently with classes of preliterate children—young children who either can’t read or can’t read reliably. These difficulties have made individual assessment of word meaning with such students a necessity. We believe the amount of time that has been needed for assessment is partly why educators have tended to neglect vocabulary with children in the primary grades and also why so few word meaning studies have been able to be conducted in these grades.

Young children’s general vocabulary level has been assessed to date with tests such as the well-known Peabody Picture Vocabulary Test (PPVT). However, this test, like other such vocabulary tests for students in these grades, requires individual assessment, taking 15 to 25 minutes per student. This is more time than classroom teachers can spend with each child, particularly for regular assessment at intervals following instruction. Furthermore, the PPVT is largely limited to assessment of concrete meanings, i.e., word meanings that can be readily pictured.

We have recently published a study of our method for group testing vocabulary word meanings that was conducted with 257 students in grades K, 1, and 2 (Kearns & Biemiller, 2010). The results of our validation study demonstrated that results similar to individual PPVT assessments could also be obtained with our group method. (Correlations between overall children’s PPVT raw scores and scores from our group test were r = .76 in Kindergarten, r = .71 in Grade One, and r = .70 in Grade Two. Correlations between word mean scores from the two methods were .76 (Kindergarten), .79 (Grade 1), and .94 (Grade 2). In short, these results indicate that teachers could use this method both to determine overall vocabulary needs in classrooms, and to monitor the acquisition of word meanings and success of vocabulary instruction. Researchers could use this new group method to shorten the amount of time needed to conduct vocabulary acquisition studies in these grades.

In this entry, we want to briefly describe how the Two-Questions Vocabulary method is used for assessment of word meanings in the primary grades. Additional details for conducting such assessments are available from the authors and in our journal article about the study.

When constructing a Two-Questions assessment, each word should be assessed with two short questions, each of which can be answered with either “yes” or “no,” and with one “yes” question and one “no” question used for each word. For a child to be credited with knowledge of a word meaning, both questions have to be answered correctly. (When this is done, the odds of guessing a correct answer is the same as on the PPVT, where the student must choose 1 of 4 possible alternative pictures.)

With the exception of the assessed word, all other words in the questions are of high text frequency, and the questions are kept simple (no complex clauses, etc.). The two questions for each word are placed in random order on the test; they never follow each other and are kept as far apart as possible. In the study, we used one of the questions for each word in each test session. We chose equal numbers of “yes” and “no” questions (and arranged them in random order) for each test session.

Here are some examples of Two-Questions items taken from a test administrator’s form. (Words in these sample items have not been taken from the copyrighted PPVT test.)

Items from a sample administrator’s test form. (© Kearns & Biemiller, 2010). Graphics
used are from collection of 6,000 “Literacy Support Pictures,” courtesy Slater Software.

The authors strongly recommend use of a Student Response form, particularly when assessing kindergartners. Children are asked to circle a “yes” or “no” alternative for each item. We found that use of 5”x 7” cards for tracking items were also needed with kindergartners and for many first graders. Details of how best to use the cards and Response Sheets are available in the article or from the authors.

Sample items from a Student Response Sheet. (© Kearns & Biemiller, 2010). Graphics
used are from collection of 6,000 “Literacy Support Pictures,” courtesy Slater Software.


How long does administering the Two-Questions tests take? In our experience, the first time, we recommend two sessions on different days be used for assessing approximately 20 words. The first session will take about 45 minutes, with at least half of the session used to introduce the Response Sheet and to practice answering sample test items. The second session, conducted on a different day, should take 15 to 20 minutes. Second graders may be able to be assessed in a single session of about 50 minutes.

Later in the year, after the group assessment method has been used earlier with a classroom, and after the students have become familiar with how these tests are conducted, additional word meaning assessments for instructional purposes with the Two-Questions method may be able to be completed with single sessions of about 30 minutes for 20 words.

During the course of the study, we came to believe that an additional separate initial group session for training purposes would have been desirable for the kindergarten children and should be built into other studies involving group testing of word meanings in kindergarten.

In conclusion, we believe that this method can be used to create items for both assessing general vocabulary knowledge and for regularly testing samples of word meanings taught throughout the school year in the classroom. (We recommend teaching word meanings in the context of stories read aloud by the teacher and through other on-going classroom activities. Methods of teaching are discussed in Beck, McKeown, & Kucan (2002 and 2008), Graves (2006 and 2009), Stahl & Nagy (2006), Blachowicz & Fisher (2006), and Biemiller (2010). Ways of choosing words for instruction are discussed in the Beck et al and Biemiller books.

References
Beck, I. L., McKeown, M. G., & Kucan, L. (2002). Bringing Words to Life: Robust Vocabulary Instruction. New York: NY, Guilford Press Inc.

Beck, I. L., McKeown, M. G., & Kucan, L. (2008). Creating Robust Vocabulary: Frequently Asked Questions and Expanded Examples. New York: NY, Guilford Press Inc.

Biemiller, A. (2010). Words Worth Teaching. Columbus, OH: SRA/McGraw-Hill.

Blachowicz & Fisher (2009). Teaching Vocabulary in All Classrooms. Boston, MA: Allyn & Bacon.

Graves, M. F. (2006). The Vocabulary Book: Learning and Instruction. New York, NY: Teachers College Press.

Graves, M. F. (2009). Teaching Individual Words: One Size Does Not Fit All. New York, NY: Teachers College Press.

Kearns, G., & Biemiller, A. (2010). Two-Questions vocabulary assessment: Developing a new method for group testing in kindergarten through second grade. Journal of Education, 190 (1/2), 31-41.

Stahl, S. A. & Nagy, W. E. (2006). Teaching Word Meanings. Mahwah, NJ: Lawrence Erlbaum Associates.

6 comments:

  1. Thanks, Gail and Andy. I have received several emails about your post. People are happy to hear about this methodology. Several people wrote to say, "Thank you!" but some had questions. One question was "What are the pictures for, on the response sheet?" (so I explained they are place finders for preliterate children). Another question was, "How do I get a copy of the entire article?" With your permission, I will send your article to the reader?

    Thanks again!
    Susan

    ReplyDelete
  2. Anonymous10/31/2011

    I cannot tell you how timely this article is. I have been struggling with trying to come up with a method for teachers for determining if kids know the words or not. Teachers are spending way too much time teaching words that kids already know.


    The only problem I see is that teachers do not have the time and/or possibly would not feel confidence in constructing the items. We did some similar things in a workshop just last week (“Beck-ish” type things)—and teachers did them at their tables together. As I watched, I noticed that some of the leaders at the tables did the items while others watched. It is hard to tell from that if they would feel confident on their own. But even if they can, they don’t have the time. Ideally, the publishers of the core programs need to do this for their existing editions—but I am afraid that they will wait until they come out with the next edition and do it then.

    Question: Is there some reason that this is only for preliterate students? Wouldn’t it work for those who can read as well? Or are these kinds assessments already being used within cores for kids who are reading?

    ReplyDelete
  3. Gail Kearns and Andy Biemiller11/01/2011

    Hi Susan,

    Regarding the article -- it took Andy and I the better part of three years to get it published. It was published by The Journal of Education (address for obtaining a copy: Boston University School of Education, Two Silber Way, Boston, MA 02215 or http://www.bu.edu/journalofeducation/). In recent years this journal became a refereed journal under its new editor, Dr. Roselmina Indrisano, who in earlier years was head of the International Reading Association and was also elected to its "Reading Hall of Fame." Andy and I were honored by her interest in our work and publication of our article.

    You are correct about the use of the pix in the margins of the Response Sheet and the Administrator's Sheet. They are indeed place finders for the children. (And they are on the administrator's sheet to aid in helping the children to use them.) Additional details about the study are given in the article (which is about 20 pages rather than the three used to summarize the study for your blog!).

    Cheers,

    Gail and Andy

    ReplyDelete
  4. Gail Kearns and Andy Biemiller11/01/2011

    All of the Comments under "Anonymous" are certainly valid. In May Andy and I went to IRA in Orlando and gave a 3-hour workshop meant for teachers and for those working with teachers in which we taught those who participated to create these kinds of questions. Because IRA placed the presentation in the last hours of the last day of the conference, there was a small group rather than a large one for us to work with, and we were able to get a good sense of some of the difficulties that they were having in writing the questions. However, we are not yet ready to write about any of this. We are well aware of the time constraints on classroom teachers these days. Like "Anonymous," Andy and I also hope that publishers of both core programs and supplementary books for reading aloud will be able to aid teachers in creating brief versions of "Two-Questions" assessments for use with units and with individual stories and other selections. However, publishing such assessment materials will not take the place of teachers learning to create their own questions; classroom groups are very different from each other, and the words that each group does not know well will differ from class to class and from group to group. Even with published prepared lists of chosen words and questions, the teacher will still need to be the final decision-maker in the classroom for that which proves useful and appropriate for her own groups. The beauty of this method of assessment is that it makes it possible for teachers to tailor the words they assess to the particular group they are teaching and to be guided later for instruction in word meaning by the results of their varying assessments. However, writing about these issues is going to take us quite some time, and we have no magical way of bridging the time we need to do this. But the more we hear from those who try to use our method on their own, the better we will be able to address the difficulties encountered by and reported from "Two-Questions" users. (Actually, we like to call them our "pioneers"!)

    You ask, "is there some reason that this is only for preliterate students? Wouldn't it work for those who can read as well?" The answer is that yes, Andy and I both think this method could be used with grades and ages beyond the primary grades. However, this was a small study, and it included only children in grades K, 1, and 2. So those are the only grades for which we have data, and we cannot speak to the validity of this method for use with other grades or ages. That being said, we think this method does indeed hold promise for use with almost any grade or age, from preschoolers through adult functional illiterates. There is also no reason why this method of assessment cannot be used for individual as well as for group assessment. This method also is appropriate for use with individuals who have difficulty with being assessed in groups, including those who may be too young for group assessment. Andy and I chose the primary grades for this study because we both believe that teaching many more word meanings in these grades is key to being able to bring about better reading achievement in the grades beyond the primary grades once adequate decoding skills have been acquired. Knowing which words to teach is needed for more effective use of instructional time, regardless of the instructional methodology used to teach them. When the teacher knows which word meanings are known and which are not before teaching, and which words have been learned after teaching and which have not, better use can be made of the very precious and scarce time that is able to be used for word learning in the classroom.

    Gail Kearns

    ReplyDelete
  5. Geri Mohler11/03/2011

    In working out this method of assessment, did the Happy Face/Sad Face choices ever cause confusion for the children? I can see why you would need a prompt if the child does not know the words Yes and No, but if the question was: Are beans and corn fruits? The child should choose the sad face but what does "sad" have to do with the answer? Am I being too literal?

    ReplyDelete
  6. Gail Kearns and Andy Biemiller11/04/2011

    Hi Geri,

    We used the "Smiley" and "Frown" faces for "Yes" and "No" but did not think of them as being "happy" and "sad." None of the children in any grade had difficulty with this use of our pix, and some would indeed have had difficulty keeping the words for "Yes" and "No" straight for lots of questions. You mentioned the "No" picture and described it as "Sad." I don't think the children thought it a "sad" face, and I know I did not. It was a "No" face of being against something, very similar to the actual face small children often make when they are saying "No!" very defiantly about something. At any rate, the pix did not cause difficulty of any kind as near as we could determine.

    Gail Kearns

    ReplyDelete

Comments are published after they are reviewed, to ensure they are not SPAM.