BCP/Urbanite High-Stakes Testing Forum
On Thursday, September 17, 2009 the Baltimore Curriculum Project and Urbanite Magazine held the Leading Minds High-Stakes Testing Forum at the Frederick Douglass-Isaac Myers Maritime Park.
Speakers included Harvard University Education Professor Dr. Daniel Koretz, University of Michigan Professor of Educational Policy and Economics Dr. Brian Jacob, and Michigan State University Assistant Professor of Teacher Education Dr. Rebecca Jacobsen.
Jonathan Brice, Baltimore City Public Schools Executive Director for Student Support, moderated the forum.
The forum was part of a larger strategy to saturate the public consciousness with this issue through public discourse, print media, and radio.
Urbanite’s “Learning Issue”
Urbanite Magazine’s September “Learning Issue” featured an article on high-stakes testing by Sara Neufeld. The article included interviews with BCP principal Rhonda Richetta, BCP principal Matt Hornbeck, BCP President Muriel Berkeley, and forum panelist Daniel Koretz. The “Learning Issue” was made possible by a generous grant from OSI-Baltimore.
Marc Steiner Show
On September 23, 2009 The Marc SteinerShow on WEAA 88.9 FM featured a panel discussion on national standardized testing in public schools as part of their Urbanite radio stories. The panel included BCP principal Matt Hornbeck and author Linda Perlstein.
Dr. Daniel Koretz
Dr. Koretz’s presentation focused on how high-stakes tests can create incentives that sabotage effective instruction.
“We have a really pressing need for better accountability systems and I don’t see how we can have them without testing, but what we have now isn’t working very well,” said Dr. Koretz.
“It’s not meeting the aims of the people who designed it. It is causing gains in scores that are simply not believable. And it’s in many ways corrupting educational practice.”
Tests are very small samples drawn from very large domains. Scores are only useful if one can generalize from them to mastery of the domain.
However, according to Dr. Koretz, “all that matters in our current accountability system is how kids do on that little, tiny sample of items... In many schools the requirement is very clear that the rate of improvement and performance on that little, tiny sample has to be very rapid. So what do you do? You focus on the sample.”
“That has two effects: One is that it makes for horrible instruction in many cases and the second is that it creates ... grossly exaggerated gains in scores.”
Not all test prep is bad. Good test prep gives students knowledge and skills that they can apply elsewhere such as in later education and employment.
Score inflation comes about through bad test prep, which includes reallocation (shifting instructional resources around among areas within a subject and between subjects) and coaching (focusing on the details of the test and teaching testing tricks).
Dr. Koretz said that we should make other valued outcomes count and use more than scores to evaluate schools. He also recommended setting realistic targets for student achievement gains.
The current targets under No Child Left Behind create the wrong incentives for teachers because NCLB “doesn’t say we want to see realistic, steady improvement from you, it says we want you to do miracles and the miracles come from coaching,” said Dr. Koretz.
He advocated evaluating the accountability system itself by auditing gains and monitoring changes in educational practice.
“In most areas of public policy, say drug safety or vehicle safety, vendors are required to make data public and they are required to have their products evaluated” said Dr. Koretz.
“If you see signs that Vioxx is causing cardiac problems it is not the prerogative of the CEO of the drug company to say its not in my interest to have an evaluation of Vioxx.”
But in public education, there is no expectation of independent evaluation. Restrictions on access to data severely limit evaluations of educational accountability programs.
“That is exactly what States and local education agencies say all the time… evaluation is not in our interest.” Dr. Koretz pointed out that “what matters is the interest of kids.”
“Since we know score inflation is a problem we have to find ways to audit gains, where they are real and where they are not. We need to monitor what’s happening in schools and not assume things are getting better because scores are rising.”
“I think if we actually could look under the covers; if we got people to agree to monitoring; what we’d find is that there are certainly some low-achieving schools where things have gotten better as a result of accountability pressure. We would also find some schools where things have gotten considerably worse.”
“We just don’t know which is which now and that is a pretty big problem.”
Dr. Brian Jacob
Dr. Jacob offered a mixed assessment of No Child Left Behind’s impact on student achievement.
He explained that incentives do not equal effects. There is an incentive under NCLB to focus on students who are right on the bubble of making proficiency and neglect the very low-achieving and very high-achieving students.
However, that doesn’t always mean that bubble kid achievement is higher than achievement for other students. For example, adding an extra reading period to the school day would help all students.
In his research in Chicago Dr. Jacob found there was a propensity to exclude students from testing by placing them in special education classes, but the impact was a mix of large and small effects.
He also said that context must be considered. In some Chicago schools NCLB was a vast improvement over the prior accountability system. In others it was derailing them from the valuable work they had been doing prior to NCLB.
Dr. Jacob warned that we have to be aware of what incentives are created when we create new accountability systems. The impact of accountability reforms in Chicago has been mixed.
There have been tremendous increases in standardized test scores, but these gains have not been reflected uniformly on Chicago’s low-stakes test. The low-stakes test showed similar gains for 8th grade, but for 3rd and 6th growth levelled off or decreased.
Dr. Jacob also highlighted some negative impacts of high-stakes testing in Chicago. For example, special education placements for low-achieving students in low-achieving schools jumped approximately 18% after high-stakes testing was implemented.
Preemptive retention, holding students back one to years in first or second grade to improve 3rd grade test scores, also jumped considerably Some teachers in Chicago said preemptive retention was beneficial for their students, but some research concludes it has negative effects.
Teacher cheating did increase after No Child Left Behind, but this does not account entirely for achievement gains.
According to Dr. Jacob, NCLB has generated some meaningful gains in math across the country, but not in reading. There has been no significant impact on science achievement and no effect on 4th or 8th grade reading
He recommended amending NCLB to include value-added assessment, which measures student growth, and to focus resources on the highest need schools. He said we need additional strategies for identification and improvement of failing schools, such as inspectorates. We should also look at the potential benefits of computer-adaptive testing. Additional research that focuses on teaching and learning is critical.
Dr. Rebecca Jacobsen
Dr. Jacobsen’s presentation focused on getting accountability right.
“If you’re going to get accountability right you have to first decide what we want to be held accountable for,” said Dr. Jacobsen.
“We have a number of goals for what we want our students to achieve, not just reading and math. We can’t rely on one sole measure. The more we rely on math and reading scores alone the more we try to game it and it ends up harming that overall list of goals that we value.”
The original National Assessment of Educational Progress (NAEP) had a much broader conception of what schools should be doing than the current assessment system and looked at a range of educational goals including student behavior.
Measures included multiple choice assessments, performance assessments, and observation of behavior in group situations.
For example, in order to assess social skills and work ethic students were given problems to solve in a group setting and then observed to see how well they worked together.
The early NAEP anticipated some of the challenges of NCLB such as preemptive retention. Students were assessed at specific ages, not grade levels. NAEP also assessed young adults ages 26-35 to find out if they had retained the knowledge, skills, and values gained in school.
Dr. Jacobsen recommended that we expand NAEP and reincorporate some of the original pieces of the assessment so that we can collect better information about schools than we have today.
An expanded NAEP would include assessments of all academic subject areas, paper and pencil test items, survey questions, performance observations, better demographic data, age-level assessments, and in-school and out-of-school samples.
NAEP gives us a state and national picture, but we also need a way to assess schools and provide information to parents at the local level. To address this issue, Dr. Jacobsen and her colleagues have looked at what other countries are doing.
The U.K. uses highly-trained full-time inspectors to assess 15-30 schools a year and offer recommendations.
The inspectors review test data, visit every classroom, interview students, and look at student work and writing samples. They also look at cognitive and behavioral skills.
Because the inspectors visit the school they get a much bigger picture of what’s going than they would from test scores alone.
In the U.S. we have six regional accreditation agencies that rely on a peer review process to foster continuous improvement. Evaluators are untrained volunteers. Schools set their own goals and reviewers assess how well they have attained these goals, but there is no way to hold schools accountable.
Dr. Jacobsen recommended that schools be reviewed by a trained team of evaluators every three years. There should be clearly defined state goals, spontaneous visits, and consequences for failing inspections.
She said that there are existing alternatives to our current system of accountability and that a new accountability system will be expensive, but if we want to get accountability right we must make the investment.
“From my point of view we have multiple achievement gaps and I think ...we [may be] narrowing the achievement gaps in one place but only widening them in the others,” said Dr. Jacobsen.
“By removing gym time we may be narrowing the gaps in academic achievement scores but we may be widening the gap in terms of the number of students that are obese in particular communities in particular places.”
“Is that a trade off that we’re willing to make and, if not, how do we figure out a way to balance them? That’s a policy conversation we need to have. It really comes down to what do we value?”
Watch, Read, and Listen
Thank you
We would like to thank The Abell Foundation, Northwest Evaluation Association, and Venable LLP for sponsoring the forum; OSI-Baltimore for supporting the Urbanite’s September 2009 “Learning Issue”; Marc Steiner for hosting a show on testing; Living Classrooms Foundation for providing a wonderful event space; Brian Jacob, Rebecca Jacobsen, and Daniel Koretz for being panelists; Jonathan Brice for moderating; Tracy Ward and David Dudley at Urbanite for their partnership, and Starbucks for providing coffee.
BCP President Muriel Berkeley to Co-Chair Rawlings-Blake Education Commmittee
On Friday January 22nd incoming Baltimore Mayor Stephanie Rawlings-Blake named BCP President Muriel Berkeley as co-chair of her Education and Youth Services Transition Committee.
Dr. Berkeley will co-chair the committee with Ralph Moore, Director of St. Frances Academy Community Center. Ronald Daniels, President of The Johns Hopkins University, will also serve on the committee.
Read the Baltimore Messenger article...
Thank you for supporting the Baltimore Curriculum Project
We would like to thank all of the individuals, corporations, and foundations who supported the BCP Annual Fund, the BCP Leading Minds Series, and BCP school programs in 2009.
If you would like to support BCP please donate online at www.baltimorecp.org or send your check to:
Baltimore Curriculum Project, 2707 E. Fayette Street, Baltimore, MD 21224.
$100,000+
Baltimore City
Family League of Baltimore City
Maryland 21st Century Community
Learning Centers
$50,000+
Living classrooms Foundation
$10,000+
The Annie E. Casey Foundation
Legg Mason Charitable Foundation
Open Society Institute - Baltimore
$7,500+
The David L. Holder Education Foundation
The Jean and Sidney Silber Foundation
Ravens All Community Team Foundation
$5,000+
St. John’s Church
$2,000+
The Abell Foundation
Al and Muriel Berkeley
Walter G. Lohr, Jr. Charitable Foundation
Northwest Evaluation Association
Venable LLP Foundation
$1,000 +
Ellen and Ed Bernard
Congressional Bank
Mr. John W. Guinee III and
Mrs. Michelle K. Guinee
Johns Hopkins University School
of
Nursing
Lois and Philip Macht Family Philanthropic
Fund of the AJC
H. Canfield Pitts
Southern Baptist Church
David and Adena Testa
$500 +
George Hess
Mr. and Mrs. Stephen J. Immelt
Townsend and Bob Kent
Jeffrey and Harriet Legum
|
Michael and Cristina Niccolini
Anne S. Perkins
Bob and Sharon Proutt
$250 +
David and Bonnie Allan
Jon and Dorothy Baker
BD Diagnostics
John and Kimberly Booher
David S. Clapp
Nicholas D. Cortezi II
Friends of Patterson Park
Larry Matlack
William O. and Mary H. Miles
Heather L. Mitchell
Lee H. and Marilyn K. Ogburn
Drs. Fadia and Elias Shaya
Matthew Sitek
James A. Snead
Herb and Brooke Thomas
Up to $250
Anayezuka Ahidiana
Jeffrey and Janet Ayres
Mr. and Mrs. Raymond Bank Philanthropic
Fund of the AJC
Linda C. Barclay
Steven and Sara Barley
Lance Bendann
Kay Berney
Maxine P. Blackman
Lee Bone
David Borinsky
Jerrilyn Borneman
Mike Bowler
Darragh Brady
Ronae Brock
Deborah Callard
Anthony M. and Eleanor M. Carey
James P. Casey and
Evelyn Omega Cannon
Betty Cooke and Wm. O. Steinmetz
Abigail Cooley
Robert M. Coulbourn III
The Rev. James R. Crowder and
Mrs. Suzanne B. Crowder
Richard Davison
Claudia A. and Phil F. Diamond
Christopher J. and
Laura Rice Doherty
|
Jacquelyn Duval-Harvey
Michael and Margaret Emlaw
Barbara L. Fegley
David Fetter
Redmond C. S. Finney
Richard W. and Donna Lee Frisch
Harry Halpert
Mr. and Mrs. Michael D. Hankin
Jane Harrison
Tina Hike-Hubbard
Sam and Barbara Himmelrich
David and Becky Hornbeck
Susan and W. Carl Hossfeld
Karen Hurt
Brenda L. Kahn
Trent M. Kittleman
Nathaniel E. Knox Sr.
Joseph Langmead
Ray A. Lucas
Carol Macht
Andrew Martire
John and Betsy McDonald
Richard and Beth Mitchell
Robert and Shirley Noll
David and Sherrill Pantle
Joan Partridge
Barbara and Tibor Patalics
Barbara and George Perkins
Marion Pines
Lynn and Ray Plack
Lana and Henry Pollack
Rhonda Richetta
Riedy Family Foundation
Francis Riggs
Ben and Dee Rosenberg
Charles and Suzanne Rowins
Peter V. Savage and Deborah Tillett
Mr. and Mrs. Ronald M. Shapiro
David A. Sheehy
Stuart O. Simms, Esq.
Solomon and Elaine Snyder
John Sondheim
Beverly Stappler
Ivan Stern
Devin N. Stevenson
Jeanette Stewart
T. R. Klein & Company
Susan Truitt
Susan and Ted Wachtel
George and Suzanne Wills
Mr. and Mrs. William H. Winstead III
Beverly Winter |
|
