The Role of International Tests in U.S. Education
Why do U.S. students so often do rather poorly among developed nations in international tests.
The most interesting question and answer is this one:
2:03 Kathleen Kennedy Manzo: Gary, can you answer this question from Olga?
2:03 [Comment From Olga Amaral] Is there one common factor that has been found to be most influential in contributing to the success of countries with highest achievements?
2:05 Gary W. Phillips:
I think we do not have a final answer to this question. But it seems to me to be related to how well
education is respected and valued within the society. Those countries that care the most about
education achieve the highest. So--If American kids think high school is a joke, they won't take their
schooling seriously.
Possiblities: American high school *is* a joke. High schools everywhere are a joke, but Americans are more
willing to admit it. American culture makes them less willing to take gravely what the authorities want them
to take gravely.
There is hardly anything the Federal government can do to change basic attitudes about school, much less
increase respect for authority generally. (What is the optimal respect for authority, anyhow?) Ask instead
how
schools might change to make students more eager to learn. Since it is not possible for the central
government
to fiddle with what individual teachers and schools do, think about how "adequate yearly progress"
might be *variously* assessed at individual schools depending on what each one *now* tries to accomplish
that
gets away from one-size-fits-all for every state. That's too finely tuned, each school. Try counties.
Lots
of good thoughts in these exchanges. Read with profit.
Education Week: The Role of International Tests in U.S. Education
Asians Best U.S. Students in Math and Science
1:30 Web Person: Jennifer: Today's chat on the role of international tests in U.S. education is now
open
for questions, so please start submitting them. Our moderator and guests will here at 2 p.m. to begin
answering your questions. Thanks for joining us.
1:58 Kathleen Kennedy Manzo: Hi everyone. Thanks for joining us. Let's begin the chat.
Why don't our guests, Hal Salzman and Gary Phillips, introduce themselves first, and then they'll
start answering your questions.
1:59 Hal Salzman:
This is Hal Salzman - I'm Professor of Public Policy and sociologist at the Bloustein School of Public
Policy and the Heldrich Center for Workforce Development at Rutgers University. With colleagues, I have been
examining the science and engineering workforce “demand” and “supply”, which led us to look closely at the
education performance of U.S. students, finding that our best students are actually doing quite well,
contrary
to the widely-cited proclamations of poor performance.
2:00 Gary W. Phillips:
I am Gary W. Phillips the Vice President & Chief Scientist at the American Institutes for Research
(AIR).
I direct a team of psychometricians and statisticians on large scale assessment projects. I recently
published
several reports that involve statistically linking NAEP to TIMSS. The first report compared U.S. state-NAEP
results to TIMSS in math and science (Chance Favors the Prepared Mind, 2007), the second compared U.S.
School
District results to TIMSS (Counting on the Future, 2008) and a new report is about to be released that
compares states and school district to TIMSS using letter grades A, B, C, D and F. (International Benchmarks
in Mathematics, 2009). Prior to coming to AIR, I was the Acting Commissioner at the National Center for
Education Statistics.
2:01 Kathleen Kennedy Manzo: Thank you to both of you for sharing your insights and your expertise on this
topic.
Can each of you briefly characterize your views on the value of international comparison tests, and what the
results do and do not tell us about how our schools are doing?
2:03 Hal Salzman:
There is certainly value in comparing schools and education systems to learn from others. However, the conclusions often drawn from these results go far beyond what the actual data support. U.S. academic performance at the top is actually quite good comparatively, as the most recent tests show.
2:03 Gary W. Phillips: International assessments give the United States a broader context within which to
understand our own student learning and teaching practices. They do not provide a silver bullet but they add
to the knowledge base that helps us decide on best practice.
2:03 Kathleen Kennedy Manzo: Gary, can you answer this question from Olga?
2:03 [Comment From Olga Amaral]
Is there one common factor that has been found to be most influential in contributing to the success of
countries with highest achievements?
2:05 Gary W. Phillips:
I think we do not have a final answer to this question. But it seems to me to be related to how well
education is respected and valued within the society. Those countries that care the most about education
achieve the highest.
2:06 Kathleen Kennedy Manzo: Here's one Hal from Nick
2:06 [Comment From Nick]
Shouldn't we be more worried about U.S. graduation rates, and performance in local schools, rather then
worrying about American scores on international tests? Why is it that international test are being looked at
to help shape our education policies? Shouldn't we be looking towards better training for teachers and
focus on curriculum rather then focusing on changing policies?
2:07 Hal Salzman:
I would agree and we've argued that we have more to learn from high performing districts and states (e.g., Massachusetts and Minnesota) than from Finland -- if only beause of greater transferrability to US context.
2:07 Gary W. Phillips: All of these factors are important to look at. We should be looking at graduation
rates, achievement as well as better taecher traing and improved curriculum.
2:07 Kathleen Kennedy Manzo: Gary there are a couple of questions about the comparisons themselves, when the
tests are given, the comparability of groups.
I'll post one or two for you.
2:07 [Comment From Robert D. Ferrari]
How are the comparison groups of various countries selected? Are they representative groups based on the
same
set of criteria?
2:08 [Comment From Stuart Hobbs]
My question concerns differences in educational goals between the US and many other countries that makes the
comparisons on international tests (which I find interesting and a good idea) somewhat problematic.
Specifically, in the US we are trying, more or less, to educate everyone to the same level, while in other
places by our equivalent of middle school students have been separated on academic or vocational tracks. If
that is so, won't that mean that while we can certainly improve on our test scores, we will always be
running behind other countries if their test results include only the academic track and ours include
everyone?
2:08 Hal Salzman: The testing groups try to use comparable samples but the reality of a large, complicated
study is that samples often are not comparable. In the PISA, sampling differences may account for a
significant portion of the outcome differences.
2:09 Gary W. Phillips: Each sample in each country is a scientifically representative sample. An
international
sampling referee makes sure that each country complies with sampling requirements.
2:10 Hal Salzman: Gary, I would agree that they try to do so, and make a good effort at it, but they
don't
always succeed. There are substantial differences in the final sample and the U.S. sample in particular has
been a problem in several of the tests. In our papers we discuss in detail the impact of sampling
problems.
2:11 Gary W. Phillips: I disagree with Dr Salzmans conclussion. If there is one thing that we can trust in
these studies it is that the sampling is done well. The minor sampling problems that due emerge due no make
much difference.
2:11 Kathleen Kennedy Manzo: Can either of you answer this question from Kris?
2:11 [Comment From Kris Palfrey]
What countries are achieving the highest scores and what examination boards are they using?
2:13 Kathleen Kennedy Manzo: In this week's Education Week we have a series of articles on other
countries
that are doing well and some of the factors that might be contributing to their high performance on
international assessments.
www.edweek.org
2:13 Gary W. Phillips: The highest achieving countries are Chinese Taipei, South Korea, Singapore, Hong Kong
and Japan and Finland. Each has a different exam system.
2:15 Kathleen Kennedy Manzo: Hal, can you address this question from Jerry?
2:15 [Comment From Gerald W. Bracey]
Given the Japan scores among the highest nations in the world but has seen 20 years of economic recession
and
stagnation, and given that high scoring Iceland became an overnight economic basket case with a national
debt
equal to 850% of its Gross Domestic Product, and given that middle-scoring United States has long been
ranked
first in global competitiveness by both the Institute for Management Development and the World Economic
Forum,
why should anyone pay any attention to these test scores?
2:17 Hal Salzman: There are two questions here --- is there anything to be learned from educational testing
and is economic performance the primary outcome that we should be interested in. There is a qualified yes to
the first and, as you make clear in the question, economic performance is due to many other factors. It is,
quite obviously, not reasonable to focus on education as the primary driver of economic performance.
2:18 Kathleen Kennedy Manzo:
Just some background on these tests and the results, from my colleague Sean Cavanagh:
American students have fared much better on one international test, the TIMSS, than on another PISA. There's a lot of speculation about why. The TIMSS, like the domestic NAEP test, measures school-based curriculum. The PISA measures “yield,” or the skills that students have picked up in and out of school, and their ability to apply them to real-world contexts.
For PISA results: http://nces.ed.gov/pubs2008/2008016.pdf
And TIMSS: http://nces.ed.gov/timss/results07.asp
2:18 Kathleen Kennedy Manzo: Gary, you have done some studies linking the scores for international tests on
a
scale with NAEP results. What did you find?
2:20 Hal Salzman:
As additional follow up to Jerry's earlier question, it is also worth noting that the U.S. tops the
scale
on reading and other subjects. It is arguable that only math and science are the most important
subjects.
2:20 [Comment From Milad]
Are there alignment studies between TIMSS and international tests?
2:20 Gary W. Phillips:
This observation is an artifact of how the two surveys calculate the international average to which we
compare the United States. PISA uses the 30 OECD countries so our standaing relative to them is lower. That
is
why we appear to do less well on PISA than on TIMSS.
2:21 Kathleen Kennedy Manzo: Anne is referring to efforts by states to rewrite state standards using
international benchmarks. Can one of you answer her question?
2:21 [Comment From Anne]
2. We are hoping our state will rewrite its standards, using international standards as benchmarks. What
other
factors should we consider?
2:21 Gary W. Phillips: There are content aligment studies between all the international assessments and they
can be found by going to NCES web.
2:23 Gary W. Phillips: There "is" a way to internationally benchmark state achievement standards
using PISA, TIMSS and PIRLS. I am doing a workshop on this with state testing directors the first week in
May.
2:24 Kathleen Kennedy Manzo: This is a hot topic right now.
Two influential organizations, the National Governors Association and the Council of Chief State School Officers, in December called for setting common state standards in reading and math and benchmarking standards in high-performing countries.( See Ed Week's story on a report issued by the two organizations, at http://www.edweek.org/ew/articles/2008/12/18/16nga.h28.html)
And just last week, representatives of 41 states convened at a conference in Chicago to explore those issues. The event was co-sponsored by the NGA and CCSSO. Efforts to promote national standards have been promoted, and fizzled, in the past. Will this new effort overcome that fate, and what role would international benchmarks play, if any?
http://www.edweek.org/ew/articles/2009/04/16/29standards.h28.html
Just yesterday the House Education and Labor Committee announced a hearing, set for April 29, on “to
examine
how states can better prepare their students to compete in a global economy by using internationally
benchmarked common standards.”
2:24 Kathleen Kennedy Manzo: Hal, what do you think these efforts to identify international benchmarks and
align state standards accordingly?
2:26 Hal Salzman:
This, of course, raises the question about what we think is important for competing globally. Presumably it
is performance after they leave school and not clear that the U.S. is at a disadvantage. We still lead the
world in innovation, R&D, etc. Perhaps there is something about our education system that leads to the
US
being the world's leaders in science, technology, innovation, entrepreneurship. Moreover, it is our
universities that seem to produce the world's best graduates.
2:26 Kathleen Kennedy Manzo: Gary, Patti has a question about participation on these tests, particularly for
India and China.
2:26 [Comment From Patti]
Is it possible to know how U.S. students compare academically with students from mainland Chinese and
India--because those countries don't participate in the PISA or TIMMS?
2:27 Hal Salzman: If one factors in the 30-50% illiteracy or near-literacy rates of those countries, their
averages would be pretty close ot the bottom.
2:28 Gary W. Phillips:
We do not know how the U.S. compares to these two countries. However, I believe PISA plans to include both
countries (or at least a portion of both countries) in their next assessment.
2:29 Kathleen Kennedy Manzo: This next question is for Gary, regarding your earlier response on
high-performing countries
2:29 [Comment From Robert Pondiscio]
I'm intrgued by your response that "those countries that care the most about education achieve the
highest." How are you defining caring? Economically? Or through some manner of value judgement?
2:29 Kathleen Kennedy Manzo: Hal, can you answer Karen's question?
2:29 [Comment From Karen]
In the Education Week article, Mr. Salzman, you mention that many U.S. students fall out of the math and
science pipeline. Is that a problem for these other high-performing countries as well?
2:31 Kathleen Kennedy Manzo: I have a better link for the article Karen is referring to, as well as all of
Education Week's recent coverage of this issue over the last year, in commemoration of the 25th
anniversary of the A Nation at Risk Report
http://www.edweek.org/go/nar
2:31 Gary W. Phillips: If you look inside the cultures of the high achieving countries you will see that
teachers are well respected, teaching is a competitive job, parents are involved with school and students
take
school seriously.
2:31 Hal Salzman: We don't know because we haven't tracked enfollments in those countries. We do
find
many engineers entering management, for example, in Japan and Germany, so there is some career mobility.
Overall there tends to be more tracking by discipline.
2:32 Kathleen Kennedy Manzo:
Here's some more background from Sean Cavanagh:
In a 2007 paper, Hal Salzman and Lindsay Lowell, of Georgetown University, argued that the United States
K-12
system produces an ample number of students with skills necessary to compete in science and engineering
fields, contrary to widespread belief. American students' math and skills are equal and in some cases
better than they were two decades ago, they conclude. The problem lies in graduate programs and science- and
engineering-related companies, which are ineffective at recruiting talented students and convincing them to
stay. http://www.edweek.org/ew/articles/2007/11/07/11report-b1.h27.html
2:33 [Comment From Louise Johnson]
Would the formation and implementation a national curriculum have an affect upon international assessments?
If
so, what are your opinions?
2:33 Gary W. Phillips:
Hal Salzman believes that in raw numbers, the U.S. produces many more top-tier students. This is only
partially true. One example where it is not true is grade-8 mathematics. In an analysis of 2007 TIMSS
grade-8
mathematics Korea had 40% advanced, Japan had 26% advanced while the United States only had 6% advanced. If
you look to see how many students that represents in the population, both counties still out rank the United
States in absolute numbers. That is because they have such a huge percentage of high achieving
students.
2:34 Kathleen Kennedy Manzo: Sorry, I jumped the gun on that last question. But I think Louise is asking
what
others are wondering as well, particular as policymakers here debate this issue. Can either of you chime in
>?
2:35 Hal Salzman:
It raises the question of what are we measuring and why. I'm not sure that Japan's economy has
benefitted from its production of high performing math students. Maybe the reason the US is the world's
leading innovator is because of its diversity and produces broadly educated students?
2:35 Kathleen Kennedy Manzo: Louise, I want to refer you to my story this week about Australia's effort
to
create a national curriculum. It has a lot of traction after a very similar history as ours with national
standards development
http://www.edweek.org/ew/articles/2009/04/22/29nar-australia_ep.h28.html
2:36 Gary W. Phillips: It is hard to know if a national curriculum would help. I think it probably would.
Howver, most of the high achieving countries have national curriculums but so do most of the low achieving
countries.
2:39 [Comment From Ty]
Dr. Salzman indicated earlier that 'US academic performance at the top is actually quite good.'
Earlier TIMSS studies compared just the top US students with the top students from other countries and I
believe that even US AP students scored below average compared with the top students from other countries.
Can
you confirm and comment?
2:40 Gary W. Phillips: This is correct. In the Advanced TIMSS study the United States still did not compare
well with the other countries.
2:41 [Comment From Spencer Siegel]
Which states in the U.S. perform the best and worst on international exams? Do some states do just as well
as
some of the top-performing countries?
2:42 Hal Salzman:
The recent TIMSS showed the two states that paid for larger samples so they could be compared
internationally
came out at the top. There are sampling issues in these small sample comparisons such as AP only students.
Also question about who takes the tests, etc. Interpreting small differences goes far beyond what the
international testing methods will reasonably support.
2:42 Kathleen Kennedy Manzo: Good question Spencer. Massachusetts and Minnesota took the TIMSS in the last
go
around and did nearly as well as some of the highest performing countries.
Here is our story on it: http://www.edweek.org/ew/articles/2008/12/09/16timss.h28.html
2:42 Gary W. Phillips: The only states that took an international exam recent were Mass. and Minn. Both
scored
very well on the exam.
2:43 [Comment From Mike Bouldin]
Hal said that "our best students are actually doing quite well...." Is that enough for our society
to continue to function as a vibrant democracy with a robust economy? Or are there factors that will in the
future require us to have a greater proportion of our students be high achievers? I'd like to hear both
Hal and Gary address this.
2:44 Hal Salzman: One also needs to note that the US population is far more diverse, and comes to schools
with
much greater educational challenges than in any of the other top performing schools. such as not speaking
native language at home, singleparent household, etc.
2:45 [Comment From Anne]
1. Is the problem that U.S. achievement is dropping, that other countries have caught up and moved past the
U.S., a combination of these, or something else?
2:46 Gary W. Phillips:
Let me repeat my statement from before. As I stated above Dr Salzman believes that in raw numbers, the U.S. produces many more top-tier students. This is only partially true. One example where it is not true is grade-8 mathematics. In an analysis of 2007 TIMSS grade-8 mathematics Korea had 40% advanced, Japan had 26% advanced while the United States only had 6% advanced. If you look to see how many students that represents in the population, both counties still out rank the United States in absolute numbers. That is because they have such a huge percentage of high achieving students.
Even if Dr. Salzman were right, we all know how the story of David and Goliath turned out. The lesson from
that story is that brains trump brawn and we cannot rely on size.
2:46 Hal Salzman: Economic performance is not closely linked to education performance at the top. A broadly
educated poplus is the most important we contend, so problems in education at the bottom are much larger
worry. For more on this,
One short overview article appeared last year in Nature:
Salzman, Harold and Lindsay Lowell. (2008). "Making the Grade." Nature. 453, 28-30.
http://www.nature.com/nature/journal/v453/n7191/pdf/453028a.pdf
and
Lynn, Leonard and Harold Salzman.(2007). "The Real Global Technology Challenge." Change.
July/August
2007, 9-13.
2:47 [Comment From Barb Kapinus]
I believe it was Andreas Schleicher who said that the Us had one of the higest correlations of achievement
to
SES on PISA. This could be the source of a nother reason other countries outperform us. Their poor and
minority students have access to better haelth and social services. We did not participate in the background
part of PISA last time. Would it help determine some of the factors impacting differences in achievement if
we
did participate in that part of the study next time?
2:47 Gary W. Phillips: I think it a combination of both. Basically we remain in the middle of the
race.
2:48 Hal Salzman: We should take some great comfort in the not widely reported test results showing the US
is
one of the only nations that shows consistent improvement across all subjects (math, science, reading), all
years, all grades.
2:49 Gary W. Phillips:
Lack of participation by the U.S. is a chronic problem. We are a large and wealthy nation so we should
participate in just all of these suevys.
2:50 Hal Salzman: SES and other factors are significant factor explaining the differences in outcomes.
Moreover, 90% of the scores differences are within country; between country differences account for only 10%
of the variance.
2:50 [Comment From Peggy]
What are some of the best practices in education that nations who outperform the U.S. are using?
2:51 Gary W. Phillips: This is a great question and will be explored thoroughly by the "Race to the
Top" initiative. I will wait to see what we find out.
2:53 Kathleen Kennedy Manzo: Here's another angle from Emma
2:53 [Comment From Emma White]
Have researchers studied the psychological well-being of the students in different countries? It seems the
children in the high performing countries spend their childhoods in an academic pressure cooker of delayed
gratification for the sake of future professional achievement at the expense of developing a well-rounded
human being who has a broader range of interests and abilities. Even though US children are less
disciplined,
most of them go on to do reasonably well as responsible adults. Perhaps our system takes into account the
developmental stages of a person. Of course we should continue to strive to improve our system. But perhaps
the strengths in our system is less quantifiable.
2:54 Hal Salzman: Emma asks a great question. There have been some studies on this issue -- they are far
from
conclusive, but in S. Korea, for example, youth suicide is a quite significant problem and some studies
there
link it to the pressure placed on youth.
2:56 Gary W. Phillips: I do not believe there is an easy answer as to why some countries do well and others
do
not. This needs to be better researched in the future.
2:56 Kathleen Kennedy Manzo: Gary: Can you describe in more detail the Race to the Top initiative you
mentioned?
2:58 Gary W. Phillips: This is an initiative from the Obama administration that is intended to fund state
efforst to recah challenging performance standards and reduce the variablity of existing standards across
states. The details are not yet available but will be made public sometime in June.
2:59 Kathleen Kennedy Manzo: Hal, aren't some of the high-performing countries also sending teams of
educators here to study our best practices? I recall when I visited Japan five or so years ago, they were
revising the curriculum and changing some teaching practices to a more Americanized version that emphasized
critical thinking, creativity, collaboration... those things that are now being described as 21st Century
Skills. I know Singapore and China have also expressed interest in this approach.
3:00 Kathleen Kennedy Manzo: Gary, Barbara has a question about the length of the school year. I know that
the
Obama Administration has talked about extending school time and after school programs. Can you answer
Barbara's question?
3:00 [Comment From Barbara Moody]
There doesn't seem to be a clear correlation between a longer school year and high achievement on
International tests. For example, Finland has exceptionally high performance but has a school year
equivalent
to the US while some Asian countries have high achievement but many more school days. What does the data
say?
3:01 Hal Salzman:
As you note, Singapore has a large creativity initiative in which the US is the focus. Silicon Valley is
the
goal of about every nation in the world. The iPod another example of US advantage. So, yes, there are many
distinct US advantages and it would be a shame to lose those advantages through too much emphasis on test
scores that probably don't capture what have been the unique US competitive advantages.
3:02 Kathleen Kennedy Manzo: Here's a great question to end with, from Nick, for each of you.
3:02 [Comment From Nick]
Standardized tests face great scrutiny in the U.S. These international tests are being looked at to possibly
adjust U.S. education policy. What arguement can you make towards using these test to help shape our
education
policies, to someone who strongly opposes these type of test?
3:02 Gary W. Phillips: This true. However, you are comparing the length of school years between countries.
As
Dr Salzman said 90% of the variance is within countires. Therefore the length of the school year may not
make
a difference between countries but it might within a country.
3:04 Hal Salzman: The tests, in my view, probably do the greatest good by raising the types of questions
we've seen today. It focuses attention on education and, one hopes, leads to creative solutions. It is a
mistake to think testing well is the only goal, but they do serve a good purpose if used well and in
context.
3:05 Gary W. Phillips: We need this type of information. It gives the country an international context
within
which to think about policy.
3:07 Kathleen Kennedy Manzo: Ok, that'll be the last word. This has been a great discussion. Thank you
to
both of our panelists for their insights, and thanks for all the great questions! A transcript of the chat
will be available momentarily on this same page.