The Role of International Tests in U.S. Education

Why do U.S. students so often do rather poorly among developed nations in international tests.

The most interesting question and answer is this one:

2:03 Kathleen Kennedy Manzo: Gary, can you answer this question from Olga?

2:03 [Comment From Olga Amaral] Is there one common factor that has been found to be most influential in contributing to the success of countries with highest achievements?

MOTIVATION

2:05 Gary W. Phillips:
I think we do not have a final answer to this question. But it seems to me to be related to how well education is respected and valued within the society. Those countries that care the most about education achieve the highest. So--If American kids think high school is a joke, they won't take their schooling seriously.
Possiblities: American high school *is* a joke. High schools everywhere are a joke, but Americans are more willing to admit it. American culture makes them less willing to take gravely what the authorities want them to take gravely.
There is hardly anything the Federal government can do to change basic attitudes about school, much less increase respect for authority generally. (What is the optimal respect for authority, anyhow?) Ask instead how schools might change to make students more eager to learn. Since it is not possible for the central government to fiddle with what individual teachers and schools do, think about how "adequate yearly progress" might be *variously* assessed at individual schools depending on what each one *now* tries to accomplish that gets away from one-size-fits-all for every state. That's too finely tuned, each school. Try counties. Lots of good thoughts in these exchanges. Read with profit.

Education Week: The Role of International Tests in U.S. Education
Asians Best U.S. Students in Math and Science

1:30 Web Person: Jennifer: Today's chat on the role of international tests in U.S. education is now open for questions, so please start submitting them. Our moderator and guests will here at 2 p.m. to begin answering your questions. Thanks for joining us.
1:58 Kathleen Kennedy Manzo: Hi everyone. Thanks for joining us. Let's begin the chat.

Why don't our guests, Hal Salzman and Gary Phillips, introduce themselves first, and then they'll start answering your questions.
1:59 Hal Salzman:

This is Hal Salzman - I'm Professor of Public Policy and sociologist at the Bloustein School of Public Policy and the Heldrich Center for Workforce Development at Rutgers University. With colleagues, I have been examining the science and engineering workforce “demand” and “supply”, which led us to look closely at the education performance of U.S. students, finding that our best students are actually doing quite well, contrary to the widely-cited proclamations of poor performance.
2:00 Gary W. Phillips:

I am Gary W. Phillips the Vice President & Chief Scientist at the American Institutes for Research (AIR). I direct a team of psychometricians and statisticians on large scale assessment projects. I recently published several reports that involve statistically linking NAEP to TIMSS. The first report compared U.S. state-NAEP results to TIMSS in math and science (Chance Favors the Prepared Mind, 2007), the second compared U.S. School District results to TIMSS (Counting on the Future, 2008) and a new report is about to be released that compares states and school district to TIMSS using letter grades A, B, C, D and F. (International Benchmarks in Mathematics, 2009). Prior to coming to AIR, I was the Acting Commissioner at the National Center for Education Statistics.
2:01 Kathleen Kennedy Manzo: Thank you to both of you for sharing your insights and your expertise on this topic.
Can each of you briefly characterize your views on the value of international comparison tests, and what the results do and do not tell us about how our schools are doing?
2:03 Hal Salzman:

There is certainly value in comparing schools and education systems to learn from others. However, the conclusions often drawn from these results go far beyond what the actual data support. U.S. academic performance at the top is actually quite good comparatively, as the most recent tests show.

2:03 Gary W. Phillips: International assessments give the United States a broader context within which to understand our own student learning and teaching practices. They do not provide a silver bullet but they add to the knowledge base that helps us decide on best practice.
2:03 Kathleen Kennedy Manzo: Gary, can you answer this question from Olga?
2:03 [Comment From Olga Amaral]
Is there one common factor that has been found to be most influential in contributing to the success of countries with highest achievements?
2:05 Gary W. Phillips:

I think we do not have a final answer to this question. But it seems to me to be related to how well education is respected and valued within the society. Those countries that care the most about education achieve the highest.
2:06 Kathleen Kennedy Manzo: Here's one Hal from Nick
2:06 [Comment From Nick]
Shouldn't we be more worried about U.S. graduation rates, and performance in local schools, rather then worrying about American scores on international tests? Why is it that international test are being looked at to help shape our education policies? Shouldn't we be looking towards better training for teachers and focus on curriculum rather then focusing on changing policies?
2:07 Hal Salzman:

I would agree and we've argued that we have more to learn from high performing districts and states (e.g., Massachusetts and Minnesota) than from Finland -- if only beause of greater transferrability to US context.

2:07 Gary W. Phillips: All of these factors are important to look at. We should be looking at graduation rates, achievement as well as better taecher traing and improved curriculum.
2:07 Kathleen Kennedy Manzo: Gary there are a couple of questions about the comparisons themselves, when the tests are given, the comparability of groups.
I'll post one or two for you.
2:07 [Comment From Robert D. Ferrari]
How are the comparison groups of various countries selected? Are they representative groups based on the same set of criteria?
2:08 [Comment From Stuart Hobbs]
My question concerns differences in educational goals between the US and many other countries that makes the comparisons on international tests (which I find interesting and a good idea) somewhat problematic. Specifically, in the US we are trying, more or less, to educate everyone to the same level, while in other places by our equivalent of middle school students have been separated on academic or vocational tracks. If that is so, won't that mean that while we can certainly improve on our test scores, we will always be running behind other countries if their test results include only the academic track and ours include everyone?
2:08 Hal Salzman: The testing groups try to use comparable samples but the reality of a large, complicated study is that samples often are not comparable. In the PISA, sampling differences may account for a significant portion of the outcome differences.
2:09 Gary W. Phillips: Each sample in each country is a scientifically representative sample. An international sampling referee makes sure that each country complies with sampling requirements.
2:10 Hal Salzman: Gary, I would agree that they try to do so, and make a good effort at it, but they don't always succeed. There are substantial differences in the final sample and the U.S. sample in particular has been a problem in several of the tests. In our papers we discuss in detail the impact of sampling problems.
2:11 Gary W. Phillips: I disagree with Dr Salzmans conclussion. If there is one thing that we can trust in these studies it is that the sampling is done well. The minor sampling problems that due emerge due no make much difference.
2:11 Kathleen Kennedy Manzo: Can either of you answer this question from Kris?
2:11 [Comment From Kris Palfrey]
What countries are achieving the highest scores and what examination boards are they using?
2:13 Kathleen Kennedy Manzo: In this week's Education Week we have a series of articles on other countries that are doing well and some of the factors that might be contributing to their high performance on international assessments.
www.edweek.org
2:13 Gary W. Phillips: The highest achieving countries are Chinese Taipei, South Korea, Singapore, Hong Kong and Japan and Finland. Each has a different exam system.
2:15 Kathleen Kennedy Manzo: Hal, can you address this question from Jerry?
2:15 [Comment From Gerald W. Bracey]
Given the Japan scores among the highest nations in the world but has seen 20 years of economic recession and stagnation, and given that high scoring Iceland became an overnight economic basket case with a national debt equal to 850% of its Gross Domestic Product, and given that middle-scoring United States has long been ranked first in global competitiveness by both the Institute for Management Development and the World Economic Forum, why should anyone pay any attention to these test scores?
2:17 Hal Salzman: There are two questions here --- is there anything to be learned from educational testing and is economic performance the primary outcome that we should be interested in. There is a qualified yes to the first and, as you make clear in the question, economic performance is due to many other factors. It is, quite obviously, not reasonable to focus on education as the primary driver of economic performance.
2:18 Kathleen Kennedy Manzo:

Just some background on these tests and the results, from my colleague Sean Cavanagh:

American students have fared much better on one international test, the TIMSS, than on another PISA. There's a lot of speculation about why. The TIMSS, like the domestic NAEP test, measures school-based curriculum. The PISA measures “yield,” or the skills that students have picked up in and out of school, and their ability to apply them to real-world contexts.

For PISA results: http://nces.ed.gov/pubs2008/2008016.pdf

And TIMSS: http://nces.ed.gov/timss/results07.asp
2:18 Kathleen Kennedy Manzo: Gary, you have done some studies linking the scores for international tests on a scale with NAEP results. What did you find?
2:20 Hal Salzman:

As additional follow up to Jerry's earlier question, it is also worth noting that the U.S. tops the scale on reading and other subjects. It is arguable that only math and science are the most important subjects.
2:20 [Comment From Milad]
Are there alignment studies between TIMSS and international tests?
2:20 Gary W. Phillips:

This observation is an artifact of how the two surveys calculate the international average to which we compare the United States. PISA uses the 30 OECD countries so our standaing relative to them is lower. That is why we appear to do less well on PISA than on TIMSS.
2:21 Kathleen Kennedy Manzo: Anne is referring to efforts by states to rewrite state standards using international benchmarks. Can one of you answer her question?
2:21 [Comment From Anne]
2. We are hoping our state will rewrite its standards, using international standards as benchmarks. What other factors should we consider?
2:21 Gary W. Phillips: There are content aligment studies between all the international assessments and they can be found by going to NCES web.
2:23 Gary W. Phillips: There "is" a way to internationally benchmark state achievement standards using PISA, TIMSS and PIRLS. I am doing a workshop on this with state testing directors the first week in May.
2:24 Kathleen Kennedy Manzo: This is a hot topic right now.

Two influential organizations, the National Governors Association and the Council of Chief State School Officers, in December called for setting common state standards in reading and math and benchmarking standards in high-performing countries.( See Ed Week's story on a report issued by the two organizations, at http://www.edweek.org/ew/articles/2008/12/18/16nga.h28.html)

And just last week, representatives of 41 states convened at a conference in Chicago to explore those issues. The event was co-sponsored by the NGA and CCSSO. Efforts to promote national standards have been promoted, and fizzled, in the past. Will this new effort overcome that fate, and what role would international benchmarks play, if any?

http://www.edweek.org/ew/articles/2009/04/16/29standards.h28.html

Just yesterday the House Education and Labor Committee announced a hearing, set for April 29, on “to examine how states can better prepare their students to compete in a global economy by using internationally benchmarked common standards.”
2:24 Kathleen Kennedy Manzo: Hal, what do you think these efforts to identify international benchmarks and align state standards accordingly?
2:26 Hal Salzman:

This, of course, raises the question about what we think is important for competing globally. Presumably it is performance after they leave school and not clear that the U.S. is at a disadvantage. We still lead the world in innovation, R&D, etc. Perhaps there is something about our education system that leads to the US being the world's leaders in science, technology, innovation, entrepreneurship. Moreover, it is our universities that seem to produce the world's best graduates.
2:26 Kathleen Kennedy Manzo: Gary, Patti has a question about participation on these tests, particularly for India and China.
2:26 [Comment From Patti]
Is it possible to know how U.S. students compare academically with students from mainland Chinese and India--because those countries don't participate in the PISA or TIMMS?
2:27 Hal Salzman: If one factors in the 30-50% illiteracy or near-literacy rates of those countries, their averages would be pretty close ot the bottom.
2:28 Gary W. Phillips:

We do not know how the U.S. compares to these two countries. However, I believe PISA plans to include both countries (or at least a portion of both countries) in their next assessment.
2:29 Kathleen Kennedy Manzo: This next question is for Gary, regarding your earlier response on high-performing countries
2:29 [Comment From Robert Pondiscio]
I'm intrgued by your response that "those countries that care the most about education achieve the highest." How are you defining caring? Economically? Or through some manner of value judgement?
2:29 Kathleen Kennedy Manzo: Hal, can you answer Karen's question?
2:29 [Comment From Karen]
In the Education Week article, Mr. Salzman, you mention that many U.S. students fall out of the math and science pipeline. Is that a problem for these other high-performing countries as well?
2:31 Kathleen Kennedy Manzo: I have a better link for the article Karen is referring to, as well as all of Education Week's recent coverage of this issue over the last year, in commemoration of the 25th anniversary of the A Nation at Risk Report

http://www.edweek.org/go/nar
2:31 Gary W. Phillips: If you look inside the cultures of the high achieving countries you will see that teachers are well respected, teaching is a competitive job, parents are involved with school and students take school seriously.
2:31 Hal Salzman: We don't know because we haven't tracked enfollments in those countries. We do find many engineers entering management, for example, in Japan and Germany, so there is some career mobility. Overall there tends to be more tracking by discipline.
2:32 Kathleen Kennedy Manzo:

Here's some more background from Sean Cavanagh:

In a 2007 paper, Hal Salzman and Lindsay Lowell, of Georgetown University, argued that the United States K-12 system produces an ample number of students with skills necessary to compete in science and engineering fields, contrary to widespread belief. American students' math and skills are equal and in some cases better than they were two decades ago, they conclude. The problem lies in graduate programs and science- and engineering-related companies, which are ineffective at recruiting talented students and convincing them to stay. http://www.edweek.org/ew/articles/2007/11/07/11report-b1.h27.html
2:33 [Comment From Louise Johnson]
Would the formation and implementation a national curriculum have an affect upon international assessments? If so, what are your opinions?
2:33 Gary W. Phillips:

Hal Salzman believes that in raw numbers, the U.S. produces many more top-tier students. This is only partially true. One example where it is not true is grade-8 mathematics. In an analysis of 2007 TIMSS grade-8 mathematics Korea had 40% advanced, Japan had 26% advanced while the United States only had 6% advanced. If you look to see how many students that represents in the population, both counties still out rank the United States in absolute numbers. That is because they have such a huge percentage of high achieving students.
2:34 Kathleen Kennedy Manzo: Sorry, I jumped the gun on that last question. But I think Louise is asking what others are wondering as well, particular as policymakers here debate this issue. Can either of you chime in >?
2:35 Hal Salzman:

It raises the question of what are we measuring and why. I'm not sure that Japan's economy has benefitted from its production of high performing math students. Maybe the reason the US is the world's leading innovator is because of its diversity and produces broadly educated students?
2:35 Kathleen Kennedy Manzo: Louise, I want to refer you to my story this week about Australia's effort to create a national curriculum. It has a lot of traction after a very similar history as ours with national standards development

http://www.edweek.org/ew/articles/2009/04/22/29nar-australia_ep.h28.html
2:36 Gary W. Phillips: It is hard to know if a national curriculum would help. I think it probably would. Howver, most of the high achieving countries have national curriculums but so do most of the low achieving countries.
2:39 [Comment From Ty]
Dr. Salzman indicated earlier that 'US academic performance at the top is actually quite good.' Earlier TIMSS studies compared just the top US students with the top students from other countries and I believe that even US AP students scored below average compared with the top students from other countries. Can you confirm and comment?
2:40 Gary W. Phillips: This is correct. In the Advanced TIMSS study the United States still did not compare well with the other countries.
2:41 [Comment From Spencer Siegel]
Which states in the U.S. perform the best and worst on international exams? Do some states do just as well as some of the top-performing countries?
2:42 Hal Salzman:

The recent TIMSS showed the two states that paid for larger samples so they could be compared internationally came out at the top. There are sampling issues in these small sample comparisons such as AP only students. Also question about who takes the tests, etc. Interpreting small differences goes far beyond what the international testing methods will reasonably support.
2:42 Kathleen Kennedy Manzo: Good question Spencer. Massachusetts and Minnesota took the TIMSS in the last go around and did nearly as well as some of the highest performing countries.
Here is our story on it: http://www.edweek.org/ew/articles/2008/12/09/16timss.h28.html
2:42 Gary W. Phillips: The only states that took an international exam recent were Mass. and Minn. Both scored very well on the exam.
2:43 [Comment From Mike Bouldin]
Hal said that "our best students are actually doing quite well...." Is that enough for our society to continue to function as a vibrant democracy with a robust economy? Or are there factors that will in the future require us to have a greater proportion of our students be high achievers? I'd like to hear both Hal and Gary address this.
2:44 Hal Salzman: One also needs to note that the US population is far more diverse, and comes to schools with much greater educational challenges than in any of the other top performing schools. such as not speaking native language at home, singleparent household, etc.
2:45 [Comment From Anne]
1. Is the problem that U.S. achievement is dropping, that other countries have caught up and moved past the U.S., a combination of these, or something else?
2:46 Gary W. Phillips:

Let me repeat my statement from before. As I stated above Dr Salzman believes that in raw numbers, the U.S. produces many more top-tier students. This is only partially true. One example where it is not true is grade-8 mathematics. In an analysis of 2007 TIMSS grade-8 mathematics Korea had 40% advanced, Japan had 26% advanced while the United States only had 6% advanced. If you look to see how many students that represents in the population, both counties still out rank the United States in absolute numbers. That is because they have such a huge percentage of high achieving students.

Even if Dr. Salzman were right, we all know how the story of David and Goliath turned out. The lesson from that story is that brains trump brawn and we cannot rely on size.
2:46 Hal Salzman: Economic performance is not closely linked to education performance at the top. A broadly educated poplus is the most important we contend, so problems in education at the bottom are much larger worry. For more on this,

One short overview article appeared last year in Nature:

Salzman, Harold and Lindsay Lowell. (2008). "Making the Grade." Nature. 453, 28-30.

http://www.nature.com/nature/journal/v453/n7191/pdf/453028a.pdf

and
Lynn, Leonard and Harold Salzman.(2007). "The Real Global Technology Challenge." Change. July/August 2007, 9-13.
2:47 [Comment From Barb Kapinus]
I believe it was Andreas Schleicher who said that the Us had one of the higest correlations of achievement to SES on PISA. This could be the source of a nother reason other countries outperform us. Their poor and minority students have access to better haelth and social services. We did not participate in the background part of PISA last time. Would it help determine some of the factors impacting differences in achievement if we did participate in that part of the study next time?
2:47 Gary W. Phillips: I think it a combination of both. Basically we remain in the middle of the race.
2:48 Hal Salzman: We should take some great comfort in the not widely reported test results showing the US is one of the only nations that shows consistent improvement across all subjects (math, science, reading), all years, all grades.
2:49 Gary W. Phillips:

Lack of participation by the U.S. is a chronic problem. We are a large and wealthy nation so we should participate in just all of these suevys.
2:50 Hal Salzman: SES and other factors are significant factor explaining the differences in outcomes. Moreover, 90% of the scores differences are within country; between country differences account for only 10% of the variance.
2:50 [Comment From Peggy]
What are some of the best practices in education that nations who outperform the U.S. are using?
2:51 Gary W. Phillips: This is a great question and will be explored thoroughly by the "Race to the Top" initiative. I will wait to see what we find out.
2:53 Kathleen Kennedy Manzo: Here's another angle from Emma
2:53 [Comment From Emma White]
Have researchers studied the psychological well-being of the students in different countries? It seems the children in the high performing countries spend their childhoods in an academic pressure cooker of delayed gratification for the sake of future professional achievement at the expense of developing a well-rounded human being who has a broader range of interests and abilities. Even though US children are less disciplined, most of them go on to do reasonably well as responsible adults. Perhaps our system takes into account the developmental stages of a person. Of course we should continue to strive to improve our system. But perhaps the strengths in our system is less quantifiable.
2:54 Hal Salzman: Emma asks a great question. There have been some studies on this issue -- they are far from conclusive, but in S. Korea, for example, youth suicide is a quite significant problem and some studies there link it to the pressure placed on youth.
2:56 Gary W. Phillips: I do not believe there is an easy answer as to why some countries do well and others do not. This needs to be better researched in the future.
2:56 Kathleen Kennedy Manzo: Gary: Can you describe in more detail the Race to the Top initiative you mentioned?
2:58 Gary W. Phillips: This is an initiative from the Obama administration that is intended to fund state efforst to recah challenging performance standards and reduce the variablity of existing standards across states. The details are not yet available but will be made public sometime in June.
2:59 Kathleen Kennedy Manzo: Hal, aren't some of the high-performing countries also sending teams of educators here to study our best practices? I recall when I visited Japan five or so years ago, they were revising the curriculum and changing some teaching practices to a more Americanized version that emphasized critical thinking, creativity, collaboration... those things that are now being described as 21st Century Skills. I know Singapore and China have also expressed interest in this approach.
3:00 Kathleen Kennedy Manzo: Gary, Barbara has a question about the length of the school year. I know that the Obama Administration has talked about extending school time and after school programs. Can you answer Barbara's question?
3:00 [Comment From Barbara Moody]
There doesn't seem to be a clear correlation between a longer school year and high achievement on International tests. For example, Finland has exceptionally high performance but has a school year equivalent to the US while some Asian countries have high achievement but many more school days. What does the data say?
3:01 Hal Salzman:

As you note, Singapore has a large creativity initiative in which the US is the focus. Silicon Valley is the goal of about every nation in the world. The iPod another example of US advantage. So, yes, there are many distinct US advantages and it would be a shame to lose those advantages through too much emphasis on test scores that probably don't capture what have been the unique US competitive advantages.
3:02 Kathleen Kennedy Manzo: Here's a great question to end with, from Nick, for each of you.
3:02 [Comment From Nick]
Standardized tests face great scrutiny in the U.S. These international tests are being looked at to possibly adjust U.S. education policy. What arguement can you make towards using these test to help shape our education policies, to someone who strongly opposes these type of test?
3:02 Gary W. Phillips: This true. However, you are comparing the length of school years between countries. As Dr Salzman said 90% of the variance is within countires. Therefore the length of the school year may not make a difference between countries but it might within a country.
3:04 Hal Salzman: The tests, in my view, probably do the greatest good by raising the types of questions we've seen today. It focuses attention on education and, one hopes, leads to creative solutions. It is a mistake to think testing well is the only goal, but they do serve a good purpose if used well and in context.
3:05 Gary W. Phillips: We need this type of information. It gives the country an international context within which to think about policy.
3:07 Kathleen Kennedy Manzo: Ok, that'll be the last word. This has been a great discussion. Thank you to both of our panelists for their insights, and thanks for all the great questions! A transcript of the chat will be available momentarily on this same page.