NRT and CRT Test Comparisons
Transcript of Walter Hussman
On Testing, NRT, CRT, and Test Scores
To Arkansas Joint Committee on Educational Adequacy
August 12. 03
My name is Walter Hussman, and I am representing myself as a life long citizen of Arkansas, publisher of the Arkansas Democrat Gazette; but I am really here as a citizen, very concerned about our educational system. Several people have said, "How did you get so interested in public schools?" For a long time I didn't have the interest that I have had the last year. I guess Lakeview was one of them. My wife is another. She got me interested in this. And frankly, my association with our newspaper over in Chattanooga got me interested because I saw in Chattanooga something I didn't think could happen. I saw them take the worst schools in Chattanooga with almost all minority low income kids and make unbelievable progress with those kids in two years...
One thing I heard today which I thought was fantastic and that was when the teachers' union, Sid Johnson and his people, said they are in favor of longitudinal tracking which is something the Arkansas business community is absolutely in favor of. Maybe Sid and I can go over to the department of education and see if somehow we can persuade them that it is a good idea.
I want to commend Dr. Brown and Dr. Fisher today on what they had to say. It is very encouraging. In the business community when we have a business problem, what we try to do is look around the US to see if somebody else has already solved the problem. I have done that in the newspaper publishing business. I wouldn't be in the newspaper publishing business any more if I had not done that. And so that is what a number of business people in Arkansas have done in this year. We have gone around the US and tried to find who is having success, and two states that have a lot of success are Florida and Tennessee. I am so pleased that somebody from Florida and Tennessee were here today to explain to you what they were doing.
House bill 2528 in the last session was endorsed by all the major companies in Arkansas, Murphy Oil, Dillard's, Alltel, J. P. Hunt, Southwestern Bell, Entergy. I could go on and on. All the major companies endorsed that bill. The Little Rock Chamber of Commerce endorsed it, the the Arkansas State Chamber of Commerce endorsed it, and Northwest Arkansas Council endorsed it. . It was endorsed by Governor Huckabee. Even Ray Simon testified that he was in favor of the bill. A lot of that bill is based on what they did in Florida, and a lot of the components of that bill are the things that they have done in Tennessee. I know a little bit more about Tennessee than I do Florida. I have spent a lot more time over there. They have some of the just basic building blocks of accountability that we just donít have in Arkansas.
They have basic building blocks like annual testing. They have been doing this for ten or 12 years, annual testing. We donít have that in Arkansas. We are going to go to it, got to go to it because the federal government is forcing us to go to it, and we arenít going to go to it a minute too soon. We are going to it in 2005 the year the federal government is forcing us to do it. But anyway, in Tennessee they have been doing it for years. A second basic building block they have in Tennessee is nationally normed test. We have those in Arkansas for a few grades, but we are moving from that in Arkansas which I think is a big mistake. Another thing they have in Tennessee is longitudinal tracking which is basically following every single kid every single year, and you all know what that is. Now longitudinal tracking is not the goal. Longitudinal tracking is just a measurement device to get us to the goal of the value added system. I understand they have that in Florida, and I know they have it in Tennessee. The state of Pennsylvania has adopted value added system.. A value added system, the business community in Arkansas is convinced that a value added system is the way to go and I have learned. I have learned a lot about some of this. I am going to learn a lot more I guess in education.
But one of the things I have learned is that there are two kinds of value added systems. Thereís a kind of value added systems like they use in the state of North Carolina, and they use in the state of Tennessee. That is one kind. Then there is a different kind of value added system, and that is the kind they use in Pennsylvania and the kind they are going to use in Ohio. These systems are different.
Let me try to explain what I understand about how those two systems are different. In Tennessee and in North Carolina what they use is a nationally normed referenced test. For example, this is the way I can best understand it. In Tennessee they use a Terra Nova test. The Terra Nova test is given in 15 to 20 states. Millions of kids all over the US take the Terra Nova test. So because the Terra Nova test is vertically scaled, which our CRT in Arkansas is not, but because it is variably scaled, the kids in the 4th grade will score at a certain level, say they score 350. Well, that is a national average of all the kids in the US who took the Terra Nova test last year. They may score say 350. The next grade up, say the 5th grade, they will score at say 425. I am just picking those numbers out of the air and just using them as an illustration. The 5th graders nationwide will score higher on the Terra Nova test than 4th graders will. And so if the average point gain is say 75 points from the 4th grade to the 5th grade, that is how they know that the child has made a typical average year gain. And so for example in Chatanooga, and Dan will tell you more about this, they expect all their kids to have at least a 75 point gain, the national average. And if they do, they gain one full year. For example, in Chatanooga, if they gain 15% more than the national average, then that is great and they reward them for that.
So that is a kind of easy value added system to understand. It is based on a national average. You have to test the kids each year to do that. You use national averages to do it.
Now, a different kind of value added system is what they are using in Pennsylvania. They are going to this in Pennsylvania, and they are using this in a number of school districts in Ohio. In Pennsylvania they didnít test every year. In Pennsylvania they didnít use a nationally normed test. So how are you going to do a value added system? Well, it is not as easy and it is probably not as good but you can do it. And the way they do it is basically they take a look at all the childrenís academic records and for each student they say well we expect based on their past academic record and there may be some other information maybe demographic information, etc. Here is what we expect that child to be a year from now. Here is what we expect they will score on a test a year from now. And that is going to be the expectation. So when the student actually takes the test a year from now, they will compare that to what was expected of the student. So if he scores above the expected level, he had a year of gain. If he scores less, he had less than a year gain.
(This paragraph was in response to Senator Argue who asked ďWhy couldnít we use the Tennessee model and use the state average?) I think you can see in those two models and apparently you can use both models. I tend to have a little more confidence in the Tennessee/North Carolina model because it is based on quantified numbers. It is based on a national average of all the other students in the US who took that test Ė how much they gained. This is one of the reasons why the business community in Arkansas feels that a nationally normed test is very important.
First, of all, you could use a state average instead of a national average on an NRT for example. or you could use a school district average to see if some students were performing as well as others. So it doesnít have to be a national average. You could use any average. I am told by people who know a lot more about education than I do that it is very difficult to track how students do unless the test is vertically scaled.
So let me talk a little bit about the testing we do here in Arkansas.
Yesterday, I think the state of Arkansas took a very big step backward when they de-emphasized the nationally normed test, and now we are going to give it only two years instead of three years as we did in the past. In the past the nationally normed test was the basis of accountability; now it has nothing to do with accountability, and so I am discouraged to see that. Let me tell you why.
First of all, a criterion referenced test like Arkansas benchmark test, the theory behind that is a solid theory. It is a great theory. When Arkansas went to that, it was a good theory. I am going to tell you why I think it is a good theory because the idea the state is supposed to develop their own academic standards. Once they develop those academic standards, then what they do is develop a curriculum to teach those standards. And after they develop that curriculum, then they develop tests in order to see if the students have learned the curriculum. And so it is a great idea. The problem we have in the business community with the way this works in Arkansas is that we think our standards are woefully inadequate.
There is an organization called Fordham Foundation. They are the only organization I know of that does this only on academic standards. They read the standards for all 50 states Ėall 50 states. They read everyone of them. When they do that, they give every state a letter grade. The last time they did it was in 2000. They did it in 1998 so I imagine another one is coming up pretty soon. But Arkansasís standards got an F. Only five states got an F. Five out of 50 states got an F. Arkansas is one of them. Mississippi got a C-; Louisiana got a C-; Texas got a B. Arkansas got an F.
So why did Arkansas get an F? Well, I was going to show you if this power point presentation would work. Let me just read you from this.[Technical problems kept him from presenting some of this information the way he planned.] This is on their website and they explain why they gave us an F. It isnít they think our standards are really bad; they think are standards are really vague. For example, they say they are not specific enough to be useful. They are too broad. They are broad philosophical goals such as ďStudents understand the goal of reading is to construct meaning.Ē Now, their idea [Fordham Foundation] of a good standard is, You teach Shakespeare in the 10th grade by teaching Hamlet and McBeth. There is a specific standard. The idea of learning to read and construct meaning is not a very specific standard. If we had a good grade in our standards in Arkansas, I think the business community would feel much better about our state benchmark test. But since we get an F by outside party, we are very concerned about that.
So who is the Fordham Foundation. I know that the people at the Department of Education has tried to disparage them. I would probably try to do the same thing if I worked up there. But it is a foundation; they have nothing against the state of Arkansas. They look at the standards in all 50 states; Chester Finn is the head of that foundation; He is on the K-12 Education Task Force. He is one of the top eleven educational people in the country, and the Department of Education may tell you people here in the legislature; ďDonít believe it; we get good grades from other people and that there is something in education that gives us a blending grade on accountability and standards that where maybe we get a C. Well, I would give Arkansas very high marks for accountability; Arkansas is one of the few states in the country where every single student in the state has to take some kind of accountability test since the start of assessment every year.I am talking about just the academic standards, not anything that is blended.
So in conclusion we think until our standards are better we have got to give an NRT. And we think giving it every year is important.
I was told by Ken James the former superintendent of the Little Rock Schools that if he canít give the NRT in Little Rock it is going to thwart the desegregation efforts here. He is going to lose white parents who are going to take their kids out of the public schools and put them in private schools because they want to know how their kids do against the kids in Dallas, Memphis, and St. Louis; and they donít want to know just how their kids do against kids in Fordyce, Ft. Smith, and Fayetteville and Forrest City.
So I think a nationally normed test is important. Ideally it can be a single test; it can be an augmented test. You have heard testimony today that is what they are doing in Tennessee. It works well over there. Why canít we do it in Arkansas? The Department of Education is absolutely opposed to it. They donít want to do it. I am sorry, but if they donít want to do this, we would be happy with having two tests. But if I were able to show you this presentation, what I could show you is that we went back and graphed the Stanford 9 and Stanford 8; befsore it was the Stanford 8 it was the MAT-6. We went all the way back to 1988 and we graphed the 4th grade (which actually turned into the 5th grade the last five or 6 years); we graphed the 7th grade and we graphed the 10th grade. And what you are going to see if you saw those graphs is that they [nationally normed tests scores] are basically flat.
We have made a little progress in the last few years Ė from 1996 to 2001. I think our Stanford 9 test scores probably went up from like 45% to 51% [4th grade] State spending on K-12 education went from a 1.4 billion to 2.8 billion. You know that is actually unbelievable. In six years we have doubled the spending on K-12 education in Arkansas. And we have about a 10% increase in our test scores. Also, Jim Argue brought this point up earlier. In 1996 Ė 2001, you look at the 4th grade for the Stanford 9. I think we went from a bout the 45% to 51%. That is really the first progress we made in Arkansas since 1988 on the Stanford test scores.
The other thing if we look at this Ė Jim Argue brought this up Ė if you compare the percent of kids scoring on the CRT, the Arkansas benchmark test, you are going to see a slope that looks like this and you compare that on the graph where the kids have gone from 35% maybe to the 67%. Yet, you look at the Stanford 9 test scores, they are relatively flat. (Looking at graphs on screen.)
Here when you look at the same benchmark scores from 1998-99 we have gone from the first 44% above proficient to 67% scoring above proficient. You would think education is doing great in Arkansas.
Next slide, if you look at this, (I have copies of this if anyone would like it) this is 4th grade reading on the ACTAAP [Benchmark] scores, we found a 44% to 69%. Gee, things are great. What is wrong with the Supreme Court? Donít they know how great the academics are in Arkansas?
Next slide. But if you compare it to NAEP  , the state says 65% of students are proficient on the ACTAAP [benchmark], but NAEP says only 29% are proficient.
(Interruption by Mahoney)Mahoney: Let me see if I get this straightened out a little bit> . We have heard testimony today that proficient in the state of Arkansas is not the same as proficient in NAEP. (Someone else said AMEN and loud laughter followed) What I think the problem may be; I think we may need to change the name of what we are talking about. As I understand proficient in Arkansas is the same as Basic on NAEP. Am I right on that Charity? (Charity Smith from ADE who heads up accountability) We have got two standards, one called proficient over here and one called proficient over here. I am hearing that what is called proficient over here is only Basic over here and I mean. I would like an explanation of that. I can tell you from my viewpoint I suspect that the idea of doing grades [Letter grades, A, B, C, D, for schools as in 2528 I suppose Mahoney means) as set forth today particularly with this problem with what proficient means over here and what proficient means over there. That does not promote understanding by that motherof two back there or by the business community or the public.
Hussman again:Jody, I think what you will find that Arkansas is not unique and I am not trying to cast aspersions on the Department of Education. You have heard of grade inflation in America. Today what we have is CRT inflation. It is true in all the states. They are scoring much higher, and you know the problem is someone in Arkansas is going to make the decision where the cut off is above proficient and below proficient; someone in this state is going to do it. When they give that test in Florida, someone in Florida is going to make that decision. But you know what when you give the Stanford 9, no one in Arkansas is going to make any decision like that. That is why the business community has more confidence in a nationally normed test.
Jim Argue Ė The reason I am still confused is that other counsel I look to that is reliable tells me that the written definition of proficient in NAEP is verbatim to the written definition in our state test. I want an answer to thatÖ.Fourth grade math results [benchmark] have gone if my memory serves me right, from 41% in 2000 to 67% as released just last week I think. Charity, can you confirm that? . So we are showing improvement on the CRT side, and we will know in a matter of days what the 2002 number is for NAEP in 4th grade math.
Hussman: So that is basically how we have grade inflation basically with the benchmark tests and that is a problem all over the United States..
I do have the graphs. These are the nationally normed tests that have been given in Arkansas since 1988. This is from Arkansas Department of Education. You see there was no progress from 1988 to 1992 in 4th grad.e
` From 1992 they changed and they went from Stanford 8. There was no progress from 1992 to 1995. From 1996 to 2001 we have made a little progress. We have gone from the 45% to the 51%,and all we had to do was go from spending a 1.4 bilion t 2.8 billion.
Next slide; This basically compares the Stanford 9 grade 5 scores which are in red with the ACTAAP [benchmark] grade 4 scores which are in blue. What we are looking at on the Stanford 9 is the percent of the mean NCE which is kind of the way they score that and you see that has been a fairly flat line. It has gone up a little bit. But you see we have made great progress on ACTAAP [benchmark]
Next slide. This is a green line that shows the ACTAAP [benchmark] improvement on the percent proficient and you can see varying increases have gone from about 35% proficient for kids in math scoring in proficient in 1988-89 and that has gone up almost to 70%. Yet the nationally normed test remains relatively flat.
We think a nationally normed test is an essential check on trying to determine on how well we are really doing on our state benchmark tests.
Next Slide. Now this is what they have done in Florida. Florida is probably the state has done more education reform than any other in the country. This is the percent of 4th grade students reading at grade level. African American students have gone from 26% to 42%.
Nex slide. This shows the percent of schools scoring A, B, C, D, F. Now let me just say a word about Florida. How well do you believe that [the information on the chart] How credible is that. One thing that makes me a little skeptical and actually all newspaper people are supposed to be a little bit skeptical. Thatís based on their ACTAAP which is called FCAP or something like that. If that data had been based on a nationally normed test, I would have a lot more confidence than I do. And some detractor might say that is Jeb Bush is trying to run for president, trying to make his record look better in Florida. I donít know; I just know I would have more confidence if it were based on a nationally normed test than on a state only test.
I am finished. Any Questions.
Mahoney. Said he was familiar with the Nationís Report Card which is published in Education Week which as you know is an outstanding Bible, I guess, in public education and they show Arkansas coming from a D to a B in two years.
Hussman said that was on accountability and standards, not just standards.
Transcribed by Debbie Pelley