WHERE THE NAEP SCORES ARE: Reporting the gaps, ignoring the gains!

FRIDAY, SEPTEMBER 16, 2016

Part 3 in this series

Part 4—Keller's folly: The National Assessment of Educational Progress provides our only reliable educational test scores.
Colloquially, the federally-run program is known as the NAEP. Journalists routinely describe it as "the gold standard" of educational testing, as "America's report card."

As far as anyone has ever said, the NAEP has not been subject to the cheating scandals which have frequently rocked our high-stakes, state-run testing programs. Now for a brief side trip:

(We first wrote about this type of outright cheating in the Baltimore Sun in the mid-1970s. Forty years later, the nation's educational experts and education reporters began to notice the problem. In 2006, we exposed a statewide score-inflation scam which was being conducted by the state of Virginia. A leading educational expert sat on the state of Virginia's board as this scam was being conducted. Our inquiry began when a leading education reporter at the Washington Post was taken in by the scam, penning a glowing front-page report about the wonders of a certain miraculous low-income school. As we demonstrated, the miraculous school actually had the second lowest passing rate in third-grade reading in the whole state of Virginia. Eventually, the chairman of the Virginia board graciously acknowledged the scam, which he said had been unintentional. The Washington Post, where our interest in this matter began, failed to report any part of this matter. This is the way our leading newspapers report on the nation's schools.)

As far as anyone knows, the NAEP hasn't been subject to the cheating scandals which have frequently rocked our state-run testing programs. Beyond that, the NAEP is believed to be run by people who are technically competent. It isn't clear that this is the case with our various statewide testing programs, whose profusion is sometimes said to have placed a strain on the nation's psychometric resources.

As far as we know, no one has ever said that the NAEP isn't a competent program. As far as anyone can say, the major problem with the NAEP is, alas, journalistic:

Routinely, the nation's education reporters report the substantial "achievement gaps" which can be found in NAEP data. They fail to report the "truly spectacular gains" (Richard Rothstein) which can also be found there.

They report the gaps, disappear the gains! It's hard to know why this shouldn't be seen as journalistic malpractice.

Let's offer some basic background. The NAEP came into existence long ago, under President Nixon. In this overview, the nation's leading authority spells out most of the acronyms and offers some basic info:
The National Assessment of Educational Progress (NAEP) is the largest continuing and nationally representative assessment of what American students know and can do in various subjects. NAEP is a congressionally mandated project administered by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education. The first national administration of NAEP occurred in 1969...

NAEP results are designed to provide group-level data on student achievement in various subjects, and are released as The Nation’s Report Card. There are no results for individual students, classrooms, or schools. NAEP reports results for different demographic groups, including gender, socioeconomic status, and race/ethnicity.
As the program has developed, its reach and utility have grown. For our purposes, it now has three major components:

The Long-Term Trend study has been in operation throughout the life of the NAEP. In this program, the NAEP tests 9-year-old, 13-year-old, and 17-year-old students, most notably in reading and math.

The program called The Main NAEP came into existence in xxx. In this program, the NAEP tests students in Grade 4, Grade 8 and Grade 12.

A third program, the Trial Urban District Assessment, continues to grow in its value and reach. In this branch of The Main NAEP, the NAEP tests representative samples of students in participating urban school districts. In the 2015 assessment, twenty-two urban districts took part, including those which serve the nation's five largest cities.

The aforementioned National Center for Education Statistics provides voluminous data from all these testing programs. Substantial achievement gaps can be seen in the data from all three programs. But all three programs also display those "spectacular gains."
Your nation's press corps reports the gaps, persistently skips past the gains.

Just how large are the score gains in the NAEP's voluminous data? We've often detailed those gains at this site. Doing so is utterly pointless within our journalistic culture, which is allergic to information and is almost wholly driven by adherence to preapproved story-lines (by devoted adherence to "narrative").

For today, we'll restrict ourselves to a sample from the Main NAEP. We'll restrict ourselves to the three largest groups within our student population. For a reason which will be clear below, we'll stop at 2013:
Average scores, Grade 8 math, NAEP
2000 / 2013

White students: 282.98 / 293.19
Black students: 243.27 / 262.73
Hispanic students: 251.75 / 271.02
Black kids gained almost twenty points over that 13-year period. Hispanic kids recorded a similar gain.

As you can see, substantial "achievement gaps" still existed, in 2013, in the scores of those three groups of kids. (Though you'll note that the gaps had grown smaller.) It's also true that very large gains could be observed in those scores.

This brings us to a basic question—how large are those gaps and those gains? The average score of black eighth-graders rose by 19.46 points in those thirteen years. Should that be seen as a lot or a little? Absent some sort of measuring stick, there is no way to say.

This brings us to a rule of thumb which is often applied to NAEP scores. When Richard Rothstein and Sean Reardon wrote about NAEP scores in 2011 and 2013, each man seemed to apply this rule of thumb is a fairly straightforward fashion.

(Reardon said this: "The average 9-year-old today has math skills equal to those her parents had at age 11, a two-year improvement in a single generation." Presumably, he was applying some form of the ten-point rule to math scores from the Long-Term Trend study.)

In our view, the rule of thumb to which we refer should be regarded as an extremely rough rule. But according to that rule of thumb, ten or eleven points on the NAEP scale is often equated to one academic year.

If we apply that rule of thumb, we can draw two conclusions about those recent math scores:

The achievement gaps they reveal are somewhat smaller than before, but they're still quite substantial. But so are the academic gains recorded by all three groups of kids. That's especially true of the gains recorded by the black and Hispanic students.

If we go all the way back to the start of the NAEP, the score gains recorded by these groups are substantially larger. It's astounding to think that these basic facts are almost wholly withheld from the public.

Everyone hears about the gaps. Very few people have heard of the gains. When our journalists play this double game, we'd say they're involved in malpractice.

Gains of this type can be seen in the data from all the NAEP's major programs. We'll offer a few points of caution:

Has cheating ever occurred on the NAEP? Generally, it's assumed that cheating doesn't occur on the NAEP. The NAEP has never been a high-stakes test. No one has ever seemed to have anything to gain from invalidating the program. We'll voice a concern below.

How competent is the NAEP? People who love the gloom we're handed won't want to believe that those gains are real. They may wonder if something is "wrong" with the NAEP.

It's a perfectly valid question. If any newspaper ever chose to report the score gains found on the NAEP, we'd expect its reporters to explore that question, along with the question of cheating. (Now that the NAEP is widely discussed, state superintendents have an incentive to garner high statewide scores. Trust us: If there's a way to gimmick the sample of students who get tested, some superintendent has already done so.)

How large are those gains? Thirteen years later, did black eighth-graders really know that much more math? Were they really two years ahead of their counterparts from the 2000 testing?

That is an excellent question. Because the nation's major news orgs refuse to report those impressive score gains, no one has ever stooped to the task of asking such obvious questions.

A few more points should be mentioned:

We've been looking at "disaggregated" scores—at the scores which have been recorded by different population groups. Overall score gains tend to be smaller, for reasons we'll discuss before our report is done.

Beyond that, some readers may be disturbed by the relatively low number of students who are rated "proficient" on the NAEP. Gloom and doom are always appealing, but we should consider two points:

First, the question of what counts as "proficient" is a subjective assessment. The late Gerald Bracey frequently argued that the NAEP has set its bar for "proficiency" at unnaturally high levels. If newspapers ever return to reporting, they should examine this question.

A second point remains basic. The number of students rated "proficient" is higher than it was in the past. This fact is lost when major newspapers refuse to report those score gains.

It's also true that many of Finland's miraculous kids would presumably fail to rate "proficient" on the NAEP, just as our own kids do. As we noted last week, Massachusetts students matched their Finnish counterparts on the 2012 PISA, exceeded them on the 2011 TIMSS. Despite those facts, our journalists keep flying to Finland to gaze on the wonders found there! The American public is never told that Bay State kids are exceeding the distant miracle workers about whose ballyhooed greatness we are constantly and propagandistically told.

We hear about gaps, but not about gains. We hear about the wonders of Finland. Those Bay State kids can go hang.

What happens when a nation's press corps plays such a double game? Consider what Bill Keller said, back in 2013.

Keller is a major American journalist. In 1989, he won a Pulitzer for his international reporting. From 1997 through 2001, he was managing editor of the New York Times. He was executive editor of the Times from 2003 through 2011.

In 2013, Keller was writing a weekly column for the Times. He had never been an education reporter, but he felt he knew enough to make a dramatic statement:
KELLER (8/19/13): The Common Core, a grade-by-grade outline of what children should know to be ready for college and careers, made its debut in 2010, endorsed by 45 states. It is to be followed in the 2014-15 school year by new standardized tests that seek to measure more than the ability to cram facts or master test-taking tricks...

This is an ambitious undertaking, and there is plenty of room for debate about precisely how these standards are translated into classrooms. But the Common Core was created with a broad, nonpartisan consensus of educators, convinced that after decades of embarrassing decline in K-12 education, the country had to come together on a way to hold our public schools accountable.
When he wrote that column, Keller apparently thought that our public schools had experienced "decades of embarrassing decline." But in the previous thirteen years, black and Hispanic eighth-graders had recorded very large score gains on the NAEP math test.

Presumably, Keller had never heard about that. Very few people had. In fairness, the gentleman had an excuse:

He reads the New York Times!

Coming—Week 3 of this report: "Where the gaps are"

21 comments:

  1. From the Boston redevelopment agency:

    "Boston is the location for 35 colleges, universities, and community colleges.1 — there are about 152,000 students at Boston's institutions of higher learning. While Boston comprises 10% of the state's total population, it has 34% of the state's total college enrollment."

    Less obviously, many of the graduates of these colleges and universities remain in the Boston area after graduation. This is especially true of graduates of Harvard and MIT, who like the Cambridge atmosphere and find jobs in the area.

    A consequence is that Boston has a wealth of highly educated people who are the parents of the kids who populate its schools. Of course, not all children have such parents, but there are enough to skew the test results, above and beyond whatever the schools are doing right.

    In Irvine, California, the neighborhood high school adjacent to the university (UCI) has far more National Merit scholars and higher academic accomplishment than the other Irvine high schools in this overall upscale city in a wealthy part of Orange County. Again, the sons and daughters of professors and the proximity to a center of learning skews the average test results beyond the simple impact of high income on its excellent schools. Irvine is racially diverse, as is Boston.

    Somerby doesn't like professors much. I find his anti-intellectualism at odds with his desire to improve schools. Given his attitude, I think he should be looking for demonstrable gains in working class neighborhoods, not in the schools most influenced by academic elites.

    ReplyDelete
    Replies
    1. That is an anti-intellectual jump: from "not liking professors" to "anti-intellectualism."

      Delete
    2. Dave the Guitar PlayerSeptember 16, 2016 at 12:41 PM

      I think that Bob's point is that Finland isn't the only place where you can find "success" in education. Boston may not be the right place to look for programs that work in working class neighborhoods, but neither is Finland. Reformers need to avoid holding up places like Finland as an example of how to succeed without making sure their programs can actually be replicated in our neighborhoods.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. Somerby referenced Massachusetts test scores, not exclusively Boston test scores. Are you under the impression that there are no universities and people with post graduate degrees nationwide in Finland

      Delete
    5. The major industry of Boston is higher education. If you consider the whole state, you bring in Mt. Holyoke, Williams, and Amhearst. It doesn't solve the problem. There are an abnormal number of universities in MA and it skews their results.

      Delete
    6. Maybe other places should start opening more universities.

      Delete
  2. Why does Trump take a lose dose aspirin every day if his heart tests were all normal?

    ReplyDelete
    Replies
    1. As a precaution. 70 years old, overweight, poor diet, no exercise, he's relatively high risk. I'm more concerned about Trump's suspect mental health though.

      Delete
    2. Outside the use of non-routine, extremely invasive testing, it's been said that often the first sign of heart disease is sudden death.

      Delete
    3. Many doctors recommended that EVERYONE take a children's aspirin daily. Based on this article, it's a good idea.

      Many people take a low dose of aspirin every day to lower their risk of a further heart attack or stroke, or if they have a high risk of either. While the anticipated benefit is a lower chance of vascular disease, taking daily aspirin is not without danger: for instance it raises the risk of internal bleeding. Hence the important need to discuss beforehand with the doctor, "In my case, doc, should I be taking daily aspirin?"

      But this week, the publication of three studies in The Lancet, has added a new benefit to the equation: cancer prevention, and stirred up the pros and cons debate.

      In those studies, Professor Peter Rothwell of Oxford University in the UK, a world expert on aspirin, and colleagues, confirm that for people in middle age, a daily dose of aspirin can cut the risk of developing several cancers, with effects starting after only two to three years rather than the ten or so previously thought.


      http://www.medicalnewstoday.com/articles/243265.php

      Delete
    4. I took aspirin daily for a couple of weeks, but then I got nosebleeds, so I stopped taking it.

      Delete
    5. There is a risk of bleeding and ulcers. I believe they are now only recommending it to people with previous heart problems or risk (e.g., family history).

      Do they routinely report men's testosterone levels? They don't routinely report women's estrogen levels. Is this ego or part of a normal health exam? Being female, I wouldn't know.

      Delete
    6. I have Kaiser and no one has ever recommended that I take an aspirin daily. I too am 70 and overweight like Trump.

      Delete
  3. From Bob Somerby, relentless promoter of NAEP score coverage.

    "The program called The Main NAEP came into existence in xxx. In this program, the NAEP tests students in Grade 4, Grade 8 and Grade 12."

    Bob Somerby was born in www. He started his blog in yyy. His readers response is zzz.

    ReplyDelete
    Replies
    1. You are so predictable.

      Delete
    2. Hey 3:29,

      With that joke, please never again call Bob a failed comedian.

      I'll try to be a hypnotist: from now on, whenever you type "failed comedian" you will think "www, xxx, yyy, zzz" - and feel shame.

      Delete
  4. Hello everyone out there my name is ROLLO KATE.This is a very joyful day of my life because of the help Dr.Ogudu has rendered to me by helping me get my ex husband back with his magic and love spell. i was married for 8years and it was so terrible because my husband was really cheating on me and was seeking for a divorce but when i came across Dr.Ogudu email on the internet on how he help so many people to get thier ex back and help fixing relationship, and make people to be happy in their relationship. i explained my situation to him and then seek his help but to my greatest surprise he told me that he will help me with my case and here i am now celebrating because my Husband has change totally for good. He always want to be by me and can not do anything without my present. i am really enjoying my marriage, what a great celebration. i will keep on testifying on the internet because Dr.Ogudu is truly a real spell caster. DO YOU NEED HELP THEN CONTACT DOCTOR OGUDU NOW VIA EMAIL: [OGUDUSPELLTEMPLE@YAHOO.COM] WHATSAPP NUMBER +2348106058254 OR WEBSITE
    http://oguduspelltemple.wixsite.com/drogudu

    ReplyDelete
  5. Am here to testify of a great spell caster who is very straight forward with his words of casting spells. When my Wife left me for another Man i was so confused,but one day a friend of mine gave me the email address of this great spell caster called Awo Babatunde, I contacted this man and explained to him all that i was going through,he then told me to be calm and not worry anymore because my case is a very easy one. At first i never believe all that he was saying until when my wife returned home two days later begging me to forgive him all that he has done to me and to please accept him back into my life,I was so amazed,I never believe it was going to happen that fast.
    This man is truly great indeed. Contact him today for a perfect solution to your love problem, Awobabatunde50@gmail.com or whatsApp him +2349045101972

    ReplyDelete