Periodically, President Obama, Secretary of Education Duncan, or various newspapers will highlight so-called “miracle” schools (often charters) that have made quick and dramatic gains in test scores or graduation rates. The message is — since they can do it, why can’t the rest of us? Often, however, the student attrition rates behind those achievements are ignored. In other words, these schools often don’t make these gains with the same students, or they really don’t have a 100% graduation rate if you look at who they started with…

I thought it would be useful to create a beginning list of posts and articles documenting some of the facts behind these inaccurate claims. This is just a beginning list, though, and I hope readers will made additional suggestions:

Here are my choices for The Best Posts About Attrition Rates At So-Called “Miracle” Schools:

Miracle schools, vouchers and all that educational flim-flam is by Diane Ravitch.

Urban Prep and The Whole Story is by Chris Lehmann.

Where have all the KIPPsters gone? is by Caroline Grannan.

You’ll find an interesting dialogue in the comments section about Caroline’s post. This next article might also contribute to that discussion:

Do self-selection and attrition matter in KIPP schools? appeared in the Washington Post.

A question about Memphis school Obama chose was published in The Washington Post.

KIPP’s Atrocious Attrition and Expulsions from New Orleans Charters are posts by Gary Rubinstein.

Garry Rubinstein has also helped start a Miracle Schools wiki to coordinate research on this phenomena. I’m adding it to the same “The Best…” list.

I’ve previously written two posts about this topic:

One Very Disappointing Part Of President Obama’s Speech Today

TIME Magazine Can Do Better Than This…

An Interesting Few Days is by Diane Ravitch and The “90/90/90 Schools” Myth is by Justin Baeder at Ed Week. Though they don’t specifically deal with attrition issues at so-called “miracle” schools, they raise other important challenges to them, so I’m adding both posts to this list. Education ‘Miracles’ Don’t Survive Scrutiny is by Mike Rose and is in a similar vein.

Some Charters finally admit attrition — then rationalize it is by Gary Rubenstein.

Peer Effects And Attrition In High-Profile Charter Schools is another great post from The Shanker Blog.

Virgin Mary On A Grilled Cheese And Other Miracles is by Gary Rubinstein.

If You Believe in Miracles, Don’t Read This is by Diane Ravitch.

Do You Believe in Miracles? is by Diane Ravitch.

Getting Real About Turnarounds is by Diane Ravitch.

Frozen Water & “Miracle” Schools

Here’s something that is connected to the topic of this post: What Really Happened At The School Where ‘Every Senior Got Into College’ is from NPR.

Feedback and suggestions are welcome!

If you found this post useful, you might want to consider subscribing to this blog for free.

You might also want to explore the nearly 700 other “The Best…” lists I’ve compiled.

Grannan’s post is not one of the best posts. She is dishonest in refusing to admit that a recent and much more thorough study from Mathematica refutes her claims about KIPP. See http://www.mathematica-mpr.com/publications/pdfs/education/KIPP_middle_schools_wp.pdf

Stuart,

Thanks for your comment. I’ve asked Caroline to leave a response…

Larry

OK, Stuart, finally you inspired me to update my research, at least on the San Francisco KIPP schools included in the Mathematica study, the only California KIPP schools in the study. I’m very familiar with the California Department of Education database and am not at all familiar with those in other states.

Mathematica looked at the attrition/mobility of the KIPP schools in every possible permutation and from every possible angle — except one. They didn’t simply give us the size of the class cohort and show us if and how much it increased or decreased. I still can’t figure out how Mathematica’s various findings jibe with what I found when I did look at that, but here are my findings. Larry, I hope there’s no length limit on these comments.

Mathematica looked at mobility figures from grade 5 to grade 8 and also from grade 6 to grade 8. Most KIPP schools are grades 5-8, and at many (most?) of them, there’s an increase as the class moves from grade 5 to grade 6. I’m going to address that in a separate comment. So anyway, I looked at the numbers that way too. But because 6th grade is when the schools are closest to fully enrolled, I’m interpreting the change from 6th-8th grade as most significant. In the two San Francisco KIPP schools, looking at three years’ worth of class cohorts each, the drop from 6th to 8th grade ranged from 28.58% to 52.9%. And (as I’ll remind you repeatedly), those figures are for the BEGINNING of the school year, so the 8th-grade figure does not tell us how many of those students finished 8th grade.

Here’s the attrition information for the two San Francisco KIPP schools, which Mathematica included in its study. I researched the same years Mathematica did.

KIPP Bayview

The class that was in grade 8 in the 08-09 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 05-06: 69 students

6th grade, 06-07: 79 students

7th grade, 07-08: 53 students

8th grade, 08-09: 40 students

Change in size of student cohort from 5th-8th*: -42.03%

Change in size of student cohort from 6th-8th*: -49.37

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 07-08 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 04-05: 76 students

6th grade, 05-06: 88 students

7th grade, 06-07: 58 students

8th grade, 07-08: 47 students

Change in size of student cohort from 5th-8th*: -38.16%

Change in size of student cohort from 6th-8th*: -46.60%

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 06-07 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

6th grade, 05-06: 85 students

7th grade, 06-07: 55 students

8th grade, 07-08: 40 students

Change in size of student cohort from 6th-8th*: -52.9%

*again, this is the BEGINNING of 8th grade.

KIPP San Francisco Bay

The class that was in grade 8 in the 08-09 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 05-06: 50 students

6th grade, 06-07: 84 students

7th grade, 07-08: 62 students

8th grade, 08-09: 60 students

Change in size of student cohort from 5th-8th*: 20% (yes, it increased)

Change in size of student cohort from 6th-8th: -28.58% (interestingly, a 60% jump from 5th grade to 6th — then a drop by 8th grade.)

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 07-08 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 04-05: 49 students

6th grade, 05-06: 75 students

7th grade, 06-07: 55 students

8th grade, 07-08: 44 students

Change in size of student cohort from 5th-8th*: -10.2%

Change in size of student cohort from 6th-8th: -41.33%

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 06-07 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

6th grade, 04-05: 78 students

7th grade, 05-06: 56 students

8th grade, 06-07: 33 students

Change in size of student cohort from 6th-8th: -56.7%

*again, this is the BEGINNING of 8th grade.

But does this happen at comparable San Francisco Unified School District middle schools? No. I researched two of SFUSD’s lowest-performing middle schools, chosen randomly. The changes ranged from -2.76% to an increase of 11.43%.

Horace Mann Middle School, San Francisco

The class that was in grade 8 in the 08-09 school year went from 149 in grade 6 to 148 in grade 8, a decrease of <1%

The class that was in grade 8 in the 07-08 school year went from 181 in grade 6 to 176 in grade 8, a decrease of 2.76%

The class that was in grade 8 in the 06-07 school year went from 204 in grade 6 to 205 in grade 8, an increase of <1%

Visitacion Valley Middle School, San Francisco

The class that was in grade 8 in the 08-09 school year went from 105 in grade 6 to 117 in grade 8, an increase of 11.43%

The class that was in grade 8 in the 07-08 school year went from 114 in grade 6 to 115 in grade 8, an increase of <1%

The class that was in grade 8 in the 06-07 school year went from 122 in grade 6 to 132 in grade 8, an increase of 8.2%

Since KIPP and its supporters have retorted that "The San Francisco KIPP schools are outliers." But that's false. Other California KIPP schools showed the same pattern in my research, confirmed and amplified by a study of the Bay Area KIPP schools by the research organization SRI International. Today, I looked at the figures for just one other KIPP school outside of San Francisco, KIPP Bridge in Oakland:

KIPP Bridge, Oakland

The class that was in grade 8 in the 08-09 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 05-06: 63 students

6th grade, 06-07: 73 students

7th grade, 07-08: 54 students

8th grade, 08-09: 39 students

Change in size of student cohort from 5th-8th*: -38.1%

Change in size of student cohort from 6th-8th*: -46.57%

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 07-08 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

5th grade, 04-05: 76 students

6th grade, 05-06: 75 students

7th grade, 06-07: 54 students

8th grade, 07-08: 44 students

Change in size of student cohort from 5th-8th*: -41.34%

Change in size of student cohort from 6th-8th*: -42.11%

*again, this is the BEGINNING of 8th grade.

The class that was in grade 8 in the 06-07 school year – important note that these figures are from the beginning of the school year, so we have no information on the number who finished 8th grade.

6th grade, 04-05: 78 students

7th grade, 05-06: 47 students

8th grade, 06-07: 39 students

Change in size of student cohort from 6th-8th*: -50%

*again, this is the BEGINNING of 8th grade.

So, as you can see, the KIPP schools studied do show significant attrition, and the comparable public schools show no such pattern.

Regarding the bump in KIPP enrollment from the 5th to the 6th grade:

The actual reason is fairly evident, at least in my community. All the feeder schools are K-5s. Families are happy enough with their elementary schools (perhaps very happy) that they don’t want to pull the kids out to switch schools before grade 5, and kids want to stay with their class for 5th grade and enjoy the gradu-motion activities.

That doesn’t really jibe with the script that all urban public schools are pits of hell and families are clamoring to get their kids out at the earliest possible moment, so it doesn’t get discussed much and my take is that KIPP wants to shush it.

Unless I missed something else, when Mathematica mentioned the common bump in grade 6, it only mentioned lightly that it may be due to students being retained to repeat grade 6.

One reason I suspect KIPP doesn’t encourage discussion of this is that it confounds the universal (though false) claim that “all KIPP schools have long waiting lists.” Obviously, if the 5th grades aren’t full and attrition is emptying seats by 7th grade, there should be openings aplenty — no lottery, no waiting lists.

I’ve asked why KIPP’s model starts at grade 5 given that situation. The answer (I forget who it came from, maybe Jay Mathews) is that 5th grade is young enough that the students will buy into the KIPP culture. By 6th grade, they’re more likely to resist.

A few things, Caroline:

1. As Mathematica found, KIPP schools have a wide range of attrition compared to nearby public schools. Some have more attrition, some have the same, some have LESS. On average it’s the same, but to focus on two or three where the attrition is more looks like cherrypicking.

2. You aren’t discussing how you picked the “comparable” public schools to which KIPP is being compared.

3. You haven’t explained why just looking at the numbers of students in a cohort is better than, rather than immeasurably worse than, looking at where individual students enrolled (which is what Mathematica did). If some students were retained in 6th grade, for example, 6th grade looks larger and 7th grade looks smaller, and for you to attribute that to attrition would be plainly wrong. Mathematica has the far superior approach here.

The numbers speak for themselves. Mathematica’s approach is to use all kinds of complicated detail, and mine is simply to look at how much the numbers dropped. I don’t need to explain why simply counting the students in the class is the irrefutable way to discern the level of attrition. That’s what attrition is, by definition.

I chose those two KIPP schools because they are the California KIPP schools in the Mathematica study.

I was going to use the two officially “failing” middle schools in SFUSD — Horace Mann and Everett — but Everett actually had a big surge in enrollment. One class literally doubled in size from 6th to 8th grade. I assume that’s due to some artificial factor that I’m unaware of. So I randomly picked a second high-poverty, low-performing middle school, Visitacion Valley.

I don’t need to explain why simply counting the students in the class is the irrefutable way to discern the level of attrition. That’s what attrition is, by definition.No, it most certainly is not. See point 3 of my previous comment.

OK, if by your definition, attrition does NOT refer to the number of students who leave a school, then no wonder we disagree. But that’s what it means by my definition.

Do you know the difference between “retention” and leaving a school? That’s what I was referring to.

So your position is that if every year the 8th grade is ~half the size of the 6th grade, that’s due to students’ being retained in 6th grade? OK, so be it.

No, that’s a straw man.

My position is that retention does occur, and that the only way to know how much retention vs. attrition is occurring is to do what Mathematica did: track what happens to individual students over time.

In that comparison, some KIPP schools have more attrition than comparable public schools, some have the same, and some have LESS attrition. Overall, the average is about the same.

What Mathematica did is far superior to your simplistic method of counting up a cohort’s numbers and assuming that everything is attrition. The only reason to do that is if one has an ax to grind.

The numbers speak for themselves.

But your interpretation of the numbers is dishonest, if you’re trying to claim that the drop in numbers is necessarily attrition. You know that KIPP has a high retention rate.

Sorry. By definition, it’s attrition if students are gone who were previously there. Sure, some students may have been retained — they would still show up in the following year’s cohort — but overall, it’s unclear on the concept to deny that a drop of more than half the students doesn’t mean that more than half the students left.

It’s unclear on the concept of “retention” to say that just because the 8th grade class is smaller than the 6th grade class (from 2 years prior), they all “left,” which is what your last sentence seems to say. No: if some students were retained, they’re not in 8th grade that year because they’re still back in 6th or 7th grade.

An example from your figures:

5th grade, 05-06: 63 students

6th grade, 06-07: 73 students

7th grade, 07-08: 54 students

8th grade, 08-09: 39 students

But we know that retention rates can be as high as 15% or even higher in KIPP schools. So out of the 73 students in 6th grade in 2006-07, what if 11 or 12 were retained in 6th grade in 2007-08? You’re saying, “Aha, they’re gone!”, but they’re still in the school somewhere.

This is just one example that could be going on throughout all of your different figures. That’s why individual student data is so crucial.

Then I need to make my little graphs again and figure out how all the grades’ cohorts changed.

However, I do have to point out that KIPP itself has never made these non-credible claims that its retention rate is so high that it only *appears* to be sky-high attrition. KIPP spokespeople have at least at times — once others reported the attrition — owned up to it and said they were working on it.

So you’re changing THEIR story, Stuart Buck.

Also, KIPP and other observers (both charter cheerleaders and charter critics) acknowledge that one reason families leave schools is that they’re told their child won’t be promoted to the next grade. Kids hate it and families hate it. So the notion that droves of kids and their families are meekly accepting being told they’re repeating 6th or 7th grade also doesn’t jibe with what KIPP itself has said.

So what? KIPP does have high retention. From the initial Mathematica report, page 17:

“KIPP middle schools retain students in grade at significantly higher rates than traditional public schools in the same districts. . . . Retention rates range from 2 to 18 percent in fifth grade, and one to 12 percent in sixth grade at KIPP schools; other public schools in only one district retain students at rates higher than 4 percent in either year.”

Typo: “The message is — since they can’t do it, why can’t the rest of us?”

Thanks, Dave. It’s fixed now.