Recently, we argued that one should prefer death-rate statistics over incidence data because the quality of reporting is higher (The Curious Incidence of Death Rates in the 20th Century). We deliberately avoided actual conclusions about vaccine effectiveness, which is a topic to which we now turn. We’ll be looking at a particular vaccine for a particular country, and then revisit effectiveness for other viruses and other countries over time and over various blog-posts.
As we saw, the anti-vaccine community often cites data and charts like the following to justify the inefficacy of vaccines. It shows an exponentially decreasing rate in Measles-related mortality. And it looks like vaccines did nothing to eliminate the disease, although we will find that there is more to this story.
Pro-vaccine individuals argue that one should use incidence data (and ignore quality issues) because medical advancements also caused a decline in deaths. The dramatic reduction in incidence following vaccine licensure is shown below.
I argued via basic probability theory that an intervention in infections will modulate measured death-rates provided that the sample-size is large enough. Restating the math, we had:
p(death) = p(death | sick) * p(sick)
In math-speak, the vertical pipe “|” means “given”. So, in other words, this says that the probability of dying (mortality rate) is equal to the probability of dying given that you are infected (case-fatality rate) multiplied by the probability of getting infected (incidence-rate). So, we’re saying that mortality rates, or p(death), has been going down because p(sick) has been going down, but the CDC says, “no, p(death) has been going down because p(death | sick) has been going down”. In other words they are attributing the beneficial outcome to other medical advancements, and so therefore p(sick), or incidence, is a more direct measure of disease. And thus, the argument for incidence is compelling because it measures infection directly without other confounding variables, but it is also less compelling because “completeness of reporting” is highly variable and subject to fashion as several studies have shown.
However, if “death-rates were declining because of medical advancements” then why are case-fatality ratios actually increasing dramatically post-vaccine era? The following chart shows that the percentage of Measles cases that resulted in deaths (the average is about 0.10% or about 1 in every 1,000 cases) tripled between 1955 and 1970, which starkly contradicts the claims by the CDC and the David Gorsky’s of the world.
If both mortality rates and incidence data from the CDC are correct, then medical advancements and the introduction of antibiotics contributed to deaths rather than helped prevent them. Coincidentally, this happened especially after the Measles vaccine was licensed in 1963. The graph below shows deaths divided by cases multiplied by 100.
Of course, there is a simple explanation: That people were frantically over-reporting measles incidence before vaccine introduction and they stopped reporting after the introduction of vaccines, causing the dramatic drop in incidence data and therefore a climb in case-fatality ratios.
Level vs. Trend
Both the CDC and anti-vaccine groups are overly focused on levels of their preferred respective data-sets, whether that be incidence or mortality statistics. I depart from the literature by hypothesizing that the main instrument for measuring vaccine efficacy should not be the levels of death-rates, but changes in already declining trends. Below I take the log-transform of raw mortality-rate data, which allows us to zoom in on the vaccine-licensed era and it also gives us a tractable set of linear models. I compared the slope from 1920 to 1962 with the slope from 1963 to 1978 (after which rate-data become meaningless due to small sample-sizes). This wasn’t a piece-wise linear model; I just compared the slope of two separate regressions and the differences were very statistically significant (-0.04 and -0.11 with a t-stat of 5) , which suggests that vaccines played a very strong role in speeding up the decline in Measles-related mortality rates. Visually, the change in slope is clear. The two light-blue lines show upper and lower confidence intervals using a Poisson distribution. Again, the red line is the Measles Vaccine coverage rate.
I used post-1920 data because of possible consequences of World War I, which seem to play a significant role in muting the declining trend in measles mortality rates, and had I incorporated the pre-1920 data, the statistical significance would only have been stronger.
This is a strong win for vaccines. 408 people died from Measles in 1962 and 24 people died in 1968, and so this represents a reduction of 384 people per year. Given the trajectory before 1963, it would have taken until 2010, instead of 1989, to completely wipe out measles deaths and the difference between actual, vaccine-mediated deaths and that trajectory suggests that a total of about 840 lives were saved from 1963 to 2010.
About 8 in 100,000 people died from Measles in 1920 and in 1968, 5 years after the vaccine coverage rates soared, that number was brought down to 0.02 (people out of 100,000), representing a 99.8% drop. However, in terms of levels, only 1 percent of that is attributable to the vaccine because the rate was 0.22 in 1962 and it would have been 0.09 in 1968 if it continued to follow its annual trend. The small impact of medical interventions to mortality statistics is further scrutinized in The Questionable Contribution of Medical Measures to the Decline of Mortality Statistics in the United States. This paper puts an upper bound of 3.5% contribution to medical measures in curbing infectious diseases. However, the above again focuses on levels, and in this particular case, the change in trends is still irrefutable and it strongly suggests that many more lives in exponential orders of magnitude could potentially have been saved if only we had introduced the vaccine earlier.
So, yes, sanitation, medical advancements, and education played the predominant role in wiping out measles deaths and the vaccine made a minuscule dent in levels (so small we could not see it at all without taking the log-transformation of the raw data), but it made a dramatic dent in trends. It is disappointing that the vaccine was introduced late, but the timing of the vaccine introduction is a separate issue from vaccine efficacy, and it is good to be clear-thinking about those differences.
Again, these conclusions apply to a particular vaccine in a particular country and they do not incorporate an evaluation of the risks of measles-vaccine administration. In other words, we have evaluated the benefits portion of a cost-benefit analysis, leaving the rest for later.