Diet Modification Trials: Notes on Study Design

The other day, my internet meanderings brought me back to a review of fat modification trials conducted by the Cochrane collaboration. This is a not-for-profit group known for its rigorous meta-analyses.

They selected 27 studies that reduced saturated fat or total fat (in some cases along with increased PUFA), and fit several inclusion criteria. The results:

There was no significant effect on total mortality (rate ratio 0.98, 95% CI 0.86 to 1.12), a trend towards protection form cardiovascular mortality (rate ratio 0.91, 95% CI 0.77 to 1.07), and significant protection from cardiovascular events (rate ratio 0.84, 95% CI 0.72 to 0.99). The latter became non-significant on sensitivity analysis.

Trials where participants were involved for more than 2 years showed significant reductions in the rate of cardiovascular events and a suggestion of protection from total mortality ["suggestion" = not statistically significant]. The degree of protection from cardiovascular events appeared similar in high and low risk groups, but was statistically significant only in the former.

"Sensitivity analysis" is a statistical method that allows investigators to take into account the limitations of their model, in this case, the way in which they aggregated the studies' data. Another way of putting their findings is that they found no significant effect of fat modification on mortality or cardiovascular mortality, and they found a reduction in non-fatal "cardiovascular events" (more on this phrase later) only in a subset of the data.

I'll be the first to admit the meta-analysis isn't perfect. They cast too wide a net, not allowing them to distinguish the effect of reducing total fat from the effect of reducing saturated fat. They lumped both together, which from a practical standpoint isn't actually a problem because both sets of studies show essentially the same thing: zilch. But it's still not the best way to conduct a meta-analysis. They also omitted the Sydney Diet-Heart study for mysterious reasons, which was a five year randomized trial that found an increase in mortality in volunteers substituting vegetable oils for animal fat. Then there's the conclusion, which boggles the mind:

Lifestyle advice to all those at high risk of cardiovascular disease (especially where statins are unavailable or rationed), and to lower risk population groups, should continue to include permanent reduction of dietary saturated fat and partial replacement by unsaturates.

Are these the same people who wrote the results section? I don't understand how they arrived at that conclusion from their own results.

In any case, this brings me to my main point. There are two types of outcomes you can measure in these trials: "hard endpoints" and "soft endpoints". Hard endpoints are not subjective. The hardest endpoint is death. Either you're dead or you aren't; there's no room for interpretation there. A bit less hard is death from a particular cause, such as heart attack. In that case, you're definitely dead, but the physician has to guess what you died of. That involves some judgment on the part of the physician and can introduce bias, depending on the study design. The softest endpoints are non-fatal events like angina, bypasses and stents. These depend on the judgment of both the physician and the patient, and are the most susceptible to bias.

The gold standard for controlled trials is a design known as "double-blind", where neither the participant nor the physician knows which group the participant is in. This design eliminates bias from both the participant and the physician side, allowing correction for the placebo effect and subtle bias in diagnosis. This is easy to do for drug trials, where placebo pills look just like the drug. But it's more difficult to pull off in a diet trial, where the patient knows what foods he's eating. Still, it can be done by giving participants similar-looking margarines containing either saturated or polyunsaturated fats, or sometimes by controlling diets in an institutionalized setting.

There have been three double-blind trials comparing the incidence of heart attack and/or mortality in volunteers eating either saturated animal fat or polyunsaturated vegetable fat: the 1968 National Diet-Heart trial (2 years), the 1969 Los Angeles Veterans' Administration trial (8 years), and the 1989 Minnesota Coronary Survey trial (4.5 years). The two studies that reported total mortality found no significant difference between groups. Two out of three found no difference in heart attack deaths. Of the two that reported on non-fatal cardiovascular endpoints, one found a significant difference. The V.A. trial was the only one to find a significant difference in heart attack deaths (18% decrease) and non-fatal events. There were significantly more heavy smokers in the animal fat arm of the V.A. trial, which was an unfortunate consequence of the participant randomization process. So that result is difficult to interpret.

The three double-blind diet trials, with the least potential for bias, really give no support to the idea that saturated/animal fat contributes to cardiovascular disease. As the participants were already eating a diet high in omega-6 to begin with, there is also no detectable effect of increasing omega-6 on cardiovascular health.

Many of the trials of this nature have been "single-blinded", where the participant knows which group he's in, but the physician doesn't. In this case, all endpoints involving death will be unbiased, because the physician deciding the diagnosis is not influenced by knowing what group the patient is in. However, soft, non-fatal events such as angina are still highly susceptible to the placebo effect. This is because they depend on the subjective judgment of the patient, who knows which group he's in.

I think it's interesting to note that very few dietary fat modification trials have found reductions in total mortality, which is the hardest endpoint and the least susceptible to bias. This is reflected in the Cochrane collaboration's findings. However, a number of the non-blinded and single-blinded studies have found differences in non-fatal cardiovascular events, sometimes creating absurd results. For example, in the 1966 Anti-Coronary Club trial, participants in the vegetable oil group had a significant reduction in non-fatal cardiovascular events, but a massive increase in cardiovascular deaths and total mortality. The former result could result from a placebo effect, due to the non-blinded nature of the trial.

The fact that the Cochrane review found statistically significant benefits of fat modification in soft, non-fatal endpoints (for a portion of the data set), but not endpoints involving death, suggests to me that what we're seeing may actually be a placebo effect resulting from the fact that patients were not blinded in the majority of these trials.

The only "fat modification" intervention that consistently reduces total and cardiovascular mortality is omega-3 fat supplementation, ideally in combination with omega-6 restriction. This is supported by the results of the DART trial, the Lyon Diet-Heart trial, the ISIS trial and the the GISSI-prevenzione trial.

The Finnish Mental Hospital Trial

This diet trial was conducted between 1959 and 1971 in two psychiatric hospitals near Helsinki, Finland. One hospital served typical fare, including full-fat milk and butter, while the other served "filled milk", margarine and polyunsaturated vegetable oils. Filled milk has had its fat removed and replaced by an emulsion of vegetable oil. As a result, the diet of the patients in the latter hospital was low in saturated fat and cholesterol, and high in polyunsaturated fat compared to the former hospital. At the end of six years, the hospitals switched diets. This is known as a "crossover" design.

The results were originally published in 1972 in the Lancet (ref), and a subset of the data were re-published in 1979 in the International Journal of Epidemiology (ref). They found that during the periods that patients were eating the diet low in saturated fat and cholesterol, and high in vegetable oil, male participants (but not females) had roughly half the incidence of heart attack deaths. There were no significant differences in total mortality in either men or women. The female data were omitted in the 1979 report.

This study is often cited as support for the idea that saturated fat increases the risk of heart attack. The reason it's cited so often is it's one of a minority of trials that came to that conclusion. The only other controlled trial I'm aware of that replaced animal fat with polyunsaturated vegetable oil (without changing other variables at the same time) and found a statistically significant decrease in cardiovascular deaths was the Los Angeles Veterans' Administration study. However, there was no difference in total mortality, and there were significantly more heavy smokers in the control group. The difference in heart attack deaths in the V.A. trial was 18%, far less than the difference seen in the Finnish trial.

I can cite three controlled trials that came to the opposite conclusion, that switching saturated fat for vegetable oil increases cardiovascular mortality and/or total mortality: the Anti-Coronary Club Trial (4 years), the Rose et al. corn oil trial (2 years), and the Sydney Diet-Heart trial (5 years). Other controlled trials found no difference in total mortality or heart attack mortality from this intervention, including the National Diet-Heart Study (2 years) and the Medical Research Council study (7 years). Thus, the Finnish trial is an outlier whose findings have never been replicated by better-conducted trials.

I have three main bones to pick with the Finnish trial. The first two are pretty bad, but the third is simply fatal to its use as support for the idea that saturated fat contributes to cardiovascular risk:

1) A "crossover" study design is not an appropriate way to study a disease with a long incubation period. How do you know that the heart attacks you're observing came from the present diet and not the one the patients were eating for the six years before that? The Finnish trial was the only trial of its nature ever to use a crossover design.

2) The study wasn't blinded. When one wants to eliminate bias in diagnosis for these types of studies, one designs the study so that the physician doesn't know which group the patients came from. That way he can't influence the results, consciously or unconsciously. Obviously there was no way to blind the physicians in this study, because they knew what the patients in each hospital were eating. I think it's interesting that the only outcome not susceptible to diagnostic bias, total mortality, showed no significant changes in either men or women.

3) The Finnish Mental Hospital trial was not actually a controlled trial. In an editorial in the November 1972 issue of the Lancet, Drs. John Rivers and John Yudkin pointed out, among other things, that the amount of sugar varied by almost 50% between diet periods. In the December 30th issue, the lead author of the study responded:

In view of the design of the experiment the variations in sugar intake were, of course, regrettable. They were due to the fact that, aside from the fatty-acid composition and the cholesterol content of the diets, the hospitals, for practical reasons, had to be granted certain freedom in dietary matters.

In other words, the diets of the two hospitals differed significantly in ways other than their fat composition. Sugar was one difference. Carbohydrate intake varied by as much as 17% and total fat intake by as much as 26% between diet periods (on average, carbohydrate was lower and total fat was higher in the polyunsaturated fat group). The definition of a controlled trial is an experiment in which all variables are kept constant except the one being evaluated. Therefore, the Finnish trial cannot rightfully be called a controlled trial. This places it in the same category as other observational studies, in which variables are not controlled and one can only guess what factors caused the difference in disease incidence. The fact that the result has never been replicated casts further doubt on the study.

I could continue listing other problems with the study, such as the fact that the hospital population included in the analysis had a high turnover rate (variable, but as high as 40%), and patients were included in the analysis even if they were at the hospital for as little as 50% of the time between first admission and final discharge (i.e., they came and went). But what's the use in beating a dead horse?

The Finnish trial is still very useful, however. I use it as a litmus test to determine which papers are solid and which are desperate for data that confirm their biases. Any author who cites the Finnish trial in support of the idea that saturated fat causes heart attacks either isn't familiar with it, or is not objective.

Unrefined vs. Refined Carbohydrates and Dental Cavities

There's a definite association between the consumption of refined carbohydrates and dental cavities. Dr. Weston Price pointed this out in a number of transitioning societies in his epic work Nutrition and Physical Degeneration. Many other anthropologists and dentists have observed the same thing.

I believe, based on a large body of anthropological and medical data, that it's not just an association-- sugar and flour cause cavities. But why? Is it that they lack micronutrients-- the explanation favored by Price-- or do they harm teeth by feeding the bacteria that participate in cavity formation? Or both?

I recently found an interesting article when I was perusing an old copy of the Journal of Dental Research: "A Comparison of Crude and Refined Sugar and Cereals in Their Ability to Produce in vitro Decalcification of Teeth", published in 1937 by Dr. T. W. B. Osborn et al. (free full text). I love old papers. They're so free of preconceptions, and they ask big questions. The authors begin with the observation that the South African Bantu, similar to certain cultures Dr. Price visited, had a low prevalence of tooth decay when eating their native diet high in unrefined carbohydrate foods. However, their decay rate increased rapidly as modern foods such as white flour and refined sugar became available.

To test whether refined carbohydrates have a unique ability to cause tooth decay, the investigators took pieces of teeth that had been extracted for reasons other than decay (for example, crowding), and incubated them with a mixture of human saliva and several different carbohydrate foods:

• crude cane juice
• refined cane sugar
• whole wheat flour
• white wheat flour
• whole corn meal
• refined corn meal

After incubating teeth in the solutions for 2-8 weeks at 37 C (human body temperature), they had trained dentists evaluate them for signs of decalcification. Decalcification is a loss of minerals that is part of the process of tooth decay. Teeth, like bones, are mineralized primarily with calcium and phosphorus, and there is a dynamic equilibrium between minerals leaching out of the teeth and minerals entering them.

The researchers used teeth incubated in saline solution as the reference. The dentists were "blinded", meaning they didn't know which solution each tooth came from. This is a method of reducing bias. Here are some of the results. Cane juice vs. refined sugar:

Unrefined cane juice was not very effective at causing decalcification, compared to refined sugar. This was a surprise to me. Here is the result for wheat:Note that the scale is different on this graph. Wheat, and particularly refined wheat, is very good at decalcifying teeth in vitro. Corn:

Refined corn is much more effective at decalcifying teeth than whole meal corn. Next, the investigators performed an experiment where they compared the three types of refined carbohydrate to one another:
As one would predict from the graphs above, refined wheat is worse than refined corn, is worse than refined sugar. This is really at odds with conventional wisdom.

It's important to keep in mind that these results are not necessarily directly applicable to a living human being, who wouldn't let a mouthful of wheat porridge sit in his mouth for five weeks. But it does show that refining carbohydrates may increase their ability to cause cavities due to a direct effect on the teeth (rather than by affecting whole-body nutritional status, which they do as well).

The authors tested the acidity of the different solutions, and found no consistent differences between them (they were all at pH 4-5 within 24 hours), so acid production by bacteria didn't account for the results. They speculated that the mineral content of the unrefined carbohydrates may have prevented the bacterial acids from leaching minerals out of the teeth. Fortunately for us, they went on to test that speculation in a series of further investigations.

In another paper, Dr. T. W. B. Osborn and his group showed that they could greatly curb the decalcification process by adding organic calcium and phosphorus salts to the solution. This again points to a dynamic equilibrium, where minerals are constantly leaving and entering the tooth structure. The amounts of calcium and phosphorus required to inhibit calcification were similar to the amounts found in unrefined cane sugar, wheat and corn. This suggests the straightforward explanation that refined sugar and grains cause decay at least in part because most of the minerals are removed during the refining process.

However, we're still left with the puzzling fact that wheat and corn flour decalcify teeth in vitro more effectively than cane juice. I suspect that has to do with the phytic acid content of the grains, which binds the minerals and makes them partially unavailable to diffusion into the teeth. Cane juice contains minerals, but no phytic acid, so it may have a higher mineral availability. This explanation may not be able to account for the fact that refined sugar was also less effective at decalcifying teeth than refined wheat and corn flour. Perhaps the residual phytic acid in the refined grains actually drew minerals out of the teeth?

No, I'm not saying you can eat sugar with impunity if it's unrefined. There isn't a lot of research on the effects of refined vs. unrefined sugar, but I suspect too much sugar in any form isn't good. But this does suggest that refined carbohydrates may be particularly effective at promoting cavities, due to a direct demineralizing effect on teeth subsequent to bacterial acid production. It also supports Dr. Price's contention that a food's micronutrient content is the primary determinant of its effect on dental health.

Reversing Tooth Decay
Preventing Tooth Decay
Dental Anecdotes

LDL Calculator

Commenter Kiwi Geoff kindly wrote a program that calculates LDL using the Friedewald equation and the equation from this paper, which may be more accurate for people with a total cholesterol over 250 and triglycerides under 100. For people whose triglycerides are over 100, the Friedewald equation should be relatively accurate. You can plug your total cholesterol, HDL and triglycerides into the program (in mg/dL), and it gives you both LDL values side-by side. Here it is:

LDL Cholesterol Calculator

Thanks, Geoff.

Another Fatty Liver Reversal

Just to show it wasn't a fluke, reader "Steve" replicates the experiment:

I had a similar problem as what Sam described, and it just happened to coincide with my discovery of and commitment to a new eating plan (based on low/good carb, high in good fat and omega 3, and good protein--basically a mix of paleo, primal, low carb, whatever they call it). I consider myself lucky to have had great fortune in my timing of finding out about my fatty liver.

My ALT and AST [markers of liver damage] had been at 124 and 43 respectively, and then still at 80 and 30 in a follow up a few months later. I weighed in at about 205 (I'm 6'1.5" on a slimmish frame), which was my heaviest. I had been on a basic American (bad) diet. The whole thing shocked me, especially after a CT with contrast showed the fatty deposits on my liver (and prior to that, when the muddy ultrasound revealed a fatty liver and a possible pancreatic mass, later ruled out by the CT). Like Sam, though I was surely overweight, I was not fat or heavy. (Most people have noticed I look leaner, but are shocked when I disclose how much weight I have lost since they say "I cannot believe you had that much to lose.")

At about the same time I found out about my liver issue, I had been getting into reading about diet and health (something I had done once when I read the Zone stuff from Sears many years ago). I practically dove through Taubes, Eades, Cordain, and a bunch of blogs (including yours), and I made a commitment to fix my problem.

I started a pretty severe regimen at first, which included only protein and good fats with a minimal amount of non-starchy fruits and vegetables. Almost immediately, I started losing weight and body fat (as measured by an electrical impedance scale). I have always supplemented with fish oil, but I added krill oil and I also started eating grass-fed beef and pastured eggs and pastured pork as much as possible. I have added some coconut oil and pastured butter to my diet as well. I have dropped almost 40 pounds, I am down to about 10-11% body fat (from 24%), and my ALT/AST on my last test was 24/14 [normal]. I am getting another test soon, and I expect similar results.

And a later comment:

I can add to the story that I first found out about the fatty liver on a routine new patient blood screening when I moved to a new town. I can also add that it took a bit of initiative on my part to get to the right diagnosis. The first doctor suspected hepatitis, but when blood work ruled that out, he ordered the imagining tests. Once I was referred to a GI specialist, it was a quick diagnosis. Still, I had to undertake myself to figure out the best diet. The GI recommended eliminating white bread, rice, pasta, starches, etc. but also recommended lowering fat intake. Having done some of my reading on diet and health, I knew to follow the former advice and to modify the latter to be "get plenty of fat, but make sure its the right kind."

Steve took the initiative and fixed his damaged liver. He modified his GI doctor's advice based on what he had read about nutrition, with excellent results. I suspect his doctor will be all ears next time Steve comes into his office.

The liver is a remarkable organ. Besides being your "metabolic grand central station", it's the only organ in the human body that can regenerate almost completely. It can be 75% obliterated, and it will grow back over time. Fatty liver and NASH are largely reversible.

When Friedewald Attacks

I don't get very excited about nitpicking blood lipids. That's not to say they're not useful. There's definitely an association between blood lipids and certain health outcomes such as cardiovascular disease. The thing that tires me is when people uncritically interpret those associations as evidence that lipids are actually causing the problem.

Low-density lipoprotein, or LDL, is the cholesterol fraction that typically gets the most attention. High LDL associates with heart attack risk in Americans and some other groups. Statins reduce LDL and reduce heart attack risk in a subset of the population, and this has been used to support the idea that elevated LDL causes heart attacks. This is despite the fact that lowering LDL via diet doesn't seem to reduce heart attack risk (typically by reducing total fat and/or saturated fat). Statins may in fact work because they're anti-inflammatory, rather than because they reduce LDL. But both explanations are speculative at this point.

The fact remains that if you want to know if Mr. Jones is going to have a heart attack in the next five years, measuring his LDL will give you more information than not measuring his LDL. This association doesn't seem to apply to all cultures or to Americans eating atypical diets. Then you can get into the fractions that associate more tightly with heart attack risk, such as low HDL, high triglycerides, small dense LDL, etc. Triglycerides vary with HDL (that is, when trigs go up, HDL generally goes down) and the ratio also happens to be a predictor of insulin sensitivity. Total cholesterol is virtually useless for predicting heart attack risk in the general population. This is something I'll discuss in more detail at another time.

When you walk into the doctor's office and ask him to measure your cholesterol, the numbers you get back will generally be total cholesterol, LDL, HDL and triglycerides. All of those except LDL are measured directly. LDL is calculated using the Friedewald equation, which is (in mg/dL):

LDL = TC - HDL - (TG/5)

Low-carb advocates have known for quite some time that this equation fails to accurately predict LDL concentration outside certain triglyceride ranges. Dr. Michael Eades put up a post about this recently, and Richard Nikoley has written about it before as well. The reason low-carb advocates know this is that reducing carbohydrate generally reduces triglycerides, often below 100 mg/dL. This is the range at which the Friedewald equation becomes unreliable, resulting in artificially inflated LDL numbers that make you have a heart attack just by reading them.

I had a lipid panel done a while back, just for kicks. My LDL, calculated by the Friedewald equation, was 131 mg/dL. Over 130 is considered high. Pass the statins! But wait, my triglycerides were 48 mg/dL, which is quite low. I found a paper through Dr. Eades' post that contains an equation for accurately calculating LDL in people whose triglycerides are below 100 mg/dL*. Here it is (mg/dL):

LDL = TC/1.19 + TG/1.9 - HDL/1.1 - 38

I ran my numbers through this equation. My new, accurate calculated LDL? 98 mg/dL. Even the U.S. National Cholesterol Education Panel wouldn't put me on statins with an LDL like that. I managed to shave 33 mg/dL off my LDL in 2 minutes. Isn't math fun?

*This equation was designed for individuals with a total cholesterol over 250 mg/dL.

Letter to the Editor

I just got a letter to the editor published in the journal Obesity. It's a comment on an article published in October titled "Efficiency of Intermittent Exercise on Adiposity and Fatty Liver in Rats Fed With High-fat Diet."

In the study, they placed rats on a diet composed of "commercial rat chow plus peanuts, milk chocolate, and sweet biscuit in a proportion of 3:2:2:1," and then proceeded to simply call it a "high-fat diet" in the title and text body, with no reference to its actual composition outside the methods section. We can't tolerate this kind of fudging if we want real answers from nutrition science. Rats eating the "high-fat diet" developed abdominal obesity, fatty liver and hyperphagia, but this was attenuated by exercise.

As I like to say, the problem isn't usually in the data, it's in the interpretation of the data. The result is interesting and highly relevant. But you can't use terminology that tars and feathers all fat when your diet was in fact high in linoleic acid (omega-6), low in omega-3 and high in sugar and refined grains. Especially when butter and coconut oil don't cause the same pathology. I pointed out in the letter that we need to be more precise about how we define "high-fat diets". I also pointed out that the study is highly relevant to the modern U.S., because it supports the hypothesis that a junk food diet high in linoleic acid and sugar causes metabolic disturbances and fatty liver, and exercise may be protective.