Fish oils, cardiovasc disease, and the problem with statistics

another meta-analysis just came out finding that omega-3 fatty acid supplementation had no significant benefit in patients with known cardiovascular disease, but the devil is in the statistics.... (see fish oils and CAD jamacardiol2018 in dropbox, or doi:10.1001/jamacardio.2017.5205 ).  Thanks to Steve Stovitz for bringing the article to my attention, as well as  the issue of yielding to an arbitrary measure of "statistical significance"

Details:

--10 trials with 77,917 individuals at high cardiovascular risk

--mean age 64, 61% males, 66% prior CHD/28% stroke/37% diabetes, 83% on statins (but these are averages, the actual numbers varied pretty widely from study to study)

--over mean of 4.4 years, 6273 coronary heart events (2695 CHD deaths, 2276 nonfatal MIs) and 12,001 major vascular deaths

Results:

--randomization to omega-3 fatty acids was associated with:

    ​--coronary heart disease rate, 7% decrease, RR 0.93 (0.83-1.03), p=0.05, nonsignificant

    --nonfatal MI, 3% decrease, RR 0.93 (0.87-1.08), p=0.43, nonsignificant

    ​--any coronary heart disease events, 4% decrease, (RR 0.96 (0.90-1.01), p=0.12, nonsignificant

--also, no significant benefit by subgroups: age < vs >65, male vs female, prior CHD, stroke, diabetes, HDL, LDL, TG, statin use

Commentary:

--there have been pretty variable results for the benefit of omega-3 fatty acids, with benefit found in the large Italian GISSI-Prevenzione trial and the Japanese JELIS trial (which was particularly impressive given the baseline high intake of fish in the population), but there were other negative studies. 

--These omega-3 supplement studies were done based on lots of epidemiological studies finding those people consuming higher fish diets had less cardiovascular disease. A recent meta-analysis found that in 19 studies with 45,637 individuals, increased omega-3 levels (from seafood and as well as a-linolenic acid from plants) were associated with lower risk of fatal CHD in a dose-response manner, about 9% fewer fatal CHD events per 1-SD increase in omega-3's (see fish oils and CAD jamaintmeds2016 in dropbox, or doi:10.1001/jamainternmed.2016.2925). This study was important because it assessed the actual individual blood levels of omega-3’s, which not only controls for baseline omega-3 intake (eg, higher in Japan), but provides evidence of benefit on an individual basis based on their own omega-3 levels. And it does not rely on potentially inaccurate self-reported food consumption. But with the caveat that even though these were multivariate-adjusted evaluations, there might have been unaccounted for biases in those who chose to eat more omega-3 fats.

--a 2017 Am Heart Assn Science Advisory concluded: "Treatment with omega-3 FA supplements is reasonable for these patients [those with prevalent CHD such as a recent MI]. Even a potential modest reduction in CHD mortality (10%) in this clinical population would justify treatment with a relatively safe therapy. We now recommend treatment for patients with prevalent heart failure without preserved left ventricular function to reduce mortality and hospitalizations (9%)" (see cad omega3 AHA recs circ2017 in dropbox or Siscovick DS. Circulation. 2017;135: e8670 ). Of note, they do not recommend a specific omega-3 dose, since the studies used different doses (the doses used in the studies were typically low dose, around 1000 mg of omega-3/day, though the JELIS study did use 1800 mg. there are ongoing studies with omega-3 higher dose, the REDUCE-IT and STRENGTH studies). This low dose used is not enough to lower triglyceride levels.

--there are a few issues with the statistics used in the above meta-analysis:

    --there are concerns in meta-analyses themselves, including which articles they include/exclude, how they combine different studies with different doses of different meds (eg in the above study, the dose of eicosapentaenoic acid varied widely from 226-1800 mg/d and docosahexanoic acid from 0 to 1700 mg/d), combining studies with different patients with different inclusion/exclusion criteria, the possibility that smaller but better and more rigorous studies are dwarfed by less good bigger studies, etc. See prior blog on omega-3 fatty acids and cardiovascular disease by Agency for Healthcare Research and Quality which goes into some of these issues: http://gmodestmedblogs.blogspot.com/2016/09/omega-3-fatty-acids-and-cardiovascular.html 

     --the p-value of 0.05 is an arbitrary one, chosen to show whether a particular intervention is "statistically significantly different" from the null hypothesis (where there is no difference between getting the intervention vs not).  it does not measure effect size or the confidence intervals. the 0.05 boundary actually means that "at least 23% of the time (and typically close to 50%)" one is incorrectly rejecting a true null hypothesis (ie, actually the intervention really does nothing). is there really a big, all-or-none difference between p=0.0499, p=0.05, p=0.051?? see http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-correctly-interpret-p-valuesand http://blog.minitab.com/blog/adventures-in-statistics-2/the-american-statistical-associations-statement-on-the-use-of-p-values  . And even a p<0.05 may mean nothing clinically (a very large study may have a "statistically significant" difference which is trivial clinically, such as a difference of 0.5 mmHg in a large blood pressure study, yet a smaller study may need a much larger effect size to be similarly "statistically significant".  Hence the importance of looking at confidence intervals and effect size.

    --my sense in general in interpreting p values is that it also really depends on the context: the intervention being done, its safety over the longterm, if there are other adequate treatments available, the magnitude of the effect size, etc. So, in the above omega-3 meta-analysis, i would point to a few issues:

        --the association with decreased coronary heart disease deaths was 7%, with p=0.05 (ie, pretty darn close to being <0.05). and the confidence intervals were 0.83-1.03  (ie, potential of 17% benefit but 3% harm; ie, though the p value is not <0.05, the confidence intervals are skewed to the side of benefit and are actually consistent with a pretty strong 17% benefit for omega-3's)

        --the association with any coronary heart disease events was decreased by 4%, with p=0.10, but the confidence intervals were from 0.90-1.01 (ie, potential of 10% benefit but 1% harm, again skewed to benefit)

        --and looking at the forest plots of the 10 individual studies in the meta-analysis, all but 2 favored treatment for decreasing nonfatal MIs, coronary heart disease deaths, any coronary heart disease, and major vascular events. ie, the individual studies pretty consistently showed a trend to benefit

        --Also, omega-3 fatty acids have been around for a long time, and have minimal and minor adverse effects (small risk of bleeding, stroke, and possible risk of too much mercury if derived from fatty fish unless this is tested for).  

            --I would argue that the case of omega-3's is pretty different from the case for yet another new and untested med which poisons several enzyme systems (eg DPP-4 inhibitors for diabetes), where we have pretty good and known safe alternative therapies (eg, see http://gmodestmedblogs.blogspot.com/2016/04/diabetes-dpp-4-inhibitors-and-risk-of.html  ).  These types of new shotgun therapies should be tested for a much longer time to see if there are ultimately bad adverse effects.

            --but, on the other hand, we may want to be more "lenient" with a new drug for a pretty untreatable condition (eg, HIV in the bad old days or hep C drugs), where the potential benefit could far outweigh the less-than-overwhelming statistical analysis. and these drugs may not be "statistically significant", not because the drug is ineffective, but because the sample size is too small. or the groups involved were not the best. or the wrong outcome was measured (eg, benefit may be difficult to show in those with minimal disease, but perhaps subgroup analysis shows benefits for those with more advanced disease, etc).  

            --so, it seems reasonable to me to hold different interventions to somewhat different standards

        ​--but, to reiterate, the p value is only one statistical marker. it may be much more clinically relevant to look at the confidence intervals and the actual treatment effect

so, i bring up this meta-analysis for a couple of reasons:

--i actually do think that omega-3 supplementation may be beneficial. probably not as much as a statin or aspirin, but it probably is a useful adjunct and should be strongly considered

--and i do think it is a pretty good poster child for us to think about how we use and interpret statistics. And as the old saying goes: "there are three ways to not tell the truth: lies, damned lies, and statistics" [though, i do think statistics are important, just need to be interpreted in a broader context.....]