South Africa’s advertising watchdog ordered electronic cigarette seller Twisp to pull a radio ad saying its products are “95% safer than smoking”. Yet the claim still appears on its website.

It’s also on the website of a Twisp competitor, Vaperite. Both say it was the conclusion of an agency of the UK’s department of health, Public Health England.

To be clear, the Advertising Standards Authority of South Africa did not judge whether the Public Health England review was credible. It ruled that the claim was not “adequately substantiated” for the products advertised.

In this fact-check, we’ll take over where the advertising authority left off. Is there sound evidence that the smoking of e-cigarettes – more commonly called vaping – is 95% safer than regular ones?

Cigarettes scored 99.6% ‘maximum relative harm’

The Public Health England report, titled “E-cigarettes: an evidence update”, was published in August 2015. However, its conclusion was based on a research report published the year before in the journal European Addiction Research.

The Independent Scientific Committee on Drugs, a UK charity, got together a panel of 12 international experts in London in 2013. The panellists had expertise in a variety of disciplines to do with nicotine and tobacco research, such as medicine, psychiatry, policy and law. (Note: Dr Kgosi Letlape, president of the Health Professions Council of South Africa was one of the panellists.)

During a two-day workshop, the experts used a model called multi-criteria decision analysis (MCDA) to estimate the harm of products that contain nicotine. When faced with an important decision, a group of people using the MCDA method will select the “best” or “most preferred alternative” among a set of options. Such a decision is made after analysing multiple criteria thoroughly and weighing their importance.  

Using this method, the panel rated electronic nicotine delivery system products such as e-cigarettes to have only 4% “maximum relative harm”. Cigarettes were ranked as the most harmful nicotine product with a score of 99.6%.

Limitations: Lack of data & selection of panel

Before you reach for an e-cigarette, it’s important to read on to two key limitations the panel itself pointed out.

The first is that they had to deal with a lack of data. E-cigarettes are fairly recent products and have therefore not been researched thoroughly when it comes to dependence, illness and associated death. Subsequently, one could expect that e-cigarettes would score very low on those key health-related criteria.

The second is that the expert panel wasn’t selected based on formal criteria, meaning bias could have crept in. This problem is compounded by the conflicts of interest some panellists disclosed.

For instance, Riccardo Polosa, a professor of internal medicine, reported serving as a consultant to Arbi Group Srl, an e-cigarette distributor. Another author, Dr Karl Fagerström, has served as “a consultant for most companies with an interest in tobacco dependence treatments”. One such company is Nicoventures, which was set up by British American Tobacco to develop and commercialise non-tobacco nicotine products.

Lancet calls research ‘extraordinarily flimsy’

Though the vaping horse had bolted, criticism quickly followed. In an editorial a week after the Public Health England report was published, The Lancet medical journal called the research “extraordinarily flimsy”.

This was for the reasons pointed out earlier: the absence of evidence, the way the experts were selected and the multiple conflicts of interest.

The Lancet concluded that “the reliance by Public Health England on work that the authors themselves accept is methodologically weak, and which is made all the more perilous by the declared conflicts of interest surrounding its funding, raises serious questions not only about the conclusions of the Public Health England report, but also about the quality of the agency’s peer review process”.

Two weeks later, the British Medical Journal followed suit. In an analysis, two British professors described the research from which the claim was extracted as “a single meeting of 12 people convened to develop a multi-criteria decision analysis model to synthesise their opinions on the harms associated with different nicotine-containing products”.

The deputy vice-chancellor for research at the Sefako Makgatho Health Sciences University in South Africa, Prof Lekan Ayo-Yusuf, has written extensively on tobacco control and usage in sub-Saharan Africa. He told Africa Check that is “questionable” whether the same percentage difference would be obtained when researchers used accepted risk assessment principles, compared to the panellists’ value judgements.  

What do we know about the safety of e-cigarettes?

So, the paper which was the basis for the claim that “vaping is 95% safer than smoking” has a number of holes. But are there perhaps other, thorough scientific studies to back it up?

Most reviews suggest it’s too early to say for sure. The Cochrane network of health researchers considered all studies up to January 2016 where e-cigarettes were used to help people stop smoking tobacco.

The reviewers wrote: “Expert consensus broadly holds that, based on all available evidence, electronic cigarettes are considerably safer than traditional cigarettes, but further studies are needed to establish their safety profile compared with established smoking cessation aids.”

An earlier review published in the journal Preventative Medicine – looking more widely at the health effects of e-cigarettes – evaluated studies published up until 14 August 2014. It concluded that “due to the many methodological problems, the relatively few and often small studies, the inconsistencies and contradictions in results and the lack of long-term results, no firm conclusions can be drawn on the safety of e-cigarettes. However, they can hardly be considered harmless.”

Because the jury is still out, the US Food and Drug Administration (FDA) has not approved e-cigarettes containing nicotine as a product to help people quit smoking.

Ultimately, the World Health Organization reported in August 2016 that “no specific figure about how much ‘safer’ the use of these products is compared to smoking can be given any scientific credibility at this time”.

In the final instance, “harm to others” must also be considered when evaluating evidence, not only the relative harm at the level of an individual smoker, Prof Lekan Ayo-Yusuf told Africa Check.

“While e-cigarettes might be an ‘efficient’ nicotine-delivery device for an individual smoker, it remains unknown how ‘effective’ this would be in reducing population harm,” he said. When e-cigarettes are available without regulation, possible community harm may include the re-normalising of regular cigarette smoking and the promotion of smoking among young people.  

Conclusion: The claim is unproven

The idea that smoking e-cigarettes are “95% safer” than regular ones comes from a research report that was published in 2014. It was catapulted to public prominence through a 2015 review by an agency of the UK department of health.

The review was quickly lambasted by two leading medical journals, The Lancet and British Medical Journal. Both highlighted that there isn’t yet enough data on this fairly new product to rate its safety and that it was essentially based on the opinion of 12 experts.

Furthermore, the deputy vice-chancellor of a South African medical school told Africa Check that it is “questionable” whether such a difference can be replicated using established techniques.

More research is needed to say with scientific credibility whether e-cigarettes are safer than regular smoking.

Researched by Patrick Ngassa Piotie, a medical doctor hailing from Cameroon and has a master’s degree in public health from the University of Pretoria. He currently works as a researcher at the Wits Reproductive Health and HIV Institute. This article first appeared on Africa Check, a non-partisan organisation which promotes accuracy in public debate and the media. Twitter @AfricaCheck and www.africacheck.org