How to critically inspect a study report?

When it comes to studies, there is no one size fits all – no one minimum sample size, one ideal methodology or best biomarker. The importance of several aspects of a study will vary depending on the ingredient, health benefit and population being studied, as well as the aims of the study.

Annegret Nielsen, senior consultant at Analyze and Realize, points out the way a study is conducted and reported will vary depending on the aim, ie. whether it is to provide evidence to support a health claim or whether it is conducted to support marketing communication.

That being said, she does point out the gold standard is always a double-blind placebo-controlled randomised trial, and for food and supplements, studies should be conducted within a healthy population to allow for the application for a health claim.

Where to start when critiquing a paper?​

Everyone has a different format that they follow when reading a study paper – many read the abstract first while others might head straight to the methodology and conclusion, and others might go straight to the backgrounds of the authors. There’s no ‘right’ or ‘wrong’, as long as the reader is critically analysing the design and findings rather than simply taking the conclusions as fact.

Miguel Toribio-Mateas, clinical neuroscientist and head of R&D at Chuckling Goat, explains that the abstract can be badly written thereby doing the paper a disservice so it’s important to read past the abstract before making a decision on the study.

“My neuro-divergent brain likes to pick up an article from several angles at a time. I like to know who the authors are... and I dip in and out of the methods section to figure out how the researchers have gone about gathering the data.

“The giveaway is the description of the methods. Who was in the study, how did they provide the data needed, how long for, etc. Those couple of lines can prove vital when trying to assess the quality of a study.”

Sample size​

As well as ensuring the sample size is large enough to create statistically significant results, Nielsen points out that it is essential to measure both the base and the post treatment health markers in order to be able to measure the change. It isn’t good enough to simply rely on the post treatment results and compare the treatment to the placebo group when you don’t know what their base levels were.

There’s a drive towards big data from personalised nutrition companies but Toribio-Mateas notes that bigger datasets doesn’t always mean better, especially when we, as an industry, want to allow for new discoveries and innovation in a wide range of fields of research.

“For innovation’s sake, I wouldn’t automatically discard evidence from well-designed RCTs with a control and only 20 participants on each arm. Testing of hypotheses needs to start somewhere, and if we demand huge sample sizes as a non-negotiable requirement, we’d end up with very few studies, mostly funded by those who already have a lot of power in the field.” 

Results​

When reading the findings of a study we want to see that the effect size is relevant for this specific indication and is the treatment period is long enough to attain the desired results.

However, Toribio-Mateas argues it is beneficial to gain participants’ human input on top of the hard data.

“Usually the participant provides samples. These are anonymised and the person becomes a number. Participant-reported outcomes, obtained by means of validated self-reported measures, are essential in my honest opinion.

“The way we experience health is subjective, and we’re not talking about clinical trials on life-saving cancer drugs. We’re talking about food and food supplements. Let’s hear what participants have to say. I’d love to see more RCTs where participants are actually interviewed so that rich data that escapes the sensitivity/specificity of validated measures can be captured.” 

Confounders​

When it comes to nutrition, and especially when it comes to probiotics, its important to take note of any confounding factors, Nielsen points out.

“If the ingredient is going through the intestine then food intake will have a huge effect on the outcomes.”

She also recommends doing a background check on potential confounders to ensure all are accounted for.

How to tell if there’s bias?​

When deciphering if there is a risk of bias, it is helpful to read the researcher profiles and the sources of funding.

Toribio-Mateas says it’s important not to assume that a company involvement automatically signs prejudice.

“Independently conducted studies are essential but, in my opinion, we need to take independence with a pinch of salt. One thing is a company funding a study and controlling the study protocol, having someone directly involved in the running of the study on their payroll, and quite clearly using the research as an excuse for marketing.

“Another completely different situation is a company providing some funding, which is matched by a legitimate research organisation, i.e. a university using government funding, where the scientists involved are advancing their own research interests.

“If we discard small studies that test ground-breaking hypotheses on the basis of potential funding bias, we risk smothering innovation. We have a huge body of evidence of the mechanics of how pro-, pre, syn-, and postbiotics work. We now need to start getting solid human evidence." 

Layman's translation​

Ewa Hudson, Director of Insights at the biotics e-commerce analysis firm Lumina Intelligence, says that her main irritation when reading scientific studies, is a lack of clarity.

“We need to be told the exact demographic and the exact ingredient that’s been studied and it helps when the findings are provided in simple language to help the reader to understand the impact on the wider market.

“Often the language can be quite scientific and medical and it can be hard to know whether a stated improvement is actually significant or not. Even if the authors just provide a couple of sentences providing their own interpretation of the findings, this goes a long way to helping the reader to better understand their study.”

Accessibility in writing and presentation of studies is key, in Toribio-Mateas’ opinion, as this ensures scientific findings can be understood by a wider audience of readers which is essential in promotion of scientific discoveries.

“Some of the studies that have had the deepest impact on me are written in language that is not overly intricate or technical. Specific terms to the field at hand will be used, of course, but the researchers make it easy for the reader to get access to the meaning of their work.,” he discusses.

“Learning should be for everyone, not for those in an ivory tower. And you shouldn’t need a PhD in bioinformatics to understand whether an intervention has worked or not.” 

How much can we rely on the peer-review process?​

Toribio-Mateas worked in scientific publishing in the past before facing publishing as a scientist so he knows “how much of a pain it can be” to get a paper through peer review.

He says: “The scientific peer review model isn’t perfect, but no model is, so I do trust journals to carry out at least some basic assurance on my behalf. The more prestigious the journal - think of Nature, for example - the more accuracy and the higher the level of finesse I’d expect in the quality assurance.”