How Do We Do Our Research?
It is really important to us here at Open Health Tools that you understand how we go about doing our research, whether it is our nutrition advice, supplement reviews, or other bio-hacking experiments.
To give you an idea of the breadth of subject mattr we cover, our most popular pages include a review of Relief Factor, a comprehensive look at the best joint supplements for athletes, a side by side comparison of turmeric and glucosamine, and even a detailed Neuriva Brain Performance review. While we focus on joint care and physical therapy, we do cover a wide range of subject matter within holistic health.
There are lots of sites on the internet right now dealing with the same subject matter we do. But we like to think we’re different in one very important way; we have a lot of respect for the scientific method, and we all have actual experience researching things like nootropics, probiotics, immune-boosting supplements, and joint health stacks.
Part of that commitment to the scientific method involves us explaining exactly how we go about conducting our research.
Why is this so important?
Well, if you don’t know how we’ve gone about testing the supplements reviewed on this site, for instance, then you can’t have confidence in our conclusions.
If you don’t know our process for reviewing scientific literature, collating resources and drawing conclusions, then how can you possibly have faith in our research? This is one thing missing from almost every bio-hacking site out there – an explanation of how they have reached their conclusions.
Lots of sites talk about their experience or their outstanding qualifications. Having a scientific background is important if you want to be a learning resource for bio-hackers, but experience and qualifications only tell you so much. To really have faith in someone’s research, you need to know about their process.
So what does our research process look like?
Let’s start with our literature reviews. What does our MDL research process look like?
How do we review scientific journals & studies?
How do you tell a good clinical trial from a bad one?
What makes a scientific study good, and what makes one bad?
These are quite easy questions to answer, but it takes some time to properly understand why certain considerations are so important. So, let’s jump right into it.
The first thing to understand is that the ability to cite a study in favor of your position does not mean your position is right. It is incredibly easy to find evidence to support your claims if you look hard enough; there are clinical trials which run counter to every claim ever made about health, longevity, and performance.
The idea is to look at the weight of the evidence on either side. How many studies are there showing that a substance has a particular effect, and how many show that it doesn’t?
If the studies showing a particular nutraceutical does nothing outnumber the trials showing it working by 10 to 1, then you know the substance probably does nothing – the studies showing it working are aberrations.
Likewise, if one study found limited benefits from a supplement but 30 others have found it to have tremendous benefits, then you should assume that it likely does produce benefits.
So what about individual studies? What makes a particular clinical trial, literature review or cohort study useful?
We can definitely prioritize certain types of evidence over others. This is actually the cornerstone of the scientific method; certain types of observations carry more weight than others. Take a look at the evidence pyramid to see what we mean:
Right at the bottom of the pyramid, you’ll see expert opinion. This is why we said learning about someone’s experience or qualifications can only tell you so much. It can tell you that someone has the education to understand the topic, but it doesn’t tell you anything about the validity of their claims. An “expert” might be motivated by pre-existing biases, financial interests, or even ideology.
By contrast, right at the top of the evidence pyramid is meta-analyses. These are studies which look at a large body of similar clinical trials.
For example, in a meta-analysis a researcher might look at 10-50 clinical trials all looking at the efficacy of Lion’s Mane Mushroom as a memory enhancer. They will typically start by gathering as many relevant studies as they can and then exclude any that don’t meet their criteria.
Common criteria for meta-analyses include:
- Study group size
- Study duration
- Quality of controls in place (is the right variable being measured?)
- Biases of authors
They will then take all the studies that meet their criteria and compare the results. At the end, they will say whether, on balance, Lion’s Mane Mushroom is an effective memory enhancer, if it isn’t, or if the evidence isn’t strong enough to say so.
That is why a meta-analysis is so strong – it looks at many studies and identifies the overarching trend of the data, rather than looking at individual clinical trials which may or may not have been carried out in ideal circumstances (not to mention the inherent variability in these trials).
While we’re carrying out our research – that includes substance guides and supplement reviews – we will always endeavor to stick to high-quality meta-analyses whenever possible.
When there isn’t a robust meta-analysis available, we will stick to the higher end of the evidence pyramid (randomized control trials and systematic reviews).
When we’re conducting research for Open Health Tools, we will always follow the same systematic approach to appraising scientific studies. We always look to ensure:
- It was published in a reputable journal
- The authors have no biases (financial, ideological, etc)
- It has not been contradicted by further research by the same authors
- It did not use a tiny sample size
- The study lasted a sufficient length of time
- It did not conflate multiple substances
- It does not use a biased sample
- Adequate controls were in place
Only by checking that studies do not fall foul of these issues can we be sure that we’re giving you good information in every single review.
How we review supplements
This might be the most important thing for you to get to grips with if you regularly read our supplement reviews. If you don’t understand our supplement review process, then you can’t possibly get maximum value from our research.
Obviously the exact process for each review will vary depending on the product in question – what is it designed to do, what ingredients are involved, how is it taken, and so on.
But on the whole, most of our reviews follow the same basic step-by-step process:
- Review candidate identification
- Review of claims made by manufacturer
- Analysis of ingredients and doses
- Overview of our thoughts on the formula
- Side effects consideration
- Price analysis
- Final conclusion – recommend or not
We find that following the same basic rubric for our reviews allows us to more easily compare different supplements in terms of quality, value for money, safety, and efficacy.
We strongly recommend that you read our supplement guides before reading individual product reviews as these will give you a good overall view of the supplement category, as well as what to expect from the highest quality stacks.