Evidence-Based Research Errors

Evidence-Based Research:

Limitations & Potential Dangers

Rory Fleming Richardson, Ph.D., ABMP

Today’s healthcare professionals are faced with challenges in using evidence-based, peer-reviewed research, in clinical practice. With this challenge comes the responsibility to understand how easily it can be misused by organizations and individuals, and mislead an entire field. Most healthcare practitioners are not experts in statistics and research to the

point of being able to dissect and analyze the potential limits of the research which is being presented. The term “standard deviation” refers to the measurement of the amount of deviation from the mean (the average found in a sample). If the patient happens to be too far off the mean (or has too high of standard deviation), their case may be perceived as statistically not significant, rather than a valid variant, resulting in a missed diagnosis and absence of proper treatment.

The first challenge to understanding the statistics is the mathematical science of determining probabilities. It is not intended to provide an absolute answer to clinical questions. Clinical judgement is a combination of best measures, the ability of the clinician to assist the patient in providing the fullest, most accurate self-report, and the insight of the clinician which is based on recognizing not only things that are supported by research, but also things that are

evasive to measure in either a quantitative or a qualitative manner.

This provides an experienced clinician with the opportunity to critically question the scientific assumptions which have been made based on our ability to measure variables and utilize the extent of human knowledge.

Werner Heisenberg, a theoretical physicist, who is best known for his work in quantum mechanics, is also known for the uncertainty principle which was presented originally in his 1927 paper. In essence, the uncertainty principle suggests that no matter how much control is maintained in an evaluation of two variables, there is always some fluctuation which will occur. This qualifies that the conclusions we draw from statistics are likely to omit variations. For this reason, all research is to be assessed with the likelihood that the conclusions will have what is known as “outliers” or cases which fall outside of the predicted norms. When one examines the limits of our ability to measure, the vast variables impacting a single patient’s lifespan, and current status, it is easy to see that any research conclusions are limited at best.

What is being measured? When designing research, one has to make sure that what is being assessed is relevant to the variable(s) being studied. For example, if the research is to determine if a visual distortion is corrected by a specific method, one would not use reading as the measure since it does not specifically test the level of a distortion, but rather the sum of a combination of variables which is not just dependent on a visual distortion.

What is our ability to accurately select a sample of individuals to be researched? A good example of the potential error here is the result of economics. In parts of the world where potential subjects for psychological research are being identified, researchers depend on diagnoses which were not made by clinicians who took extensive efforts to (1) confirm the accuracy of the diagnoses and (2) thoroughly evaluate the possibilities of other differentials.

For example, clinicians who are working in a public clinic have to diagnose individuals with “Major Depression,” but due to limited funding, a full physical examination is often not completed, limited (if any) psychological testing is completed, and interviews with other family members is limited. The training of the clinicians themselves is likely to be limited because the public sectors funding only allows for entry-level clinician salaries, with less than five years of experience, and usually less than a doctorate or any significant research training. This can result in even the research sample selection being contaminated with individuals who, in fact, are inappropriate for a study.

What are the limitations of our measures? Regardless of whether we like to accept it, there are things that exist that can not be measured with our current level of technology. As we discover more about the Universe we live in, we are continuously humbled by the discovery of things we did not know and/or may have misunderstood.

What factors and variables have not been considered? In medical laboratory science, determination of a blood count was to count the number of cells which are able to be viewed in a slide from a drop of blood through a microscope. The assumption is that the rest of the blood in the individual is represented by this drop. Even if our assumption is correct, the time of the day and date that the sample was taken, the environmental dynamics on the body at that time, medications recently taken, other body chemistry, the presence of any variation in bone marrow function, and many other factors, may alter the conclusions drawn.

Each human being is a unique entity impacted by more variables than can be effectively listed here. Besides the genetic variations, from a point prior to conception through each individual’s life, different environmental variables and individual choices mold and shape the individual. In psychological assessment, diagnosticians are responsible for interpretation of findings due to these variables and potential errors.

During the period of time I was completing my graduate training, students were encouraged to become scientists/practitioners. Unfortunately, the better trained a psychologist or other healthcare professional may be the more likely they will become too busy with either research or with practice to be active in the other. The level of demand for practitioners is so high that there are not enough dually trained scientist/practitioners to even start to meet the need. To complicate this, the level of funding is too low to put the best clinicians at the front lines of effectively supervising the entry level clinicians. Despite these severe limitations, we continue to utilize data which is collected in the public sector.

Patients are a source of information and can provide the clinician with some of the more important information necessary to make an accurate diagnosis. As forensic science has challenged clinicians for definitive answers which can be supported by research for litigious purposes, some clinicians have answered this growing demand with becoming more skeptical of what patients report. Thus, the clinician may start to discount the patient’s report and depend more on tests and documented research supporting outcome-evidenced treatment. This trend may be helpful to courts, and deciders of secondary gain, but can leave the patient without proper treatment, hope, and any quality of life. As funding becomes more limited, the likelihood of patients not receiving proper treatment increases and any

assumption that the patient is “getting proper and complete treatment” for their conditions becomes more questionable. As this happens, any outcome research which is conducted runs even more of a risk of error, further misleading clinical judgement.

We are challenged to make sure all diagnoses are accurate, consider any treatment options which appear to help the patient, and maintain “professional humility” in making decisions which could either harm or help our patients and our profession.