Evidence Based / Best Practice
FamilyLightsm: Successor to Bridge to Understanding
The terms "Best Practice" and "Evidence Based" are getting increased attention in recent years. In general they apply to standards of practice that are researched based. "Evidence Based" is generally used to apply to practice that is based upon hard research to produce certain results, given a specific set of circumstances. ("Evidence Based" is almost synonymous with "Research Based.") "Best Practice" refers to a practice that is determined by some authority that research indicates the designated practice is the practice most likely to produce the desired results, given a specific set of circumstances.
(Article continues below box)
We have four very major concerns about research, evidence, and best practices.
The second concern is that most of the schools and programs that are making claims about procedures being research or evidence based or about their practices being best practices are not providing us with documentation of those claims upon request. In some cases, we suspect the claims have some basis in fact after all, but we see no excuse for any school or program making such claims without being prepared with a written documentation of the specific basis for those claims. (more detail)
The third concern is that programs are making "best practice" and/or "evidence based" or "research based" claims based upon research that does not quite match the situation in the school or program about which the claim is made. (more detail)
The fourth concern is that procedures that are based upon behaviorism are more easily researched than methods that are not. In reality, research does not support application of procedures based upon behaviorism in preference to other methods unless fairly constructed research has compared long term outcomes between behavioristic and non behavioristic methods. (more detail)
We strongly urge that schools and programs apply Best Practice and Evidence Based methods whenever practical. When we say "whenever practical" we do not mean that Evidence Based methods and Best Practices should become strait jackets. There is still room for creativity. Remember, that which is Evidence Based was attempted by someone before there was research backing. We do want all schools and programs to be aware of Best Practices and Evidenced Based practices that are relevant to their work, and to apply them unless there is a clear and explainable reason to do otherwise. However no program can base every action or choice of procedure on what is evidence based or a best practice. We want to know that schools and programs are aware of research that might apply to what they do, and can readily demonstrate that what they do is appropriately informed by research and that expansion beyond what research tells us is based upon credible and transparent reasoning.
We realize that some programs are working in cutting edge areas where best practices have not been established and the research is quite limited. Some programs are working with innovations that appear to have potential but have not yet been adequately researched. We encourage that kind of innovation, but want programs to be transparent about what is research based, what is innovation, and what is simply the specific personality or twist of the program at issue. What is known from research must never be ignored. It should also never stifle creativity.
We frequently see and hear the terms "research based," "evidence based," and "best practice" used to promote programs. But when we ask for citations of the research involved and or documentation that someone has designated their work to be a "best practice" in almost every case the school or program has been unable to provide us with that documentation. In fairness to those programs, we believe that some of these claims are credible.
However, when we see a claim of "research based" or "evidence based" or "best practice" claims we want to see a written statement documenting the claims including information on how to access the original published reports of the research at issue. In particular, if the claim pertains to "best practices" we want the written statement to include identification of the organization that has determined that the practice at issue is a best practice. The statement should designate the specific conditions which were studied in the research involved (including the population that was studied), the results of the procedure that were documented, and the degree to which application of the procedure in the school or program corresponds to what was studied in the research and /or designation of "best practice."
When a school or program makes reference to research, "evidence based," or "best practice," and does not provide the written statement we describe in the previous paragraph, we believe misrepresentation must be suspected.
As the public becomes more aware of the designations, "Evidence Based," and "Best Practices," we expect that the terms will be used in a misleading way, so the public should be aware of the real issues. For example, the "Seven Challenges" substance abuse programming (which we at FamilyLightsm highly respect) is an Evidence Based program. However our understanding is that the studies that lead to that designation are based upon a specific population of people receiving other therapeutic intervention at the same time as they are working with the Seven Challenges.
That would suggest that the "Evidence based" designation would really only apply to a population similar to the population studied, including having that particular population receiving other services as well. Among other things, as we understand the Evidence supporting effectiveness of the Seven Challenges, it involves a population receiving additional therapeutic intervention besides the Seven Challenges.
Schools and programs that claim to be Evidence Based because they use the Seven Challenges only do that legitimately if they are serving a population demographically similar to the population participating in the studie(s) that are the basis for Seven Challenges to be Evidence Based and therapeutic support is offered quite similar to what the people in the study were offered. We want to see the fact sheet requested above verifying how the offerings of the school or program matched the terms of the study.
We do not question the validity of Seven Challenges claiming to be Evidence Based. We are only challenging the right of schools and programs adopting the Seven Challenges material to claim that they are Evidence Based if they do not document that their use of the Seven Challenges corresponds to what was studied.
Research on behavioral programs and methods speaks for itself, just as research on any other kind of program or method. But sometimes fallacious inferences are drawn from actual positive research outcomes on behavioral methods leading to claims that this research demonstrates a preference for behavioral programming over other methods. But without comparison studies, this inference is not warranted.
Behavioral methods in treatment are based upon rewarding desired behavior and, in some cases, punishing undesired behavior. Because of this, observing behavior, in some cases recording it quite objectively, is intrinsic to the function of a behavioral approach to treatment. More formalized methods of behavioral treatment require frequent notations of behavioral observations. Turning this data over to a researcher makes research easy. We also know that regardless of the presenting issues, we know that rewards and/or punishments focused on desired/ undesired behaviors bring short term results. By "short term" we mean both that the results appear quickly and that they might be temporary.
By contrast, research on non-behavioral treatment methods requires identification of specific behavioral criteria to indicate the outcome being studied -- notwithstanding the fact programs not working by behavioristic methods are not working specifically for demonstration of specific behaviors.
Research establishing short-term gains in a behavioristic program, in these days of computer data bases, might take no more effort than running a program on data in computerized client records. A comparable research study on a non-behavioristic program requires determination of behavioral criteria that are not useful to the treatment process itself, then very complex data entry. This kind of research is very complex and expensive.
Behavioristic programs, using their over-simplified of showing short-term gains, will use this kind of thing to claim they are more effective than the non-behavioristic programs. But this is a fallacy two ways: What happens very short term in a the midst of a reward-punishment environment, says virtually nothing about long term gains, and in any case does not give evidence that one approach is better than another unless both are being studied and the results compared.
A good example of this is the rivalry between "Applied Behavioral Analysis" (ABA) and "Floortime" as methods of treating autistic children. ABA is almost purely behavioristic; Floortime involves a trained adult spending time "on the floor" playing with the child being treated, both building a relationship and opening communication channels.
ABA advocates speak of the research behind their method, implying that it is more effective than Floortime. However the research we are aware of supporting ABA is strictly the kind we described above. It is short term analysis of gains in a structured behavioral setting. We know of no research appropriately comparing long term outcomes between ABA and Floortime. Until we have that, we simply do not know which is more effective. We suspect the best choice is eclectic. But programs purporting to use ABA because research demonstrates its superiority over other methods either do not understand their research studies or are deliberately disingenuous.
Our conclusion is that schools and programs need to say less and do more where research and "Best Practice" designations are concerned. We see far too little of schools and programs being genuinely and constructive guided by research and adoption of best practices. We see far too much of clearly fraudulent marketing making spurious claims about their adherence to what has been learned from research.
● Schools and programs should examine Evidence Based practices and Best Practices as they develop and refine their methods and communicate a clear rationale for any methods that differ from known Evidence Based practice and Best Practices. (Current compliance, unfortunately, is not common)
● Schools and programs which make promotional claims about their methods being research based or evidence based or a best practice should provide a fact sheet giving specific references to the research on which those claims are based. Those references need to include enough information to permit the reader to know how to access the original research on the Internet or in a well equipped library. In the case of a "best practice" claim, the information sheet should also include identification of the agency or organization giving the best practice designation.
● The fact sheet described in the above bullet should be specific enough that reader can understand the specific procedure being researched, the specific presenting conditions that the research indicates the procedure is effective in addressing (including but not limited to specific demographics of the people used as research subjects), and the specific outcome the research indicates that the procedure yields.
● Research will not be presented as advocating one procedure to be preferable to another unless the research was comparison study between the two procedures.
● Schools and programs will not use terms like "research," evidence," or "best practice" in any manner that would reasonably lead to misunderstanding and or false understanding of what the research at issue actually demonstrated.
Feedback is invited. We will publish selected feedback. Email FamilyLightResponse@yahoo.com
Disclaimer:No program review, no matter how positive, is a blanket endorsement. No criticism is a blanket condemnation. When we express our level of confidence in a school or program, that is our subjective opinion with which others might reasonably disagree. When we assert something as fact, we have done our best to be accurate, but we cannot guarantee that all of our information is accurate and up to date. When we address compliance with our guidelines, you need to remember that these are only OUR guidelines -- not guidelines from an official source. We have also set the bar very high, and do not expect any school or program to be in total compliance. It is not appropriate to draw a conclusion of impropriety (or even failure to live up to conventional wisdom) from our lack of confidence in a school or program or from less than perfect conformity to our guidelines. Some will say we expect too much. Readers are responsible for verifying accuracy of information supplied here prior to acting upon it. We are not responsible for inaccuracies.
This article is currently under revision and contains a mixture of up to date information as of the date shown below and obsolete information. Use this page with caution.
Last revised 11-28-2011
|"Solutions, Not Just Referrals"|
For questions or comments regarding this website, email firstname.lastname@example.org