—by Matthew K. Weiland and Paul M. Kubek
Editor's note: This article first appeared in print in the Fall 2007 issue of SAMI Matters newsletter, a publication of the Center for EBPs at Case (click here). ((update link to SMI matters newsletter)
Cleveland, OH—A Research Infrastructure Grant (RIG) awarded to the Center for Evidence-Based Practices (EBPs) at Case Western Reserve University has enabled the Center to expand its capacity to access and analyze fidelity-review data gathered from service organizations that are implementing Integrated Dual Disorder Treatment (IDDT) and/or Supported Employment (SE). Both are EBPs for people with severe and persistent mental illness. The evaluation of this data will help the Center's consultants and trainers disseminate practical knowledge to service providers, helping them improve the outcomes of their work with consumers.
The RIG was awarded in late 2005 to the Center by the Mandel School of Applied Social Sciences—the professional school of social work at Case—to support and promote research that improves behavioral healthcare practices. The Center is a partnership between the Mandel School and the Department of Psychiatry at the Case School of Medicine. Lenore A. Kola, Ph.D., assistant professor of social work at the Mandel School and co-director of the Center for EBPs is the principle investigator.
Six Years Of Data Collection
According to Deb Hrouda, MSSA, LISW, assistant director of research and evaluation at the Center, the RIG Project marks an important step toward understanding facilitators of and challenges to EBP implementation. She explains that there is much published research about the effectiveness of the IDDT and SE models as well as published research about the processes of organizational change in general. However, there have been very few research studies that have examined organizational characteristics and other characteristics—from big-picture macro issues like service-system policies to frontline clinical issues such as level of education and training of direct-service staff—that positively and negatively affect IDDT and SE implementation over time.
The Center has been providing evaluation services and other forms of technical assistance for EBP implementation to Ohio organizations since the onset of Ohio's IDDT initiative in 2000. For instance, the Center assembles teams of implementation experts who conduct annual fidelity reviews of IDDT and SE programs. To date, the Center has conducted over 200 fidelity reviews at 68 outpatient- and inpatient-IDDT programs and 13 SE programs in Ohio. Thus, the Center has collected a wealth of data on the 12 organizational characteristics (components) and 14 treatment characteristics that comprise the IDDT model and its fidelity scale.
The Center has also collected fidelity scores for each fidelity scale item as well as data that describes changes in those scores over time.
"We have acquired a great deal of experience, practical knowledge, and skills, which have led to the creation of a 40-page booklet about how to implement IDDT with success," Hrouda says, referring to Implementing IDDT: A step-by-step guide to stages of organizational change, published in 2006 (get resource). "The RIG Project gives us the opportunity to be more scientific about what we know and do as an organization. This new understanding will add to the knowledge base about EBPs and will give us some evidence to inform our consultation and training services."
The Evaluation Database (TED)
Under the direction of Hrouda, the RIG Project has been conducted in multiple phases, including the following:
- Building/programming The Evaluation Database (TED)
- Entering six years of fidelity-review data
- Collecting and entering additional organizational information not asked for in the typical fidelity review process (see sidebar)
- Conducting initial analyses of data
- Creating a web-based fidelity-rating structure
As part of data collection for this project, RIG team members asked chief executive officers and other representatives of 30 service organizations to participate in a substantial interview and questionnaire process. Phone interviews and surveys were completed between September 2006 and January 2007. Hrouda notes that the individuals who generously gave their time and attention to this project have contributed to the knowledge base that will help improve services to Ohio residents.
RIG Project team members have completed programming of The Evaluation Database (TED) and its web interface. They have also entered all fidelity-review and organizational data and have conducted some data analyses.
Initial Results: 24 to 36 Months to Reach High Fidelity
Notable initial results show that programs which approach or reach high IDDT fidelity (mean scores of four or more) do so following 24 to 36 months of implementation. Hrouda notes that this finding may change as the Center conducts more analyses. Thus far, fewer than one-third of Ohio's IDDT programs have been engaged in implementation for two years or more. Therefore, it is not known if this timeframe of 24 to 36 months is common to all organizations. Some organizations might take longer to reach the benchmark.
"The timeframe is fairly reasonable when considering that, for some teams, IDDT is a significant shift from their former way of approaching recovery services," says Ric Kruszynski, MSSA, LISW, LICDC, director of IDDT consultation and training at the Ohio SAMI CCOE, a program of the Center for EBPs at Case. He adds that the initial results from the RIG analysis should help implementation teams set realistic expectations for their efforts. Service team members sometimes feel discouraged and community stakeholders sometimes get impatient and desire immediate results. The CCOE's consultants and trainers often have to reassure team members, managers, and executives that their attention to programmatic and clinical details today will pay off in the near future.
Facilitators and Barriers of Successful Implementation
According to the Center for EBPs' Director of Research and Evaluation Barbara L. Wieder, Ph.D., the evaluation database enables the Center to ask research questions about a range of factors that contribute to or impede successful implementation over time. The database will not only help the researchers evaluate the effectiveness of each organization's implementation efforts, but it will also help researchers evaluate the effectiveness of the Center's technical-assistance services.
Patrick E. Boyle, MSSA, LISW, LICDC, director of implementation services at the Center for EBPs and Ohio SAMI CCOE, explains that there are many factors that might influence the success of IDDT implementation—for instance, a service organization's financial strength, the level of education and training of staff, staff turnover, or a community's support of mental health and substance abuse services via tax-levy renewals.
"We have learned that very few communities look and act the same," Boyle says. "There are so many factors that influence successful implementation of innovative practices. We want to know what they are, and see if we can identify some similar characteristics across the various sites." He adds that this knowledge will help the CCOE's consultants and trainers tailor their consultations more precisely to different types of organizational and community cultures across Ohio and across the country.
Boyle further explains that one of the biggest values of the RIG Project is that it provides a system for service organizations throughout Ohio's 88 counties to continue to contribute information about their practical, day-to-day work with consumers in their communities. By doing this, they are actively improving the behavioral healthcare system in Ohio and throughout the country. Boyle concludes that gathering data from each IDDT and SE program about consumer outcomes and organizational outcomes will be an essential next step for the RIG Project.
Matthew K. Weiland, MA, is senior writer and producer at the Center for EBPs at Case. Paul M. Kubek, MA, is director of communications. . . . The Ohio Substance Abuse and Mental Illness Coordinating Center of Excellence (Ohio SAMI CCOE) is a program of the Center for EBPs at Case.