Strengthening Programs through Research & Evaluation: Lessons Learned in a Cross-Sectoral Youth Mental Health Initiative
By Brandon Martinez, Research Associate
Research findings from a recent evaluation of the Margaret’s Place in Communities Impacted by Trauma-exposed Youth (MP-CITY) MP-CITY program have proven to be useful for strengthening the program’s operations and participant experiences. Drawing from three key lessons learned, the program streamlined its screening process and began translating materials to create a more tailored experience for students.
Program evaluations are useful tools for programs, funders, and researchers to better understand the impact of services on participants and how partnerships fared. One such evaluation of the Margaret’s Place in Communities Impacted by Trauma-exposed Youth (MP-CITY) program identified several ways to improve programmatic operations and participant experiences. Funded by the Criminal Justice Investment Initiative (CJII) — a partnership between CUNY ISLG and the Manhattan District Attorney’s Office — and designed and implemented by Joe Torre Safe at Home (JTSAH), MP-CITY supported the mental health needs of students in two Washington Heights schools. The program offered a tiered array of services that provided youth with individual and group counseling, spaces for healthy discussion about violence and trauma, and multiple school-wide approaches to learning about and addressing the impact of trauma. Using findings from its partnership with a research organization, JTSAH is strengthening its program structure through targeted modifications.
These interim and final evaluation reports of MP-CITY, conducted by the Center for Justice Innovation, identified key lessons learned for implementing cross-sector partnerships, among other recommendations. The use of research findings to inform programmatic enhancements highlights the importance of program evaluations in both documenting and informing program implementation and impact.
Lessons Learned in Developing Cross-Sector Mental Health Services
The MP-CITY program formed cross-sector partnerships with the Dominican Women’s Development Center (DWDC) and two public schools in Washington Heights. Strengths and challenges related to the implementation of the program surfaced in the evaluation highlight three key mechanisms for strengthening organizational collaboration in youth programming.
First, it is important for stakeholders across organizations to hold shared values, cultures, and visions for providing a trauma-informed approach to mental health services. Organizations should consider how they fit within the larger ecosystem of mental health providers, referrers, and existing supports in and around schools (e.g., as entry points, or as primary providers). Understanding the extent to which a program is integrated with its partners, and how institutionalized policies can extend the reach of a program, are important aspects of a program’s vision to discuss with stakeholders. For example, because school and program staff had varied beliefs about trauma, some school staff initially devalued the importance of the program. As such, working together to develop shared values and beliefs could strengthen the program's impact on students through increased staff buy-in.
Because school and program staff had varied beliefs about trauma, some school staff initially devalued the importance of the program. As such, working together to develop shared values and beliefs could strengthen the program's impact on students through increased staff buy-in.
Relatedly, developing a strong communication network between partner organizations encourages continued collaboration, clarity in program implementation and service provision, and improved outcomes for youth participants. In doing so, organization stakeholders are likely to support cross-organizational collaboration and buy into shared program visions, approaches, and logistics. For example, as school staff participated in staff trainings and workshops conducted by MP-CITY staff, they increasingly viewed the program as an essential service to address students’ mental challenges.
Finally, organizations and their partners can work with external groups (e.g., funders and grant managers) and evaluation teams to understand how a program is implemented and the potential impacts on organizational development, operations, and participant needs. Working with evaluators to interpret findings can help to build on program strengths and address potential challenges - just as MP-CITY did with the Center for Justice Innovation’s evaluation.
How MP-CITY Implemented Evaluation Recommendations
Programs don’t need to wait until final findings are published to reap the benefits of an evaluation. Collaboration between an evaluator and program staff throughout the life of an evaluation can help program staff make adjustments that benefit their participants and program operations. Similarly, mid-evaluation reports can also provide useful insights into best practices and opportunities for improvement.
Reflecting the third lesson learned above, the MP-CITY program used findings from the mid-evaluation report to improve its programmatic approach. For instance, the evaluation revealed that a universal trauma screening process was not in place, and in response, the program updated and streamlined its screening process to include a single validated trauma screener. Recent findings from the final evaluation report highlight the importance of translating program materials to additional languages — specifically Spanish — to better suit participant needs and enhance the cultural relevance of curriculum. For example, JTSAH may consider revising the curriculum of its classroom-based program, Youth Empowered to Speak (YES), to make classroom activities more engaging for students (Figure 1). Doing so could provide students with additional opportunities to learn about the topics in a dedicated setting.
figure 1. student takeaways from yes
The Importance of Program Evaluations in Improving Implementation and Impact
The examples described above underscore the value of supporting programs by also funding evaluations. Ongoing evaluation activities and evaluation work products are useful for understanding program implementation and demonstrating program impacts on participants, communities, and organizations. Carefully documenting how youth-serving programs are implemented will help to ensure that they can be replicated in other contexts such as other schools or communities. Moreover, collaborative research partnerships encourage evaluation teams and program staff to work together to identify areas where programs can be strengthened. Finally, summative recommendations and key findings from evaluations provide roadmaps for funders, programs, and related organizations to improve upon program implementation and service provision.
Collaborative research partnerships encourage evaluation teams and program staff to work together to identify areas where programs can be strengthened.
About the Program and Evaluation
The MP-CITY program helped students access mental health services in a safe space and understand the impact of violence and trauma. The final evaluation report of the program, including process and outcome findings, is available here.
More than 20 years ago, Joe and Ali Torre founded JTSAH to save lives and end the cycle of violence and abuse through education. Today, JTSAH operates in 18 schools in three states as well as two international programs.
About the Criminal Justice Investment Initiative
The Manhattan District Attorney’s Criminal Justice Investment Initiative (CJII) focuses on three investment areas—crime prevention, reentry and diversion, and supports for survivors of crime. The MP-CITY program is part of the CJII’s crime prevention-focused investments in Youth, Families, and Communities.
The CUNY Institute for State and Local Governance manages and provides technical assistance to CJII contractors, and conducts oversight and performance measurement throughout the lifetime of the initiative.
Photo by Cavan for Adobe.