By Sacha Litman and Rob Goldberg
It's a Friday afternoon at State University, and Hillel is hosting its popular Mitzvah Day program, which draws hundreds of first-year students, athletes, fraternity brothers and sorority sisters. It is a programmatic gem that combines serious Jewish content and fun. Just as arriving students are about to enter the event, they meet a checkpoint where a friendly Hillel professional asks, "would you please swipe your Hillel connection card?"
This scenario is a dream for some and a nightmare for others. Proponents of tracking student participation argue that tracking would allow Hillels to analyze how many "touches" they have had with students, identify students' frequency of program attendance, follow up with students who have gone long stretches without being involved, and target new programs to students who have attended related events in the past. The proponents include many Hillel donors and top leadership who want to measure the "return" on their investment.
Opponents, mostly Hillel professionals, contend that tracking student participation is intrusive behavior that would defeat Hillel's hard-fought victory to be perceived as an engaging institution on campus for any Jew, no matter who he or she is. Furthermore, these opponents contend that measuring participation accurately is impossible; it would require recording every single non-programmatic interaction between Hillel professionals and Jewish students — such as an impromptu conversation while standing on line at the campus caf — conversations which can easily number in the thousands for a Hillel professional each year.
Of course, both arguments are valid. After all, a key purpose of Hillel is to create positive Jewish experiences to engage college students, particularly those without strong Jewish backgrounds. This requires building trusting relationships. However, it is nearly impossible to stay on top of each relationship when the ratio can be as high as 1,000 students per professional, and with many students appearing only a few times a year.
Hillel formally began to tackle this gap between tracking and relationship-building in 1998 when it created the Campus Strategic Services Group (CSS). CSS developed a self-assessment survey, now in its sixth year, asking Hillel professionals to report how many unduplicated students were in Hillel's student email database and how many were connected to Hillel in expanding concentric circles of participation:
• Core leadership
• Frequent participants
• Infrequent participants
Despite much initial resistance, these measures have become part of the Hillel culture. It is not uncommon now for a Hillel director to tell a prospective donor, visiting parent, or a colleague that "Twenty percent of our 2,000 Jewish students are frequent participants, 50 percent are infrequent participants, and 70 percent of the Jewish students are on our email lists, which is an improvement of 10 percent over last year's figures." Furthermore, Hillel has used the data to show Hillels who have not met their potential how their participation levels stack up against those of their better-performing peers. This data has helped campuses see the gap between what Jewish life is and a vision of what it could be, initiating organizational turnarounds.
Yet, this level of data analysis does not go far enough. First, the numbers are approximated by Hillel professionals retrospectively at the end of each academic year. While our sense is that the approximations are probably within 10 percent of the real figures, our hope has been that all Hillels are likely to skew their approximations in the same positive direction, thereby making comparisons across schools and longitudinally across years fairly valid. But there are too many hopeful assumptions here and we must update the approach.
Furthermore, these concentric-circle measurements do not examine the quality of engagement. The measures favor Hillels that hold many "big draw" social programs with low levels of Jewish content or impact. This shifts Hillels away from "significant" impact achieved through smaller group interactions and the concomitant qualitative interactions that occur. To be sure, some qualitative indicators can be taken into account when tracking program participation. For example, additional weight can be given to not only participation but to the Jewish content level of the program attended. Also, by tracking a student's participation in programs over time, a Hillel can develop a rich tapestry of knowledge about the quality of the relationship with that student and that student's developing Jewish identity. Without more refined, real-time systems in place to track participation during the course of the year, Hillel is falling short.
Unfortunately, many well-intentioned program directors or fellows have created a simple tracking system only to find that behavior unrewarded, not built upon by the Hillel or maintained by the director over time. It is easy to let the student tracking slip for a week, and then for a month, and then just stop doing it altogether because it is an extra, aggravating step in the work process. Only a Hillel director who prioritizes measurement will advocate for collecting the data and regularly analyze the data, using it to decide which students to target with new programs or how to allocate staff to follow up with students.
A culture of student tracking must be set by Hillel's Schusterman International Center. The International Center must work with directors and create incentives for them to place this work higher on their priority list. Ultimately, the International Center must capitalize on newly emerged technology to help Hillels to track student participation at minimal time investment, all the while remaining vigilant about not invading a student's space.
Under the leadership of a new volunteer task force on measurement and evaluation, Hillel's International Center is considering several exciting new ideas for tackling the student participation challenge field-wide. By using several complementary methodologies at once, the committee hopes to triangulate on Hillel's true participation and impact. First, Hillel will redesign its self-assessment survey to gain more drilled-down and consistent information on participation. Second, Hillel is considering a more pervasive market survey to track the strength of Hillel's "brand image." Finally, Hillel is considering customizing Web-based software and providing accompanying training to simplify the tracking process for many Hillels.
By collecting data from Hillel professionals via the self-assessment survey, from students via campus surveys, and by developing student tracking software, Hillel hopes to move to the next level in measuring the success of its efforts and gain the ability to provide this information to donors and boards eager to see that their investments are resulting in increased student participation and increased quality of participation.
Sacha Litman is principal and founder of Measuring Success, whose clients include Hillel, the United Jewish Communities and Partnership for Excellence in Jewish Education. He can be reached at sacha@measuring-success.com. Rob Goldberg is Hillel's Vice President of Professional Leadership Development and Campus Advancement. He can be reached at rgoldberg@hillel.org.
Reprinted with permission from Contact, the quarterly journal of Jewish Life Network/Steinhardt Foundation. This story originally appeared in the Summer 2004 issue.