The Missing Pieces in Youth-Employment Data Collection and Assessment

Brief

The Missing Pieces in Youth-Employment Data Collection and Assessment

Abstract

This brief explores the ways youth-employment program providers can help young people who are out of the labor force and out of school obtain valuable work experience or further education. The analysis draws upon experiences and lessons learned from a recent day-long event that was sponsored by The Rockefeller Foundation and hosted by the Urban Institute in partnership with the Urban Alliance. That event provided a framing for how youth-employment program providers can partner with other experts, students, businesses, and technologists to improve how program data are collected and used to improve the lives of young people.

Full Publication

There are 6.7 million people ages 16 through 24 are out of work and out of school, greater than the population of 34 US states. From an economic lens, this is a problem too expensive to ignore. Over his or her lifetime, each young person who disconnects from school or work will cost society more than $700,000 in lost earnings, lower economic growth, lower tax revenues, and higher government spending.

Encouraging disconnected youth to enter the workforce or school expands beyond dollars and cents. This opportunity gap—what is keeping these youth from continuing in school or finding meaningful work—disproportionately affects our country’s most vulnerable youth. Compared with white and higher-income youth, nonwhite and poor youth are less likely to complete high school, attend college, and have a job.

Collecting and evaluating reliable, meaningful data is a significant challenge in designing and maintaining programs to help youth enter the workforce or school. Data collection tools and assessments, especially those that measure “soft skills” (such as communication, language, and interpersonal skills), need to be further developed and improved. In a recent day-long event—sponsored by The Rockefeller Foundation and hosted by the Urban Institute in partnership with the Urban Alliance (a nonprofit youth employment and college access provider serving students in Baltimore, Chicago, Northern Virginia, and Washington, DC)—volunteers sought to explore these data issues and challenges. This “DataCamp” brought together technologists, policymakers, researchers, analysts, and service providers to work with data and explore ways to improve it.

Early and meaningful work experience can reduce youth disconnection. Such work experiences help young people attain higher levels of education, obtain future employment, and boost future salaries. Couple this experience with professional or soft-skills training (also linked to educational attainment and wages) and you have a blueprint for an implementable plan to help young people get into school or stable employment and on the path to long-term success.

There are two critical elements youth-employment providers, such as the Urban Alliance, need to implement that this solution requires. First, they need to know the training they provide is both effective and high quality. In other words, they need better and more relevant youth-employment assessment data. Second, they need to prove to business partners the benefits of hiring young people. These issues represent critical performance management issues for nonprofit organizations that are already hampered by resource and budget constraints.

Capturing Relevant Youth-Employment Data

Many youth-employment providers have mastered collecting basic summary statistics for their programs, such as hours of experience, daily attendance, and program completion. These program “outputs” are commonplace measures and are recorded with relative ease. As a result, they often dominate the rhetoric organizations use to publicize their programs.

But these basic numbers are just one piece of what such organizations are trying to accomplish. Many providers strive to assess the impact of soft skills that employers claim are largely missing from their applicant pool. A 2013 survey of senior executives found the largest skills gap in the United States does not pertain to technical skills but to professional skills like communication, creativity, and collaboration. The impact from this work, however, is often missing from program-outcome discussions.

Assessing soft-skills competency should be a priority for providers. It is an essential part of youth-employment programming, backed by both the business and research communities. There are three primary reasons why soft-skill assessments have taken a backseat in these performance management systems:

  1. Soft-skill competency is difficult to observe. Although opinions vary as to the exact soft skills that are most relevant, many identify a youth’s ability to communicate clearly, work in teams, appear professional, and have an open attitude as the most significant. Progress in developing these skills is hard to quantify because it requires subjective assessments. It is meaningful if a supervisor believes a youth shows more confidence over the course of training, but observational comments are difficult to aggregate for an evaluator trying to show programmatic impact. Even with proper training, there is a high degree of subjectivity in asking adults to observe these behaviors. As a result, measuring the performance of an organization’s work tends to rely on more standardized measures like job attainment or college graduation. And though it is crucial to know that a youth achieved employment, it is just as critical to know the why and how to help steer program delivery.
  2. Skill observation is often completed by volunteers. Youth-employment providers who use direct work placements as part of their program rely on the partnering organization’s staff to supervise and direct young people in their organizations. It follows that the employment provider would ask these supervisors to provide the observations necessary for soft-skill assessment; but there are holes in this process that limit the accuracy of such assessments. Though the employment provider is often staffed by experienced youth developers, their corporate counterparts likely do not have such training. Those corporate counterparts or supervisors who are not directly engaged with the service may miss important measures of context, such as age and cultural variation. Moreover, it is important to consider how employers came to their youth supervision role. If the supervisor fashions himself as a mentor or “big brother” figure, he is likely to be biased in recording his observation. No one wants to deliver bad news or be seen as a disciplinarian during his volunteer job. It is difficult to trust the accuracy of such assessments given these hurdles, leaving evaluators with imprecise data. Seeking feedback from young workers about their experience might be another way employers and evaluators can better understand these programs.
  3. Skill assessments need to fit into the nonprofit timeline. Large, multiyear, multisite, multimethod evaluations, such as the one the Urban Institute conducted late last year, are highly valuable because they have the opportunity to affect policy and practice. But they are not always the kinds of evaluations that help the day-to-day mechanics of operating a program. Evaluators need to use assessment tools and strategies that can be quickly delivered and assessed so that program adjustments can be made during the course of the intervention. Assessments that require corralling various stakeholders are often cumbersome and time-consuming, and the processed results fall outside the window of the intervention. Evaluators are forced to depend on indicators that can be captured more consistently.

Partnering with Business

Without partnerships between providers and the business sector, there would not be youth-employment opportunities, so it is important that providers understand the need for such partnerships. Though at-risk youth are a primary focus of employment programs, providers must also make youth-employment attractive to employers by providing tools and training programs that employers need for a valuable work environment for young workers:

  1. Training supervisors in job-placement sites. Though staff at partnering organizations often supervise and direct young workers, additional training can result in better outcomes for the employer and employee. Asking businesses or service providers to train their existing staff may incur higher costs—a potentially significant challenge—but it may also result in better measures of placement programs and outcomes for both employers and employees. Not all people in all jobs are equally equipped to serve as good supervisors or mentors to young workers, thus providing training opportunities may improve program performance and, ultimately, outcomes for young workers.
  2. Providing a measurement rubric for supervisors. Youth-employment providers may want to supply employers with a rubric device to help make assessments easier and more uniform across sites and employees. One example is PerformWell, a collaborative effort initiated and maintained by the Urban Institute, Child Trends, and Social Solutions, which provides measurement tools and resources that service providers can use to manage their programs’ performances. One such measurement assessment tool contains space for 10 workplace and career-skill categories (e.g., attendance, quality of work, communication skills), each with separate spaces to evaluate the employee and definitions of each metric in a table at the end. As another example, Urban Alliance uses an assessment tool that groups skill types into different difficulty levels and by type of task, such as office administration, meeting support, technology, and data entry. And based on feedback from the DataCamp, the Urban Alliance has developed a new soft-skills assessment form, which they will pilot this summer and hope to fully implement this fall.
  3. Measuring what the provider can expect to change. Service providers who do not focus on helping participants change certain behaviors or characteristics (i.e., certain soft skills) should not expect those qualities to change because of the program participation. Providers should work with employers at the beginning of the partnership to clearly identify the skills in which they expect youth to grow.

Employers may need more education about why giving young workers employment opportunities is important. Employers may also need to be made aware that some youth cannot obtain alternative proxies for soft-skills development, such as volunteer work or participation in club activities, because of their low levels of income require them to work. Better data collection and analysis must expand to help show that internships and apprenticeships can help businesses’ bottom line. The following are some of the strategies evaluators can implement to help market the value and importance in youth-employment programs:

  1. Curb recruitment and training costs. Across America, employers hope their new hires will fit in the culture of their organization and perform at high levels. Youth-employment programs can cut down on recruiting, hiring, and training costs in important ways. First, they can offer a “try it before you buy it” model, where the human resources department can observe whether their candidates excel in the unique circumstances of their office environment. This benefit exists event if employers are not looking for full-time hires. Second, businesses that establish a consistent flow of interns through a youth-employment provider will be better able to make production plans and employment hiring schedules. When service providers evaluate their programs, they need to understand the specific needs, industries, and characteristics of both employees and employers.
  2. Increase workplace diversity. Offering internships and apprenticeships to youth have benefits to organizations that reach beyond the completed tasks. Companies are recognizing that a diverse workplace can facilitate new ideas and bring new energy to the organization and its employees. Youth-employment programs offer a direct line to increased diversity, not just of age and race, but also of life experiences.
  3. Offer managerial training to junior staff. Organizations make investments in professional development, but few trainings can rival direct experience. Youth interns need supervisors to manage their projects, set expectations around deliverables, and oversee their growth. Internships make managers out of staff who otherwise would not have had the opportunity.

Next Steps

Youth-employment programs offer a viable solution to the youth-disconnection problem, but data collection tools and assessments need to be improved. The scope needs to be expanded beyond traditionally defined outcomes, with an increased focus on capturing soft skills and employer benefits.

With the support of The Rockefeller Foundation, such strategies were debated and discussed amongst the technologists, policymakers, service providers, and researchers who attended the Urban Institute and Urban Alliance DataCamp last month. But there is more work to do and future discussions and convenings should be based on two primary goals. First, service providers should continue to work together with researchers to create standardized soft-skills assessments that not only are easy to implement and objective but are also backed by both philanthropy and businesses. Second, providers should design strategies to collect employer data that can be used to quantify the benefits of youth employment and corporate partnerships. Youth-employment service providers, businesses, researchers, and other stakeholders can work together to collect better data and use those data to help more youth achieve higher levels of success.

Photo courtesy of Urban Alliance

Cross-Center Initiative

To reuse content from Urban Institute, visit copyright.com, search for the publications, choose from a list of licenses, and complete the transaction.

LATEST IN Adolescents and Youth

To reuse content from Urban Institute, visit copyright.com, search for the publications, choose from a list of licenses, and complete the transaction.