State and Regional Roles in Sector Evaluation

State Role in Sector Evaluation

Developing an Evaluation Framework

The state can play a major role in encouraging and facilitating the collection of the right quantitative and qualitative data that captures the impact of a sector partnership. States will want to consider some version of the following 5 categories of evaluation. They include:

1. Impact on Jobseekers and Workers: How has the sector partnership met the needs of jobseekers and workers for relevant training and employment?
2. Impact on Employers and Industry: How has the sector partnership met the needs of employers for skilled workers? For an in-depth analysis and guide on capturing impact on employers, see the Aspen Institute’s Business Value Assessment.
3. Impact on the Community: How has the sector partnership changed how the “community” works together to meet worker and employer needs?
4. Effectiveness of the Partnership: Is there evidence that the partnership itself is organized, effective and efficient?
5. Systems Change: Has the sector partnership changed the behavior of education systems, workforce development programs/policies, industry Human Resource policies?

As part of the National Governors Association Sector Strategies project that created this toolkit, a draft Evaluation Framework was created by participating states. It suggests a set of “core” and “discretionary” indicators across four of the five above categories (“Impact on Community” is not included).

Remember that sector partnerships take time to set up and to realize results. Because of this, states must be deliberate about collecting data across the early, mid- and later stages of development. Consider that there are at least three stages of development: 1) the early planning and convening phase; the implementation phase; and the results phase. Here’s one way to picture this:

Specific indicators could in theory be determined within each of the above categories of evaluation and across the three phases of development. Washington State uses a dashboard for this approach with their regional Skill Panels.

Assisting Regional Partnerships in Collecting Data

But how can a state really assist regional sector partnerships to carry out the work of collecting evaluative data? First, states must acknowledge that collecting data takes time, and that local conveners may not have that time set aside. Some states are very hands-on, and even do some of the work, like looking into wage records, themselves. Others take the approach of giving sector partnership conveners the tools they need to be successful. This includes a variety of data templates, survey tools and reporting formats that regional conveners use to collect the right information. Here are some examples:

• Here are several examples of reporting templates that states use to collect data from regional partnerships. These include measures that capture impact on workers and employers, as well as effectiveness of the partnership itself.
• Here’s an example of report template in the form of a how-to guide, used in Massachusetts to help conveners identify goals for impacting business and industry: "WCTF Business Impact - Getting Started."
• Here are two other examples from Massachusetts that give partnership conveners two tools for surveying jobseekers and incumbent workers. Keep in mind that this is obviously for sector partnerships that have a training component (remember: not all sector partnerships involve training).

Regional Role in Sector Evaluation

Regional conveners play a crucial role in not only collecting relevant data to demonstrate their success, but to define success itself. Only the regional convener and public/private partners can do this effectively. Conveners should ask partners: Based on our understanding of the workforce challenge in this industry, what types of impact do we expect through implementation of a customized solution? Consider the following possibilities:

Impact on Employers and Industry

• Shared costs and risks associated with developing skill training programs
• Increased availability of skills training that responds to the actual needs of industry
• Improved human resource practices
• Reduced turnover due to improved working conditions or opportunities for advancement

Impact on Workers

• Increased availability of good jobs (decent wages, benefits, opportunities for advancement)
• Improved working conditions for entry-level and low-skilled workers
• Expanded work supports for low-income workers that lead to better job retention
• Increased opportunities for education and training for high-demand occupations

Benefits to Communities

• Increased attraction and retention of higher-wage employers
• Reduction of poverty and unemployment
• Increased community cooperation and collaboration
• More efficient and effective use of public resources

Effectiveness of the Partnership

• Evidence of appropriate partners convening, and identifying industry challenges.
• Self-identified short and long term outcomes measures.
• Evidence of a plan of action, or a roadmap with direct line of sight to goals and outcomes.
• New and leveraged funding (sources and $ amounts).

Impact on Systems Change

• New, re-structured education and training
• Changes in how education, workforce, economic development and employers work
• New social and business supports
• Changes in public policy
• Changes in employer practices