An Evaluation Framework for State Sector Strategies

This paper provides an overview of the background leading up to the framework and an explanation of the development process. It should be viewed as an evolving conversation among states, as well as a menu of evaluation ideas for states implementing or interested in designing regional, employer-driven partnerships that address skills-based economic development needs (i.e. sector initiatives). At least a dozen states are using state funds and policy tools to promote sector initiatives in their regions. All are experimenting with the best ways to measure their success. Recognizing the value of cross-state sharing and peer learning, the states participating in this project came together to jointly develop this framework as an option for states to use in evaluating their sector initiatives.

The goals of the Evaluation Framework (“Framework”) are:

• To have a framework for analyzing the results of sector strategies that can be adopted and adapted by states based on their individual needs;
• To design a benchmarking process that captures continuous improvement by states implementing sector strategies;
• To transition the system to be employer driven;
• To develop intra- and inter-state shared learning (and to save individual state
• time/resources);
• To provide employers with context and comparisons that lead to employer
• investment and advocacy; and
• To tell what happened, to put it into context, to tell the story of that context to multiple audiences, and to make decisions that improve the context.

In addition to establishing a set of goals for the Framework, a set of operating principles were agreed to and guided the work. These principles are: first, acknowledge that every state is at a slightly different phase of implementation of statewide sector strategies, so measurements must be broad enough to apply to states just beginning to design sector policy as well as states in the very mature stages of sector policy. Second, the primary intent of the Framework is to measure aggregate statewide efforts, not local area initiatives, but adaptations of the framework could be used to help guide local initiatives. Third, the measures should be realistic and feasible to collect, and they should already exist in current systems. This recognizes that performance measurement criteria already exist and often can be burdensome. It was the desire of the state members that the Framework not add to that burden. Finally, the Framework must include both quantitative and qualitative measures, with the latter based on a “preponderance of evidence” criteria.

Over the course of development, three guideposts were kept in mind: (1) What we evaluate will drive behavior; (2) “Measurement should stay close to where the dollars are operationalized,” i.e. measures need relevancy to funding sources; and (3) As the framework evolves, states should think about how data will be collected and aggregated (maintaining the focus on ease of collection and minimizing the extra burden).

After a year of development, yet recognizing that the Framework will greatly benefit from further input and experimental application, presented below are the four groups of measurement in the Framework. Each group has several sub-categories of impact, which will be described in greater detail in the full narrative sections of the report and in Appendix A that follow.

The Evaluation Framework: Groups and Measurement Categories

Measuring Outcomes of Workers

• Employment
• Earnings
• Skill Acquisition and Development

Measuring Value to Employers

• Human Resource Measures/Stabilization/Yield
• Business Productivity and Operations

Measuring Quality and Effectiveness of Partnerships

• Turning Planning, Information, and Data into Consensus and Action
• Partnership Development, Management, and Capacity
• Implementing a Worker Pipeline Aligned with Industry Needs
• Sustainability of Partnership
• Sustainability of Intent of Work

Measuring Systems Change

• Education, Training, Social, and Business Supports