A guide to consistent evaluation

Transit system performance directly impacts how cities and regions grow and prosper. As a result, transit improvements are increasingly being framed as more than a means to facilitate reliable and timely movement of people and goods.

The level of investment required to deliver major transit means that there is a growing need, in project evaluations, to capture benefits beyond increased mobility and accessibility. Evaluation of transit can capture its contributions to society by assessing project performance from multiple considerations, including alignment with strategy/policy, economic value to society, and financial and technical delivery requirements. 

A robust evaluation allows decision-makers and stakeholders to compare directly the performance of different transit projects against overarching strategic objectives.

For example, economic evaluation has an established set of tools and approaches that can be adapted and applied to most transit projects. This allows decision-makers to compare in clearly defined terms a diverse set of projects that may be addressing the same or different problems/opportunities. 

While economic analysis employs a consistent set of approaches between projects, strategic evaluation is often conducted in an ad hoc manner, which presents two challenges:

• Inconsistency: The strategic performance of two or more projects that were evaluated using different standards or approaches may not be directly comparable.
• Incomprehensive: Key strategic benefits and negative impacts may be misrepresented, absent, or difficult to understand. 

Steer Davies Gleave has recently completed guidance for conducting strategic evaluations in a consistent and comprehensive manner. This approach overcomes the challenges of ad hoc evaluation but is also flexible enough to assess the unique characteristics of a given project. It also allows strategic issues such as social equity, environmental impact, and urban development to be evaluated consistently between projects. 

A key process in this approach is to define the strategic context of a project by considering:

• an evidence based problem/opportunity that describes the key issues to be addressed;
• direct links between the problem/opportunity and the region/city’s goals or objectives;
• a desired end state or vision that speaks to changes against the same goals and objectives; and
• an analysis using a set of tools, that determine progress towards or away from objectives, and are applied consistently to each project.

When a consistent and comprehensive evaluation approach is followed, it allows stakeholders and decision-makers not only to understand the nuances of how a project contributes to broader strategic goals and plans for society but also how the project compares to other proposals and priorities.

Written by Patrick Miller.