Evaluation Programs: Managing AI at Scale

Portfolio-Level Risk Aggregation: From Systems to Enterprise View

Most evaluation literature focuses on evaluating individual systems. But organizations deploying dozens or hundreds of AI systems face a different problem: how do you assess the collective quality and risk of your AI portfolio?

Individual system metrics don't aggregate simply. If you have 40 AI systems, each with 95% accuracy, what's the portfolio risk? It's not 95%—it's the combined probability that at least one system fails in ways that matter. If each system independently has a 5% failure risk, the portfolio has much higher compound risk of multiple failures.

Aggregation Framework

An effective portfolio risk framework maps system-level metrics to portfolio-level risk using importance weights:

• System risk score: Combine system-level metrics (accuracy, fairness, safety) into a composite risk score (0-100, where higher is riskier). Different systems weight metrics differently based on what failure modes matter most.
• System importance: Weight each system by its importance to business operations. A core revenue-generating system is more important than a beta feature. Weight by users affected, business impact, and regulatory requirements.
• Portfolio risk: Portfolio risk = sum(system_risk_score × system_importance_weight), normalized. This gives a single number representing portfolio-level quality and risk.
• Risk thresholds: Define portfolio risk thresholds triggering actions. If portfolio risk exceeds threshold, trigger organizational response: pause new deployments, increase evaluation frequency, or reallocate resources.

Accounting for Interdependencies

Systems in the same portfolio interact. A failure in the authentication system might cascade to failures in all downstream systems that depend on it. A bias issue in the data preprocessing pipeline affects all systems using that data. An effective framework identifies these dependencies and accounts for them when aggregating risk.

Technologies supporting this: graph-based risk models where nodes are systems and edges represent dependencies, with risk propagating through the graph. If system A depends on system B, and system B's risk increases, system A's derived risk also increases.

Prioritizing Systems: Which to Evaluate Deeply vs. Lightly

You cannot evaluate all systems equally. Resources are finite. Portfolio evaluation requires triage: determining which systems need deep evaluation and which can be evaluated more lightly.

Risk-Based Prioritization Matrix

A standard approach uses a 2×2 matrix:

Impact × Risk: Place each system on a matrix with impact (how many users, how much business value) on one axis and risk (probability and severity of failure) on the other:

• High Impact, High Risk: Deep evaluation. Mission-critical systems with high failure risk need comprehensive evaluation. Examples: core recommendation systems, fraud detection, content moderation.
• High Impact, Low Risk: Moderate evaluation. High-impact systems that are well-understood and low-risk need ongoing monitoring but not comprehensive re-evaluation. Examples: mature systems with strong track records.
• Low Impact, High Risk: Targeted evaluation. Systems with limited reach but significant failure risk need focused evaluation on their high-risk dimensions. Examples: experimental safety features that might affect a small user segment severely.
• Low Impact, Low Risk: Light evaluation. Systems that affect few users and pose minimal risk need only baseline monitoring. Examples: low-stakes features, beta experiments with limited rollout.

Evaluation Resource Allocation

Allocate evaluation resources proportional to the matrix:

• Dedicate 60% of resources to high-impact, high-risk systems
• Allocate 25% to high-impact or high-risk (but not both) systems
• Reserve 15% for low-impact, low-risk systems and exploratory evaluation of emerging systems

This allocation ensures you're not over-evaluating stable systems while under-evaluating risky systems.

Portfolio-Level Regression Testing: Catching Breaks Across Systems

When you update one system, do you break others that depend on it? Portfolio regression testing extends single-system regression testing to the portfolio level.

Dependency Mapping

First, map system dependencies. Which systems depend on which? This might be explicit (system B calls system A's API) or implicit (system B uses data preprocessed by system A's pipeline). Build a dependency graph.

Regression Test Strategy

When system A changes:

• Direct regression tests: Run your existing tests on system A to verify it still works.
• Dependent system tests: Run regression tests on all systems that depend on system A. Check whether they still work with the new version of system A.
• Integration tests: Test the specific integration points between system A and dependent systems. Do the APIs still match? Do data formats match?
• Smoke tests: For critical systems, run quick smoke tests on the end-to-end pipeline to catch catastrophic breaks before full evaluation.

Portfolio Testing Automation

Manual regression testing doesn't scale to dozens of systems. Use continuous integration pipelines that automatically run regression tests whenever any system updates. Tools like Jenkins, GitLab CI, or GitHub Actions can orchestrate this: when system A's new code is pushed, the CI pipeline automatically:

• Tests system A
• Tests all dependent systems
• Runs integration tests
• Alerts if regressions are detected
• Blocks deployment if critical regressions are found

Portfolio Governance: Decision-Making at Scale

With 40+ systems, governance becomes essential. Who decides when to deploy? Who allocates evaluation resources? Who is accountable if a system fails?

Steering Committee Structure

Most portfolio evaluation programs establish a steering committee:

• Composition: Product leaders, engineering leads, evaluation team lead, ethics/policy representative, and relevant domain experts.
• Frequency: Weekly or bi-weekly meetings reviewing portfolio health, new system evaluations, and resource allocation.
• Decision authority: The committee approves deployments, prioritizes evaluation work, and escalates urgent issues.

Escalation Paths

Define escalation thresholds triggering committee attention:

• Portfolio risk exceeds threshold: Automated alert when overall portfolio risk crosses a defined limit. Triggers committee review of high-risk systems.
• System exceeds risk threshold: A single system's risk exceeds acceptable limits. Triggers focused discussion on that system.
• Regression detected: A system update causes unexpected regressions. Triggers decision: rollback, fix and re-test, or accept the regression with documented justification.
• External incident: A system causes a real-world incident (user complaints, regulatory inquiry, media coverage). Triggers immediate investigation and potential remediation.

Deployment Gates

Define objective criteria that must be met before system deployment:

• System risk score below threshold
• Fairness metrics acceptable across defined demographic groups
• Safety evaluation shows no unacceptable risks
• Evaluation coverage ≥ defined minimum
• Regression tests pass on dependent systems
• Steering committee approval (for high-risk systems)

Automated systems can check most gates. Deployment only proceeds when all gates are satisfied, providing accountability and consistency.

Portfolio Dashboards: Different Views for Different Audiences

Evaluation results need to be communicated to different stakeholders with different information needs. Portfolio dashboards should surface relevant information for each audience.

Executive Dashboard

Executives see portfolio-level risk, not system details:

• Portfolio risk score (0-100 scale, trend over time)
• Number of systems in each risk category (high risk, medium, low)
• Trend: is portfolio getting safer or riskier?
• Critical alerts: which systems currently exceed risk thresholds?

This dashboard allows executives to understand portfolio health and make resource allocation decisions.

Practitioner Dashboard

Evaluation teams see system-level detail:

• All systems with current metrics
• Which systems are scheduled for evaluation this sprint
• Evaluation progress: which tests are passing, which failing
• Regression alerts: which recent changes caused regressions
• Historical trend: how have each system's metrics changed over time?

Product Dashboard

Product teams see impact of their systems:

• Their system's current metrics
• Evaluation status: ready to deploy? waiting for evaluation? evaluation in progress?
• Required actions: what needs to happen before deployment is allowed?
• Related systems: what systems depend on mine, and are they healthy?

Managing Evaluation Vendor Portfolio

Many organizations don't conduct all evaluation in-house. They use external vendors for specialized evaluation (bias auditing, medical domain evaluation, etc.). Managing this vendor portfolio is itself a challenge.

Vendor Selection Criteria

When selecting evaluation vendors, consider:

• Expertise: Do they have domain expertise in the systems you're evaluating?
• Track record: References from other organizations using their services
• Methodology transparency: Will they explain their evaluation methods and allow oversight?
• Data security: How do they handle sensitive data? What protections do they offer?
• Speed and scale: Can they handle your evaluation volume and timeline?
• Conflict of interest: Are they independent, or do they have financial interests in certain outcomes?

Vendor Evaluation and Monitoring

Don't assume vendors consistently meet standards. Actively monitor:

• Deliver evaluations on promised timeline?
• Quality of evaluation: do results make sense?
• Reproducibility: if you request repeat evaluation, do results align?
• Communication: are they responsive when you have questions?

Keep vendor scorecards tracking these dimensions. Use this data to make renewal decisions.

Case Study: Fortune 500 Managing 40+ AI Systems

Consider a large financial services company managing 40+ AI systems across lending, fraud detection, customer service, and operations. Here's how they structured portfolio evaluation:

System Inventory

• 12 systems in production (core to business)
• 15 systems in growth (expanding users, improving)
• 8 systems in maintenance (stable, minimal change)
• 5 experimental systems (exploring new approaches)

Evaluation Allocation

Based on risk prioritization, they allocated evaluation resources:

• Production systems: Quarterly comprehensive evaluation, continuous monitoring
• Growth systems: Monthly focused evaluation on high-risk dimensions
• Maintenance systems: Annual evaluation, monitoring alerts for unusual performance
• Experimental systems: Initial evaluation before any rollout; light monitoring

Governance Structure

• Steering committee met weekly, including Chief Risk Officer, VP Engineering, VP Product, and evaluation team lead
• Each system had an owner (product/engineering lead) accountable for evaluation results and remediation
• Escalation: any system exceeding risk threshold required written remediation plan within 5 days

Results

After 18 months:

• Detected 6 critical issues that would have caused significant harm if undetected (bias in lending system, failure modes in fraud detection, etc.)
• Reduced evaluation costs 40% through smart resource allocation (deep evaluation of high-risk systems, lighter evaluation of proven systems)
• Improved deployment consistency—all systems deployed met clear, objective criteria
• Built culture of evaluation ownership—product teams understood why evaluation mattered and proactively flagged concerns