Change Types and Review Path
Reviewing and implementing changes to the AI agent requires two additional roles: maintainers and a generative AI (gen AI) committee. All proposed updates are classified to ensure the review rigor matches risk:
- Type A—Minor edits/clarifications: typos, wording clarifications, formatting adjustments. Approved by relevant maintainer(s) and the communications department (if necessary).
- Type B—Adding/adjusting tests and examples: New evaluation scenarios, rubric examples, or minor scoring guidance changes with minimal impact on AI tool outputs. Approved by relevant maintainer(s) and the communications department (if necessary); spot-checked by another lead when feasible.
- Type C—Control or gate change: Changes to required checks, required approvals, or required artifacts. Approved by maintainer and the communications department, reviewed by the gen AI committee, and signed off on by accountable owner.
- Type D—Major framework change: Structural changes to the framework (new principles, new evidence tiering approach, major re-scoping, etc.). Approved by maintainer and the communications department, reviewed by the gen AI committee and an independent reviewer, signed off on by accountable owner. Consider a wider stakeholder comment period if necessary.
Standard Contribution Workflow
- Submit: Contributor or other member of the committee submits a change proposal using the template.
- Triage: Maintainers acknowledge and classify changes (type A–D), assign an owner, and set a target release.
- Review: Relevant leads review for correctness, scope, and alignment with evaluation/security/transparency/responsible agentic AI expectations.
- Make necessary updates: For type B–D changes, required tests and documentation updates are run or updated (e.g., evaluation rubrics, security test categories, release artifact template).
- Decide: For type A and B changes, the maintainer decides. For type C and D changes, the gen AI committee decides and documents the rationale; the accountable owner signs off on accepted risks/exceptions.
- Merge and version: Changes are merged into the authoritative repository or location, and the toolkit version is incremented.
- Release artifact: A release note or audit trail summary is produced for the change set and shared with toolkit users in some form.
- Monitor and learn: Maintainers and relevant lead(s) collect feedback from pilots and projects, log issues, and incorporate this information into future releases.
Cadence and Operating Rhythm
The recommended cadence balances speed and quality:
- Monthly to quarterly: Maintainer triage + minor releases (type A/B).
- Quarterly: Planned release with bundled improvements and any type C items requiring committee review.
- Annually (or at major release): Review of security test categories and governance effectiveness; refresh of templates and training as needed.
Required Artifacts
- Change proposal (problem statement, rationale, impact, mapped risks, and proposed edits)
- Updates/updated rubric/test documentation (as applicable)
- Decision record (who approved, when, and why; risks mitigated vs accepted)
- Release artifact/audit trail summary for each toolkit version release
Body
Next section: Glossary and Additional Resources