From real Sessions to controlled promotion on Harbor Self-Evolving

Why self-editing is not progress

Mon, 01 Jan 0001 00:00:00 +0000

An Agent can rewrite its prompt, tools or Evaluator and still become worse. “It changed” is not “it improved.”

A trustworthy loop fixes the business cases, execution identity and scoring contract before the change. It separates the Optimizer that proposes a change from the Gate that judges promotion.

You do

Name the business failure and the owner of the final deployment decision.

Harbor records

The identities and evidence chain needed to revisit the claim.

Define the four concepts

Mon, 01 Jan 0001 00:00:00 +0000

Self-evolution is not “letting the model edit itself.” It is an inspectable learning chain: Dataset defines the problem, Generator produces results, Evaluator judges evidence, and Optimizer proposes one controlled change. The Evaluator is itself tested through Meta-Evaluation.

Meet the four roles

Dataset — business tasks and failure cases that answer what to evaluate.
Generator — runs the Candidate and produces answers or Artifacts: who answers and how.
Evaluator — applies criteria and Evidence requirements: what counts as good and whether evidence is sufficient.
Optimizer — proposes one new-version change from failure evidence: what to change next.

In an Agent system, each role is broader than a model. Generator includes prompts, Skills, tools and runtime; Evaluator includes rubric, Judge, parsing and validity rules; Optimizer may be a Coding Agent constrained by the project Skill.

Diagnose recent Sessions

Mon, 01 Jan 0001 00:00:00 +0000

Historical evaluation lowers the cold-start cost. Preview bounded Session metadata, inspect what may reach the Judge, then confirm a non-promotion Job.

You do

Select recent completed Sessions that reflect the failure, review exclusions and consent to the disclosed Judge/data boundary.

Harbor records

A private redacted Batch, frozen Judge identity, one Trial per Session, criterion applicability, coverage and reason codes.

This does not prove

Historical scores are not comparable Candidate regression evidence and never go through Promotion Gate.

Turn badcases into a Dataset

Mon, 01 Jan 0001 00:00:00 +0000

A Dataset is curated business intent, not a dump of private transcripts. Abstract the failure, preserve the relevant constraint and create an independently reviewable expected behavior.

You do

Deduplicate failure patterns, remove private identifiers, separate tuning from holdout and review every instruction file.

Harbor records

Task ids, paths, instructions, sensitive-metadata checks and immutable source digest.

This does not prove

A clean Dataset does not prove the population is complete or that one metric captures all business risk.

Run one controlled regression

Mon, 01 Jan 0001 00:00:00 +0000

Snapshot the Candidate, validate the Dataset, doctor the Stack, preview Context and run diagnostic mode before promotion-eligible work when the path is new.

You do

State one hypothesis and one allowed change. Keep Dataset, criteria, provider and runtime fixed unless the change deliberately creates a new baseline.

Harbor records

Manifest digests, Context v3, model/Judge identities, Trial output, criterion Evidence, validity, coverage and governance impact.

Read the Gate and hand off

Mon, 01 Jan 0001 00:00:00 +0000

Compare the Candidate Job with a comparable baseline under an explicit Promotion Policy. Read the overall decision together with regressions, coverage and invalid criteria.

You do

Review representative evidence, confirm the policy matches business risk and decide whether external CI/CD should consume the recommendation.

Harbor records

Baseline/Candidate Job identities, Policy identity, comparison details and deterministic Gate artifact.

This does not prove

PROMOTE is not a production deployment, Champion mutation or universal quality guarantee. It is a recommendation for the frozen evidence and policy.