Workflows on Harbor Self-Evolving

Evaluate recent Sessions

Mon, 01 Jan 0001 00:00:00 +0000

Historical evaluation is the cold start when you have real Agent interactions but no curated Dataset. It is diagnostic, not promotion evidence.

Web path

Open Historical Sessions in the DSH Harbor page.
Preview up to three recently completed top-level Sessions visible to the current DSH workspace.
Review the frozen Judge identity, projected evidence fields, redaction policy and cost/data disclosure.
Select Sessions and confirm.
The Plugin writes a private redacted Batch, materializes one Harbor Trial per Session and starts a non-promotion Historical Job.

The Agent-tool path can preview up to ten Sessions, but only from the exact current working directory. Preview returns safe metadata and a short-lived owner-bound selection token—not raw Session ids or transcripts.

Candidate evaluation and promotion

Mon, 01 Jan 0001 00:00:00 +0000

Candidate evaluation answers a narrower question than Historical diagnosis: did one controlled change improve a fixed business task under comparable evaluation conditions?

Strict sequence

Snapshot Candidate into an immutable manifest.
Validate Dataset identity, task uniqueness, paths, sensitive metadata and source digest.
Doctor Candidate, Dataset, Evaluation Stack and optional Promotion Policy.
Preview Context v3 and discover comparable baselines before spending on a Job.
Run diagnostic first when the stack or provider path is new.
Change one controlled surface—Agent or Evaluator, not both silently.
Run promotion-eligible regression with fixed identities.
Inspect progress, Trial output, criterion evidence and governance impact.
Compare and Gate against a comparable baseline.

Comparability

A baseline is not comparable merely because it used the same repository. Candidate manifest, Dataset manifest, Evaluation Stack, Context, execution environment and relevant model/Judge identities must satisfy the contract. A changed Dataset digest, stack version, provider identity or runtime boundary can require a fresh baseline.

Evaluator governance and meta-evaluation

Mon, 01 Jan 0001 00:00:00 +0000

Candidate quality and Evaluator quality are separate governance problems.

Interface and inspection

Formal Candidate Evaluators implement harbor-dsh-evaluator/v2. A descriptor identifies implementation kind (script or llm-as-judge), ternary Criteria and a bounded allowlist of editable source files. Inspection omits secret-shaped and local-path-shaped values.

Controlled update

An Evaluator update replaces one descriptor-authorized source file under optimistic concurrency. The caller supplies the expected digest and new Evaluator and Stack versions. The update never runs evaluation or Gate automatically.