Skip to content

Optimize

aworld-cli optimize is the phase-1 command entrypoint for manual self-evolve runs. The CLI parses the request and prints run artifacts; aworld.self_evolve owns target inference, candidate generation, replay, evaluator integration, gates, release checks, and apply/rollback semantics.

Use this command to create a proposal, debug target inference, run verified replay, or drain framework-owned pending self-evolve jobs.

For normal task execution with background self-evolve enabled, use the runtime flag instead:

aworld-cli --evolve
aworld-cli --evolve=online --judge-agent ~/Documents/agent.md --judge-model-profile judge
aworld-cli run --task "complete this task" --evolve=shadow

--evolve configures the selected local runtime agent for post-run self-evolve scheduling. Pair it with --judge-agent, --judge-agent-name, or --judge-backend-ref to configure the background evaluator. Use --judge-model-profile when the evaluator should use a different named model profile from the main task agent. aworld-cli optimize remains the manual/debug entrypoint for the same framework runner.

Framework Context

AWorld self-evolve architecture

One aworld-cli optimize execution creates a Run: candidate variation, selection, replay, verification, and optional release. Verified policies may continue through a bounded Campaign only when typed causal progress permits another cycle. A Goal session is not an implicit retry path; framework or shared-code work crosses that authority boundary only through an explicit handoff. See Self Evolve for the complete framework design and ownership model.

Basic Usage

Proposal-only runs:

aworld-cli optimize --target skill:demo --dataset eval.jsonl
aworld-cli optimize --target skill:login --from-trajectory trajectory.log --apply proposal
aworld-cli optimize --task "improve login retry guidance" --from-trajectory trajectory.log
aworld-cli optimize --from-trajectory trajectory.log --include-prior-runs --apply proposal
aworld-cli optimize --from-trajectory-set trajectory-set.json --include-prior-runs --apply proposal

Verified apply for an allowlisted skill target:

aworld-cli optimize \
  --from-trajectory ~/Documents/trajectory1.log \
  --apply auto_verified \
  --new-skill-policy auto_verified \
  --judge-agent ~/Documents/agent.md \
  --judge-timeout 600 \
  --judge-model-profile gpt-5.5

--judge-model-profile is the name of a configured CLI model profile. It does not directly set a provider model id. Add --target skill:<name> when you want to bypass target inference.

Targeted Skill evolution contract

Use --skill-evolution-contract <json> when the Campaign must converge on a specific Skill capability contract instead of treating any verified score improvement as completion. The contract binds capability IDs to frozen dataset case IDs, supplies required behaviors and preserved invariants to candidate generation, and can require multiple consecutive verified cycles.

{
  "schema_version": "aworld.self_evolve.skill_evolution_contract.v1",
  "target_skill_id": "agent-browser",
  "objective": "Handle large browser output without losing evidence",
  "capabilities": [
    {
      "capability_id": "bounded_large_output",
      "description": "Read large output with byte-bounded evidence extraction",
      "case_ids": ["task_large_body", "task_large_tail"],
      "required": true
    }
  ],
  "preserved_invariants": ["Preserve ordinary navigation behavior"],
  "minimum_required_coverage": 1.0,
  "required_stable_cycles": 2
}

aworld-cli optimize \
  --target skill:agent-browser \
  --dataset eval.jsonl \
  --skill-evolution-contract agent-browser.contract.json \
  --apply verified_only \
  --max-improvement-cycles 4

A case covers its capability only when candidate execution succeeds and the runtime attests both the exact candidate Skill package and an observed candidate intervention. Unpublished candidate packages are visible only in the isolated self-evolve replay child; normal runtime discovery continues to hide candidate, draft, and rejected release states.

For each replay Task, the framework copies the selected package into a Task-owned immutable mount and verifies its fingerprint before starting the child CLI. The task-time resolver binds its activation evidence to that Task. If an explicitly isolated candidate cannot produce this evidence, execution fails closed before spending the rollout budget; it cannot silently fall back to an ambient same-name Skill.

Bounded self-improvement campaigns

--apply auto_verified starts a bounded self-improvement Campaign. A Campaign keeps the original source, target-selection policy, verification contract, cumulative budget, and exact run lineage. Its default maximum is three cross-run cycles; override it with --max-improvement-cycles N. Set the value to 1 to preserve single-run behavior. --iterations remains the separate within-run candidate-population budget.

When the caller does not configure a token ceiling, the CLI preserves the existing 500,000-token allowance for each permitted cycle and derives the Campaign hard ceiling as that allowance multiplied by the cycle cap. Each run is still capped at 500,000 tokens. An explicitly supplied total_run_token_budget or legacy max_run_tokens remains a Campaign-total ceiling and is reduced by every completed run's usage.

After a rejected run, the framework continues only when typed causal evidence identifies either a newly progressed candidate-owned repair frontier or a retryable infrastructure failure. It stops on an unchanged frontier, non-retryable failure, policy/permission/evidence denial, or cumulative budget exhaustion. A framework-owned or shared blocker creates a validated Goal handoff instead of allowing a candidate to modify protected framework code. Cross-run progress is monotonic: losing an already-passing gate, recovery achievement, or reached stage is a regression, and a newly observed failure fingerprint alone is not improvement. Bounded prior feedback loads the verification-quality champion first, so a weaker recent run cannot replace a stronger repair base. Material candidate quality is also tracked in coarse buckets. A whole-point score improvement, a groundedness-tenth improvement, a better command or deterministic verification signal, or fewer failed repetitions can continue the bounded Campaign only when groundedness, command pass rate, recovery achievements, and already-passing gates do not regress. Smaller judge-score variation does not consume another cross-run cycle. An empty candidate population caused by a schema-valid but non-materializable patch is reported as a repairable candidate-generation event and receives one bounded same-run representation retry; it is not exposed as a legacy missing-disposition pause.

Resume an active or paused Campaign without replacing its source, target, or verification contract:

aworld-cli optimize --resume-campaign <campaign-id>

--resume-campaign is distinct from --from-run --rerun-evaluator: Campaign resume may start the next bounded improvement run, while evaluator resume only reuses already verified replay artifacts from one run. Complete, budget-limited, and exhausted Campaigns cannot be resumed.

If the previous local Campaign worker died before producing report.json, resume verifies that its lease is no longer live, preserves the incomplete run as an interrupted-attempt audit artifact, and retries the same bounded cycle. It will not take over a live/foreign lease or an interrupted apply operation. The incomplete attempt is conservatively charged its full reserved allowance, so recovery cannot bypass the Campaign's cumulative token, cost, or wall-time ceiling.

Drain pending post-run jobs:

aworld-cli optimize --drain-pending

Resume evaluator/gates from a previous run:

aworld-cli optimize --from-run <run_id> --rerun-evaluator

Evaluator-only reruns preserve the original candidate package fingerprint and therefore cannot rebind an inferred new-skill candidate to another run-owned draft path. Rerun the full optimize command for an inferred draft; existing-target runs can use --rerun-evaluator normally.

Data Sources

Exactly one evaluation source is normally provided:

  • --dataset <path>: JSONL eval dataset. Rows may include input, expected_output, and verification_command.
  • --from-trajectory <path>: trajectory log used to build trace packs and infer failure patterns. When the log contains multiple task trajectories, the framework automatically groups them by inferred target/task family, recovery opportunity, and context completeness before candidate generation so unrelated or unreplayable tasks do not pollute a single candidate.
  • --from-trajectory-set <path>: advanced explicit-control input for baseline trajectory collections. Most manual optimize workflows should use --from-trajectory; user-authored set files may contain baseline and operator_added members only. Framework-owned accepted, rejected, and replay members are imported from self-evolve run history rather than hand-authored by users.
  • --from-session <id>: session-backed dataset construction.
  • --batch-config <path>: batch config for a larger request.
  • --from-source <path>: an arbitrary readable file or directory. The framework defaults to the registered auto ingestor, preserves all source units, and compiles structural or semantic evidence into a frozen internal schema before target inference.
  • --frozen-ingestion-id <id>: an immutable prior semantic ingestion. Use it to promote an already reviewed graph with explicit approval and qualification artifacts without rescanning source files or rerunning the semantic model.
  • --from-run <run_id>: previous run artifacts, usually with --rerun-evaluator.

Agentic file/directory ingestion

--from-source separates the flexible external format from the stable self-evolve dataset protocol. It accepts a single file or a directory; users do not need to write --source-ingestor auto.

# Scan, map, validate, and freeze only
aworld-cli optimize \
  --from-source ~/Documents/domain-data \
  --ingestion-only

# Constrain auto discovery with a manifest
aworld-cli optimize \
  --from-source ~/Documents/domain-data \
  --source-manifest ~/Documents/domain-data/aworld-source.yaml \
  --target skill:domain_agent \
  --apply proposal

# Free-form, production-trusted two-stage flow
aworld-cli optimize \
  --from-source ~/Documents/domain-data \
  --source-manifest ~/Documents/domain-data/aworld-source.yaml \
  --ingestion-only

aworld-cli optimize \
  --frozen-ingestion-id <ingestion-id-from-first-run> \
  --semantic-evidence-approval \
    .aworld/self_evolve/ingestions/<ingestion-id>/evidence_approval_template.json \
  --semantic-qualification-report ~/Documents/qualification.json \
  --target skill:domain_agent \
  --apply auto_verified

# Select an SDK-registered structural domain ingestor
aworld-cli optimize \
  --from-source ~/Documents/domain-data \
  --source-ingestor crm-export-v2 \
  --target skill:crm_assistant \
  --apply proposal

--source-manifest constrains the default flow; it is not a separate ingestion mode. --ingestion-model-profile selects the isolated semantic/mapping deployment when deterministic decoding is insufficient. Custom ingestor names are resolved from IngestionRegistry; CLI import strings and generated parser code are not executed.

The external format is deliberately flexible. One Markdown file or many files may freely combine Harness trajectories, human comparisons, historical judge results, and analysis. The semantic swarm extracts this into framework-owned entities, cited claims, source dispositions, conflicts, actionable signals, evaluation plans, and trace attestations. It cannot grant authority or claim qualification. A strict aworld.self_evolve.canonical_semantic_source.v1 JSON/YAML source instead uses the deterministic canonical decoder with zero model calls; any canonical marker makes mixed/unknown/dangling/control-field input fail closed. Conflicting payloads in the same semantic slot must be declared as a typed conflict; undeclared contradictions are rejected and declared unresolved conflicts cannot pass auto_verified.

Free-form input remains proposal/human-review evidence unless two independent trust requirements pass:

  1. --semantic-evidence-approval is an operator-selected JSON artifact bound to the exact logical/provenance graph, SourceBundle, constitution, semantic profile, explicit manifest, and claim scope. --ingestion-only writes the review template; merely placing approval text in the source or a conventional manifest grants nothing.
  2. --semantic-qualification-report is bound to the exact model/provider/protocol/constitution/corpus/threshold identities and its fingerprint must be listed in .aworld/self_evolve/semantic_qualifications/index.json. Failed, expired, false-authority, drifted, unlisted, malformed, duplicate-key, or symlinked artifacts are rejected. Reports contain an issuance and expiry window. The versioned qualification corpus carries source-only single/multi-file inputs. The exact-deployment runner gives the tested deployment only a random opaque token and the raw documents, validates the returned frozen semantic snapshot against the exact source/model/provider/protocol, and derives scored outcomes in framework code. Gold case IDs, scenario tags, and labels are not exposed. Scoring compares real source locator/hash, canonical entity identity, claim payload/direction/relations, conflict membership, and signal case/execution relations. Required-claim recall and accepted-claim/conflict precision penalize omissions and every unmatched extra output; gold values are never copied from kind/count matches. Production reports must declare exact_snapshot_v1, the framework runner protocol fingerprint, and a per-case source/snapshot attestation bundle fingerprint. Recorded-outcome reports remain offline-only even when all metrics pass.

Qualification proves the semantic deployment, not the truth of a source claim. Approval proves operator authority for a frozen evidence graph, not model quality. Supplying only one cannot pass auto_verified. Canonical typed sources use framework deterministic verification and therefore need neither. For free-form evidence, the recommended second stage uses --frozen-ingestion-id: the framework recompiles authority, qualification, plans, quality, and the ingestion identity from the reviewed snapshot without reading source files or invoking the semantic model again. The qualification check time is frozen for audit/reload, while every new auto_verified admission rechecks report expiry against current time. A proposal or shadow snapshot cannot be reused under a verified runtime mode without deterministic promotion. Campaign creation performs that promotion before checkpointing and stores only the promoted ingestion ID, not mutable approval/report paths. The workspace registry is a framework-owned JSON file:

{
  "schema_version": "aworld.self_evolve.semantic_qualification_registry.v1",
  "trusted_report_fingerprints": ["sha256:<report-fingerprint>"]
}

SDK-registered ingestors and extractors cannot grant themselves trust. framework_builtin identity is registry-owned; workspace_allowlisted entries require a stable configuration fingerprint explicitly included in the registry allowlist; unlisted/custom entries remain external_untrusted and cannot pass auto_verified. A returned snapshot's name, version, trust, extractor fingerprints, case counts, coverage, and other derivable quality metrics are checked by the framework. Registered semantic snapshots are additionally fail-closed: external snapshots cannot supply authority/qualification, and allowlisted registered semantic authority is disabled until the framework can re-derive its claim-level attestations.

Every successful ingestion writes an immutable snapshot under .aworld/self_evolve/ingestions/<ingestion_id>/. Normalized private cases use owner-only permissions. A normal run records ingestion_ref.json and a dataset_ingestion gate. Baseline/candidate evaluation, evaluator reruns, and all cycles of one Campaign reuse that frozen snapshot and split; changes to the raw directory after Campaign creation do not alter the active dataset. Evaluator-only reruns fail closed if ingestion_ref.json is absent or if its source, mapping, normalized dataset, or split fingerprint differs from the recipe/snapshot. They also recheck the frozen rollout mode and current qualification expiry, so a proposal run cannot be upgraded to auto_verified and an expired report cannot be reused through --rerun-evaluator.

Flexible source formats do not mean an unconstrained internal protocol. Stable case/signal/plan IDs, logical and physical provenance fingerprints, complete source dispositions, citations, entailment, conflict preservation, deterministic splits, trace attestations, authority context, qualification evidence, and frozen snapshot identity remain mandatory. auto_verified requires every train/validation plan to be verified-eligible before optimizer projection.

The CLI summary includes ingestion status, normalized case count, record coverage, rejected-record count, and mapping-model call count. Mapping-model calls are conservatively debited as the frozen-dataset-ingestion item in the existing candidate-generation run/Campaign budget ledger; they are not treated as free work.

When --target is omitted, the CLI sets infer_target=True and the framework performs credit assignment. The inference inventory is filtered to target types with a registered CLI adapter before scoring. Phase 1 registers the skill adapter only, so automatic inference cannot select an unsupported prompt-section, tool-description, config, or workspace-artifact target. Low-confidence inference remains blocked when it would mutate an existing skill. A validated capability gap may instead create an isolated run-owned draft because draft evolution does not authorize mutation of an existing target.

Skill matching ignores generic action tokens such as search, read, open, and write when they appear only in tool names. Such tokens do not prove that an installed skill owns the failed behavior; target selection requires a specific skill identity, a non-generic anchored alias, or a validated capability fingerprint.

Target confidence and target provenance are independent gates. Confidence answers whether the trajectory evidence identifies the right capability; provenance answers whether that capability is authorized for mutation. Target intent independently records whether the run mutates an inventory target or creates a new draft. Inferred drafts are stored under .aworld/self_evolve/<run_id>/draft_target/<skill_id>/SKILL.md; they never use another run's draft as a baseline.

--new-skill-policy controls inferred capability gaps:

  • disabled: return a typed no-target result without creating a draft.
  • draft_only: generate, replay, and evaluate a run-owned draft, but never publish it.
  • auto_verified (default): permit publication to aworld-skills/<skill_id>/SKILL.md only after the ordinary verified gates, collision checks, and post-apply checks pass.

The apply policy is still authoritative: --apply proposal never publishes, regardless of the new-skill policy. allow_generated_target_mutation retains its separate SDK meaning and does not authorize new-skill publication. Adding skill to auto_apply_target_types is also insufficient by itself.

Supplying --target skill:<name> is strict operator selection. If that skill does not exist, optimize fails instead of falling back to inferred creation.

Aggregating multiple trajectories can strengthen target evidence and add evidence IDs, but it produces one provenance decision for the selected target. Aggregation cannot erase inventory protection or grant write trust. Behavioral replay and evaluator success never substitute for mutation authorization. Reports record the single provenance sidecar path, or a structured unresolved reason when classification is impossible.

Replay Adaptation and Portability

trajectory.log and the other data sources are dataset inputs. They may come from the current workspace or from a completely different rollout machine. The framework does not execute historical file paths or compare a candidate directly with the historical answer.

Before replay, the framework compiles each replayable case into a portable plan. It abstracts workspace paths, snapshots bounded local inputs and non-secret environment metadata, records external prerequisites, and creates one content-addressed workspace seed. Baseline and candidate repetitions each start from a separate copy of that same seed. This isolates skill changes from workspace mutations and host drift.

Multi-record trajectory logs preserve their conversation boundary by session id. When a later record is a natural follow-up, replay receives a bounded chain of prior user/assistant turns, with early user-provided URLs and artifact anchors protected from eviction by long recent answers. A follow-up with no recoverable same-session context fails adaptation as context_incomplete rather than entering a misleading candidate replay.

Context incompleteness propagates across dependent follow-ups but not across a later independent request in the same session. Automatic grouping reports context_completeness_rate, max_recovery_opportunity_tier, and opportunity-kind counts for each target group. Recovery opportunity is an ordinal structural class: unrecovered failure, recovered path, repeated-action loop, or none. Ranking considers only complete members; after selection, incomplete members are listed under excluded_context_incomplete_case_ids and omitted from replay. This applies to one or many trajectory members without a single-case branch.

Stateful external resources require a deterministic registered adapter. An unbound live URL, local endpoint, stateful browser/tool name observed in the source trace, missing continuation context, secret-like file, or unknown external path fails the replay_adaptation gate before rollout. The candidate remains available in proposal mode, but cannot pass auto_verified.

Strict baseline reuse requires the same target, current-skill fingerprint, dataset fingerprint, adaptation fingerprint, workspace-seed fingerprint, cases, and requested repetition count (an exact match). Legacy replay artifacts can still be opened for inspection, but an evaluator-resume that could authorize verified apply requires adapted replay provenance. Rerun the full optimize flow for legacy artifacts; they are not reused as a new strict baseline.

Candidate Generation, Repair, and Preflight

The CLI builds a typed, bounded EvolutionContext and runs each candidate-generation slot as an isolated AWorld task. The context contains only trainable cases, bounded trace evidence, reusable lessons, capability requirements, prior validation feedback, and acceptance constraints. Candidate output must match the framework JSON package contract; a skill candidate may contain SKILL.md, a bounded patch intent, and candidate-owned replay files such as replay/capability.json, a compiler, and a runtime.

Judge evidence feedback is normalized into typed evidence_repair_constraints. Constraint identity excludes evaluator prose and task payloads, so the same failure can be aggregated across repetitions and trajectory members. The owner and source layer decide whether the next action is a candidate repair or a framework/infrastructure blocker. In particular, a valid canonical bundle that becomes incomplete only after bounded judge projection is attributed to the framework boundary until a complete projection can distinguish unsupported candidate output from missing judge context.

Artifact projection is a bounded continuation protocol rather than a single prefix read. The evaluator can continue an indexed artifact from next_start or omit start for automatic continuation; overlapping reads are denied. Read-round, per-request, and cumulative character budgets are declared in the prompt and clamped by the runtime. If the round budget is consumed, the judge receives a finalization call and must return typed constraints instead of another read request. This behavior is uniform for one or many trajectory members; it does not encode task- or artifact-specific repair rules.

The framework ranks the generated population and validates repair candidates before an expensive task rollout:

  1. candidate_repair_conformance first proves that the candidate materially changed the failed source branch and that the request operation participates in the response data flow. Unresolved compiler/runtime owners are preserved together, and an internally inconsistent owner contract fails as a framework error rather than consuming candidate attempts. Strategy-switch telemetry distinguishes a requested switch from a verified change in owner, edited-path, or control-flow topology.
  2. The candidate-owned replay capability is compiled and frozen with immutable fixture and package fingerprints. Recorded operation responses are exposed through AWORLD_REPLAY_RESPONSE_INDEX.
  3. Compiled probe declarations must cover the observed operation and assert a non-empty value derived from the recorded response payload.
  4. The frozen runtime is started in the replay subprocess sandbox and every declared HTTP/TCP/WebSocket readiness and protocol probe is executed. Required WebSocket probes validate handshake, ping/text exchange, operation correlation, non-empty result, and recorded-response binding.
  5. Conformance always runs when a repair contract applies, including for a dataset with one replayable trajectory. Multiple trajectories are grouped by a stable semantic fingerprint covering the capability, service, operation, transport, recorded-response structure, exact probe, and assertion set. Equivalent shapes may execute once with every affected case ID recorded; every distinct shape is validated.
  6. Only a candidate that passes conformance proceeds to optional representative task screening and then the authoritative paired baseline/candidate rollout. Representative screening is a cost-control signal, not evidence that unselected conformance shapes work; the final paired replay remains the authoritative dataset-wide behavioral check. A sole candidate normally skips ranking replay, but a candidate repairing a previously observed replay counterexample must pass that counterexample through screening before it can consume another authoritative slot.

This sequence is generic: contracts are compiled from observed operations, protocol traces, fixture provenance, and the candidate package. There is no target-specific repair adapter for one trajectory case. A source-conformant candidate can still fail at runtime preflight; that result is intentionally reported before the longer task rollout and fed into the next focused repair iteration.

Repair inheritance respects causal depth and mutation authority. Once a candidate has completed replay and received judge metrics, its candidate-owned replay files are frozen, even when it owns zero replay files; subsequent evidence-quality repair is restricted to reusable target behavior. Constraints from lower-level sibling failures remain diagnostic lessons but do not force a judge-scored lineage back into compiler/runtime mutation. Candidate semantic deduplication also normalizes line endings and terminal blank lines while preserving internal source changes, preventing formatting-only retries from consuming the bounded Campaign.

Candidate-owned replay files are accepted only when replay preflight declares a capability requirement or the focused repair lineage already owns such files. When the trajectory has no replay requirement, generated compiler/runtime files do not expand the mutation surface; the candidate remains a target-behavior change. This prevents dependency-free cases from creating and then repairing an unnecessary harness.

Candidate repair gate diagnostics are summarized in a separate population conformance section in report.json; optional task sampling remains in screening. Probe-group reports contain stable fingerprints, status codes, and bounded affected-case IDs, never raw recorded-response values. When execution preflight is reached, bounded service/probe artifacts are stored under .aworld/self_evolve/<run_id>/repair_conformance/<candidate_id>/. Increasing --replay-timeout affects task rollouts but does not weaken or bypass exact repair probes.

Options

  • --agent: agent name or id used by replay/evaluator request context.
  • --task: task text used by framework target inference and dataset context.
  • --target: explicit target reference. Phase 1 CLI runs support skill:<name> end to end. Automatic inference uses the same adapter registry and therefore considers skills only. Other target forms remain framework/SDK types until general CLI adapters are implemented.
  • --iterations: maximum candidate optimization iterations.
  • --apply: proposal, verified_only, or auto_verified. The default is proposal. verified_only performs the same verified decision process as auto_verified but preserves the accepted package without publishing it.
  • --max-improvement-cycles: hard cross-run Campaign cap for auto_verified; defaults to 3.
  • --resume-campaign: resume a persisted active or paused Campaign using its immutable source/target/verification contract and remaining cumulative budget.
  • --new-skill-policy: disabled, draft_only, or auto_verified. The default is auto_verified; it affects inferred missing capabilities only.
  • --judge-agent: markdown judge agent path.
  • --judge-agent-name: configured custom judge agent id/name.
  • --judge-backend-ref: evaluator backend reference.
  • --judge-model-profile: named model profile for the judge. This is useful when the task agent uses the default .env model but the evaluator needs a separate model. It applies to markdown judges and local named judge agents. For --judge-agent <agent.md>, the same profile can also be declared in markdown front matter as model_profile: judge; the CLI option takes precedence.
  • --replay-timeout: timeout in seconds for each replay rollout.
  • --replay-total-timeout: optional hard wall-time deadline for the complete paired replay. Cases run progressively as a baseline/candidate pair, so a timeout can preserve completed causal pairs as well as completed baseline members instead of discarding the partial experiment. A bounded restart cursor is written to .aworld/self_evolve/<run_id>/replay/<candidate_id>/members/paired_replay_checkpoint.json; the timeout gate exposes it through diagnostic_refs. Verified replay uses the v2 measurement-control scheduler by default; an explicit --measurement-mode off retains the bounded legacy safety horizon. The v2 path uses planned checkpoint and Campaign deadlines. Both deadlines are cancellation-safe: they terminate the replay process group before control returns to candidate search. A shared measurement timeout stops the current candidate population instead of spending another authoritative slot.
  • --replay-max-runs: maximum aworld-cli run iterations per replay rollout.
  • --judge-repetitions: successful judge samples to aggregate per evaluator call.
  • --judge-timeout: timeout in seconds for each judge attempt.
  • --baseline-replay-repetitions: number of baseline replay rollouts.
  • --candidate-replay-repetitions: number of candidate replay rollouts.
  • --from-run: reuse artifacts from a previous self-evolve run.
  • --rerun-evaluator: reuse replay artifacts from --from-run and rerun evaluator/gates only.
  • --include-prior-runs: include prior self-evolve reports for the same target as advisory trainable cases. This does not replay old runs or expose raw trajectories; framework lesson memory still controls what feedback enters candidate generation.
  • --drain-pending: drain pending framework-owned post-run self-evolve jobs.

--apply write and --apply branch are intentionally unsupported in phase 1. Proposal artifacts are written by the framework store; direct file writes and branch management are outside the CLI contract.

Auto-Verified Defaults

When --apply auto_verified is used and the caller does not override values, the CLI supplies stricter defaults:

  • --judge-repetitions 1
  • --judge-timeout 120
  • --baseline-replay-repetitions 2
  • --candidate-replay-repetitions 3
  • --iterations 10

The 2/3 replay values are sparse-data defaults. On a sufficiently broad independent-case panel the runner adaptively uses 1/1 unless the caller explicitly provided either repetition option.

Proposal runs default to one iteration. auto_verified uses the larger budget because each verified runtime repair can expose a new protocol frontier. The runner stops early when verification succeeds or progress stalls, and it may grant up to six bounded extension iterations only for newly observed repairable failure families. Duplicate candidates and repeated failure families do not consume unbounded retries.

The CLI also enables framework replay for auto_verified. Skill candidates must have candidate replay evidence, evaluator evidence, deterministic or objective verification signals, passing gates, and a post-apply runtime-loader check before they can remain applied.

Evaluator availability is checked before those quality gates. An evaluator run with no successful judge signal is reported as evaluation_runtime_health/infrastructure rather than being converted into score, evidence, required-verification, or held-out candidate failures. The environment fingerprint is fixed by the first adaptation in one run; any later drift stops replay as environment_fingerprint_drift.

For a single original trajectory, the default two baseline and three candidate repetitions form a strict sparse-data verification path. A stable native task/candidate failure across all baseline repetitions is a conclusive negative control when the candidate succeeds in all three repetitions and evaluation supplies a deterministic signal. Infrastructure, blocked, not-run, mixed, and inconsistent baseline outcomes remain inconclusive. A multi-trajectory set is validated by its distinct members; repetition counts never replace independent member coverage.

Stage-Aware Budgets and Candidate Lifecycle

SelfEvolveConfig.total_run_token_budget is the authoritative token ceiling for a complete optimize run. per_attempt_replay_token_limit is a separate hard ceiling for one replay attempt; it does not divide or replace the total-run budget. The same run ledger can enforce max_run_cost_usd and max_run_wall_seconds when those ceilings are configured.

Before an expensive stage starts, the framework reserves its estimated token, cost, and wall usage. Completion debits observed usage and releases the reservation. Observed samples then replace cold-start assumptions with a conservative robust estimate for later units. An unknown token estimate is not zero: if a token ceiling is active, the budget gate denies the work until the backend proves zero usage or a positive cold estimate is configured. The per-unit cold-start knobs are:

  • candidate_generation_tokens_per_unit
  • candidate_screening_tokens_per_unit
  • replay_tokens_per_unit
  • evaluation_tokens_per_unit

Each has optional matching _cost_usd_per_unit and _wall_seconds_per_unit fields. Token cold estimates must be positive; cost and wall estimates may be zero but cannot be negative.

Budget units follow actual workload cardinality rather than the number of input files. Generation and local stages count per candidate attempt. Conformance counts distinct executable contract shapes, so equivalent cases share one unit while different response/probe shapes remain separate. Representative screening uses at most one case per candidate. Paired replay, evaluation, and judge usage scale with case_count * repetitions; therefore a three-trajectory dataset cannot be priced as a one-trajectory run.

Every candidate slot has an explicit funnel record: generated -> unique (or duplicate_filtered) -> local gates -> adaptation -> repair conformance -> representative screening -> paired replay started/completed/ comparable -> evaluation -> selected, rejected, blocked, or not_run. Terminal reasons and observed usage remain attached to the attempt, including work that was denied before replay.

verification_funnel.authoritative_candidate_attempt_count counts candidates that entered authoritative validation; authoritative_candidate_count is the quota consumed across Campaign cycles. Shared framework-owned measurement failures that occur before candidate observation increment only the attempt counter and release the quota reservation. Candidate execution or a candidate-owned authoritative conclusion consumes the slot. This distinction prevents a replay infrastructure deadline from falsely reporting campaign_authoritative_frontier_exhausted with zero affected candidates. Candidate-owned capability compilation or deterministic preflight failures occur before this boundary: they increment neither authoritative counter, cannot create a measurement checkpoint, and return their aggregated typed constraints to the candidate-repair frontier. A resumed measurement checkpoint is logged as measurement_resume and is not charged as a newly generated candidate.

The scheduler begins with bounded exploration. After a candidate-owned repairable failure establishes a semantic frontier, the next iteration reserves one focused repair slot for that frontier. It may add one diverse exploration slot only for a new frontier and only when a separate reservation is available. Repeated failures do not silently recreate a full population.

max_run_tokens remains readable for compatibility. When the new fields are omitted, both SelfEvolveConfig and direct SelfEvolveRunner construction map the legacy value to total_run_token_budget and per_attempt_replay_token_limit. Reports expose those decisions as max_run_tokens_to_total_run_token_budget and max_run_tokens_to_per_attempt_replay_token_limit in deprecated_config_mappings. New callers should set both fields directly; these mappings are deprecated compatibility paths, not additional independent budgets.

Trusted Improvement Measurement

SDK callers and background SelfEvolveConfig jobs can enable controlled measurement with measurement_mode="shadow", "advisory", or "required". The CLI exposes the same policy through --measurement-mode. When the option is omitted, verified skill replay (verified_only or auto_verified) uses shadow until calibration is accepted; proposal/non-replay runs use off. Select required explicitly only after reviewing calibration and transfer evidence. An explicit off remains the legacy compatibility switch. For example:

aworld-cli optimize \
  --from-trajectory trajectory.log \
  --measurement-mode shadow \
  --measurement-primary-metric task_success \
  --measurement-minimum-effect 0.02 \
  --measurement-confidence-level 0.95 \
  --measurement-min-independent-cases 4

--measurement-bootstrap-samples controls deterministic case-level bootstrap resampling and must be between 200 and 100000. Repetitions are aggregated within their case before resampling; they never increase the independent-case count. --measurement-zero-yield-patience, --measurement-invalid-control-patience, and --measurement-maximum-interval-width freeze the early-stop policy into the experiment identity before observations are collected.

Before authoritative rollout, the CLI reports planned/reused/pending work units, decision-required units, sampling stages, effective lane count, isolation fallback, minimum feasible wall time, P50/P90 decision ETA, estimate confidence, and stopping policy. The same bounded projection is retained under measurement_control.preflight in report.json.

The frozen evidence policy is compiled from the controlled observation plan, target adapter identity, replay adaptation/capability, evaluator identity, and resource limits. Capability-owned loopback services receive deterministic plan-bound endpoints; runtime endpoint drift fails closed. Independent pairs use at most two lanes only when adaptation produces two complete compatible grants. An explicit exclusive adapter remains one lane and the limiting resource is shown in preflight output.

Within an admitted stage, pair work uses the frozen stratified case order as a coverage constraint and then ranks otherwise eligible work by expected information value divided by predicted cost and failure risk. Completed pair duration and invalid-control outcomes update that bounded ranking; they never change evidence weight or expose hidden case content. First repetitions cover eligible cases before later repetitions, so slow or invalid cases can be demoted without being starved. Expansion remains limited by the frozen stage batch size. Each admission records its expected information value, selection policy, and remaining case budget in measurement_schedule.json.

Historical target-only control preflight is advisory: it may order candidate controls, but it cannot block candidate generation or qualify a newly compiled replay support package. After candidate capability compilation, control health is qualified under an identity that binds the case, baseline skill, capability package, frozen replay capability, adaptation, and timeout envelope. A baseline cache entry is reusable only with the same support and envelope fingerprints; behavior-only siblings can therefore share it when their replay support is identical. A smaller-envelope baseline timeout gets one bounded same-case escalation (up to 300 seconds) before another control is tried. If the baseline fails only with candidate-owned support while another qualified support surface succeeds, the failure is reported as candidate_replay_support_baseline_incompatible and returned to candidate repair rather than framework handoff.

The same trusted-measurement policy is enforced while authoritative replay is still running. Each baseline or candidate member has a hard phase deadline covering all evidence retries, and progress reports expose both the member deadline and the current retry. A treatment always follows its own compatible control; up to two independently isolated case pairs may overlap. Completed baselines are indexed incrementally and a phase checkpoint records completed pairs and pending cases, so an interrupted multi-case replay can reuse valid controls and continue from an auditable cursor. Scheduler messages never embed complete trajectories, stdout, stderr, or large metric payloads. Each completed member writes those artifacts normally and publishes a bounded measurement_result_projection.json containing only typed status/failure data and content-addressed artifact references. Projection or journal-finalization failure checkpoints the running work unit and preserves a bounded error artifact; it cannot leave a normal process alive with an unexplained running unit.

The first baseline in an admitted stage also qualifies the frozen control contract. Reaching an artifact quota does not prohibit provably read-only analysis of evidence already collected, but unknown or artifact-producing actions remain fail-closed. If the unchanged baseline is nevertheless rejected by the frozen evidence policy, the scheduler reports baseline_evidence_policy_infeasible as a shared framework blocker and stops before candidate execution. It does not label the candidate, synthesize a trusted-improvement result, or charge a measurement-invalid retry. In advisory and required modes, repeated invalid controls stop expansion after measurement-invalid-control-patience; remaining members are recorded as blocked rather than consuming more authoritative work. off and shadow retain their existing decision behavior. During representative screening, a framework-owned invalid control advances through every distinct case in the already-bounded representative panel. Each qualification case is itself frozen as a v2 repair_screening measurement plan before rollout. The candidate package keeps its canonical candidate ID; screening/<case-id> is an execution namespace only and cannot become a new candidate identity or authoritative observation. This prevents screening and control fallbacks from losing the predeclared experiment while keeping their artifacts isolated from the later full-panel replay. Invalid controls are remembered across Campaign cycles for the same frozen dataset and target baseline, so a later cycle suppresses the repeated control and selects a fresh case. Reports expose invalid_control_case_ids and control_case_retry_suppressed_count. Candidate-owned failures do not use this fallback and return directly to focused repair. The bounded screening horizon is not an invalid control: if both variants reach it, or the baseline reaches its hard deadline before candidate execution, the comparison is right-censored and one ranked candidate advances to authoritative replay without exhausting fallback cases. A blocked or not-run candidate emits no candidate behavior attribution. For a verified dataset with at least the configured minimum number of independent cases, the default replay policy uses one baseline and one candidate observation per case; independent cases provide the measurement sample, so duplicating every case two or three times is not the default source of confidence. Sparse datasets retain the configured repetition defaults, and explicit replay-repetition options always override the adaptive default. Baseline members are reused across compatible candidates and retries. If a candidate member makes the authoritative comparison impossible, remaining candidate members stop immediately; an invalid control is classified as framework-owned, while a candidate-owned incomparable member is returned to the repair frontier. Representative screening remains the bounded admission plane, while paired replay remains the authoritative full-panel experiment.

If the framework cannot freeze or admit that screening plan, it stops the population before physical replay and reports a shared measurement_plan_admission_failed prerequisite. The source candidate is kept as the immutable measurement checkpoint; sibling candidates, authoritative quota, and candidate-repair feedback are not consumed. In this state the bounded measurement projection is not_started / prerequisite_blocked, not control_not_comparable, because no control or treatment observation exists.

For each measured candidate, AWorld freezes the task model/executor, generator, scheduler, evaluator, dataset, environment, runtime, prompt context, and budget identities while changing only the candidate artifact. The full experiment lives under .aworld/self_evolve/<run_id>/experiments/<experiment_id>/; report.json contains only the bounded measurement summary. Use the referenced attribution_report.json for paired confidence intervals, comparable-pair counts, best/pass-at-K values, separate actual search/measurement ledgers, and token/wall curves. Missing values mean “not measured,” never numeric zero. Missing controls or incomplete usage remain explicitly invalid/inconclusive; the runner does not replace them with the best candidate score.

The declared swap axis is the attribution boundary: artifact swaps measure harness gain, task-model swaps measure model gain, and generator/scheduler swaps require equal opportunity and compare distributions plus best/pass-at-K curves. A larger N is reported as search opportunity, not as component improvement.

Output

The command prints the stable artifact paths returned by the framework:

Optimize run submitted.
Status: succeeded
Campaign: campaign-...
Campaign status: complete
Campaign cycle: 2/3
Self-improvement: complete (verified_run_succeeded)
Report: .aworld/self_evolve/<run_id>/report.json
Target selection: .aworld/self_evolve/<run_id>/target_selection.json
Replay: .aworld/self_evolve/<run_id>/replay.json
Evaluator report: .aworld/self_evolve/<run_id>/evaluator_report.json
Best candidate: cand-1

For rejected runs, the summary includes rejected gate names. If no candidate was produced, replay/evaluation/apply are skipped. If judge evaluation timed out after replay completed, the summary prints a resume command:

Resume evaluator: aworld-cli optimize --from-run <run_id> --rerun-evaluator

For Campaign runs, Status is the latest ordinary run result and Campaign status is the authoritative cross-run outcome. active means another bounded generation is eligible, paused requires an operator or Goal handoff, budget_limited means a cycle/resource/telemetry ceiling stopped the Campaign, exhausted means its typed repair frontier stopped progressing, and complete means the latest run passed the unchanged verified-apply gates. All three terminal states are non-resumable. Worker execution success is reported separately and never converts a rejected framework result into success.

If replay repetitions are missing, rerun full optimize with a larger --replay-timeout; evaluator-only resume cannot add new replay rollouts.

For an auto_verified release, treat the run as successful only when report.json has status: "succeeded", no blocking failed gate, a selected candidate, and post_apply.status: "accepted" with release_state: "verified". status: "rejected", post_apply.status: "rolled_back", or a missing post-apply record is not a successful verified release even if candidate generation or source conformance passed.

Report Files

Open .aworld/self_evolve/<run_id>/report.json for the release-facing result:

  • status: succeeded, rejected, or failed.
  • apply_policy: proposal or auto_verified.
  • selected_candidate_id: selected candidate when one exists.
  • gate_results: low-level gate decisions.
  • target_provenance: resolved sidecar path and reason, or an explicit unresolved status.
  • release_checklist: grouped release checks derived from gates.
  • content_quality_diagnostics: non-blocking publication/content quality diagnostics when evaluator metrics provide them.
  • population: generated candidates, representative screening attempts, selection reason, focused repair telemetry, concurrency, and candidate-generation token usage.
  • optimizer_lineage: candidate lineage artifact links, including content, semantic, and lesson-set fingerprints when available.
  • lessons: normalized lesson artifact links and counts.
  • harness_diagnostics: advisory framework diagnostic artifact links and counts.
  • release_normalization: normalized release fingerprint, pre-normalization fingerprint, preserved runtime constraints, and verification status.
  • replay_path: candidate replay artifact path.
  • replay.adaptation: adaptation readiness plus workspace, environment, task, dataset, and current-skill fingerprints for completed paired replay.
  • evaluator_report_paths: evaluator output artifacts.
  • post_apply: accepted or rolled-back apply details, backup path, journal path, runtime activation, and registry refresh status.
  • artifact_retention: merged startup/terminal cleanup result, including policy, protected runs, skipped runs, and removed paths.

Candidate files and diffs live under .aworld/self_evolve/<run_id>/candidates/. For skill candidates, proposal content is marked with self_evolve.release_state: candidate; verified content is marked with self_evolve.release_state: verified only after post-apply checks pass. <candidate_id>.json is the durable normalized record; <candidate_id>/ is the expanded multi-file materialization. After a run leaves the complete-artifact retention window, redundant Markdown, diff, and expanded package copies for unselected candidates may be reclaimed while selected, applied, and lineage-parent candidates remain intact.

Repair preflight artifacts live under .aworld/self_evolve/<run_id>/repair_conformance/<candidate_id>/. They contain bounded frozen-capability and service/probe diagnostics for failures that occurred before task rollout.

Replay adaptation artifacts live under .aworld/self_evolve/<run_id>/replay_adaptation/<dataset_fingerprint>/. The directory contains bundle.json, workspace_manifest.json, environment_snapshot.json, and workspace_seed/. Each executed repetition stores its disposable workspace under the corresponding replay repetition directory.

Replay lifecycle records are cardinality-neutral: one task produces one member and a trajectory set produces one member per replayable task, always in dataset order. New artifacts use members/manifest.json v2 and a lifecycle.json v2 for every baseline and candidate variant, including work that did not execute:

  • succeeded: execution started and completed with usable trajectory evidence.
  • failed: execution started and produced a typed failure event.
  • blocked: execution did not start because the event in blocked_by prevented it.
  • not_run: the framework intentionally did not schedule the variant.

A failure event records owner (candidate, task, infrastructure, or framework), stage, scope, a stable code, and repairability independently from execution status. For example, if a candidate capability fails preflight on the first member of a three-member dataset, that baseline member is failed with owner: candidate, stage: capability_preflight, and scope: candidate; all three candidate variants and the remaining baseline variants are blocked by the same event. The next candidate in the population is still evaluated. Only an explicit native shared_run event owned by infrastructure or the framework stops the population. Legacy status and failure files remain readable, but unknown legacy failures are not promoted to run-wide infrastructure failures.

Member manifests fail closed when members are missing, duplicated, unexpected, empty, or carry a request that changes root-level run, candidate, target, overlay, agent, or provenance fields. Task identity/input, task-input fingerprint, member baseline path, and repetition counts must exactly match the values deterministically derived for that dataset member; every other request field must equal the root request. Numeric repetition child directories are reconstructed when lifecycle v2 artifacts are loaded, so paired evaluation has the same repetition cardinality before and after persistence.

Artifact GC runs at optimize startup and terminal completion. It preserves the two newest runs, lineage and apply recovery dependencies, and runs with a live process lease. Raw replay, replay-adaptation, repair-conformance, evaluator, overlay, and temporary-workspace artifacts from older eligible runs are reclaimed.

The default store is <workspace>/.aworld/self_evolve. The framework does not create or manage a top-level self_evolve_artifacts/ directory.

Runtime Skill Management

Verified skill candidates are made visible to runtime discovery only after the framework marks them verified and the CLI activation hook enables the skill. Draft and candidate skills remain hidden.

Useful commands:

aworld-cli skill list
aworld-cli skill disable <skill-name>
aworld-cli skill enable <skill-name>
aworld-cli skill remove <skill-name>

skill remove can remove a runtime skill directory when it is discovered from a local runtime skill source and is not a symlink or package install. If an installed package with the same id exists, package removal takes precedence.

Boundaries

The CLI does not score targets itself. --task without --target delegates target selection to framework credit assignment after the framework filters the inventory through the CLI adapter registry. This prevents an unsupported target type from winning inference. An explicitly requested unsupported target is rejected; adding a general tested adapter is the extension path, not adding a trajectory-specific exception.

The CLI also does not own scheduler behavior, evaluator behavior, optimizer semantics, durable job formats, or agent opt-in configuration. Configure opt-in with AgentConfig.self_evolve_config, then use this command as a manual/debug entrypoint for the same framework APIs.

See Self Evolve for the framework design, safety model, SDK example, and release checklist details.