Change one decision at a time
A description revision should start with a specific observation. Perhaps the skill activates for general Docker questions, or misses requests phrased as a configuration audit. Those are different failures and may call for different wording.
Keep the task itself stable while changing the description. If you rewrite the instructions, expand the scope, and replace the evaluation cases in the same iteration, it becomes difficult to tell what caused a different result.
Keep a baseline you can return to
Save the current description and evaluation set before beginning. Record the intended task boundary and keep the earlier results. A baseline gives a new candidate something concrete to improve on; it also makes an unhelpful revision easy to discard.
The plugin offers skill_improve_description to propose a revision and skill_optimize_loop for an evaluation-and-improvement workflow. Ask the agent to explain the observed failure pattern rather than accepting a longer description as an improvement in itself.
Use skill_optimize_loop to improve this description.
Keep the intended task unchanged. Use the false positives and false negatives to guide revisions. Preserve a held-out set and compare candidates against the current description. Explain the trade-offs before choosing one.Judge a candidate on held-out cases
Training cases expose problems and guide edits. Held-out cases test whether that edit generalizes. The project’s optimization workflow evaluates candidate descriptions against both sets and uses test performance when selecting a description.
That separation reduces one kind of overfitting, but it does not make a small test set representative of every real request. When an unexpected production request reveals a gap, add a well-labeled regression case and document why the set changed.
Look at the trade-off, not only the total
A candidate can fix missed activations while introducing unrelated ones. Read which cases changed. Decide whether the new boundary matches the task before choosing the revision. A higher aggregate result can still hide a regression you care about.
If the observed difference is small, consider model variability and repeated runs. Keep the evaluation conditions comparable. Do not treat a single favorable run as a reliable general claim.
Review paired outputs
Once a description looks promising, inspect what happens after the skill is selected. Prepare paired outputs with with_skill and either without_skill or old_skill for each evaluation directory.
The local review viewer, started through skill_serve_review, supports human comparison. The default paired-data guard makes a missing baseline visible instead of letting an incomplete comparison look complete. Use skill_export_static_review when a standalone HTML review is more useful.
Finish with a reasoned choice
A useful revision record can be short: the original failure, the changed wording, the evaluation cases, the before-and-after result, and what you saw in the paired review. Include limitations and remaining misses.
Sometimes the right choice is the existing description. An improvement loop is useful because it gives you a way to make that choice with evidence, rather than because every iteration must produce a new version.