skill-creator

1# Skill Creator
2 
3A skill for creating new skills and iteratively improving them.
4 
5At a high level, the process of creating a skill goes like this:
6 
7- Decide what you want the skill to do and roughly how it should do it
8- Write a draft of the skill
9- Create a few test prompts and run claude-with-access-to-the-skill on them
10- Help the user evaluate the results both qualitatively and quantitatively
11  - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)
12  - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics
13- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)
14- Repeat until you're satisfied
15- Expand the test set and try again at larger scale
16 
17Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.
18…