claudetool/onstart: add codebase analysis tool and basic memory support
This is a preliminary approach. Big high level questions include:
* should we keep the multichoice tool prompting?
* should we push the list of quidance files or respond with them during codereview?
* should we use the list of docs and build files at all?
* are there other files we should hint (e.g. editor settings, something from aider, etc.)?
We should probably also blog about dear_llm.md to stop the endless proliferation of new files.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/loop/agent.go b/loop/agent.go
index b3fd6c4..c103919 100644
--- a/loop/agent.go
+++ b/loop/agent.go
@@ -23,6 +23,7 @@
"sketch.dev/claudetool/bashkit"
"sketch.dev/claudetool/browse"
"sketch.dev/claudetool/codereview"
+ "sketch.dev/claudetool/onstart"
"sketch.dev/experiment"
"sketch.dev/llm"
"sketch.dev/llm/conversation"
@@ -296,6 +297,7 @@
gitRemoteAddr string // HTTP URL of the host git repo (only when under docker)
outsideHTTP string // base address of the outside webserver (only when under docker)
ready chan struct{} // closed when the agent is initialized (only when under docker)
+ codebase *onstart.Codebase
startedAt time.Time
originalBudget conversation.Budget
title string
@@ -799,6 +801,15 @@
}
a.initialCommit = commitHash
+ if experiment.Enabled("memory") {
+ slog.Info("running codebase analysis")
+ codebase, err := onstart.AnalyzeCodebase(ctx, a.repoRoot)
+ if err != nil {
+ slog.Warn("failed to analyze codebase", "error", err)
+ }
+ a.codebase = codebase
+ }
+
llmCodeReview := codereview.NoLLMReview
if experiment.Enabled("llm_review") {
llmCodeReview = codereview.DoLLMReview
@@ -1808,6 +1819,7 @@
WorkingDir string
RepoRoot string
InitialCommit string
+ Codebase *onstart.Codebase
}
// renderSystemPrompt renders the system prompt template.
@@ -1827,6 +1839,7 @@
WorkingDir: a.workingDir,
RepoRoot: a.repoRoot,
InitialCommit: a.initialCommit,
+ Codebase: a.codebase,
}
tmpl, err := template.New("system").Parse(agentSystemPrompt)
@@ -1838,5 +1851,6 @@
if err != nil {
panic(fmt.Sprintf("failed to execute system prompt template: %v", err))
}
+ // fmt.Printf("system prompt: %s\n", buf.String())
return buf.String()
}
diff --git a/loop/agent_system_prompt.txt b/loop/agent_system_prompt.txt
index eae563d..a2a794e 100644
--- a/loop/agent_system_prompt.txt
+++ b/loop/agent_system_prompt.txt
@@ -1,6 +1,7 @@
You are an expert coding assistant and architect.
You are assisting the user to achieve their goals.
+<workflow>
Start by asking concise clarifying questions as needed.
Once the intent is clear, work autonomously.
Aim for a small diff size while thoroughly completing the requested task.
@@ -27,16 +28,87 @@
review before declaring that you are done. Before executing
the done tool, run all the tools the done tool checklist asks
for, including creating a git commit. Do not forget to run tests.
+</workflow>
+{{ with .Codebase }}
+<memory>
+Guidance files (dear_llm.md, cursorrules, claude.md) contain user preferences and project-specific instructions.
+All root-level guidance files have been included in this system prompt in the <guidance> section and should be respected for all work.
+Additionally, directory-specific guidance files are listed in the <other_guidance_files> section.
+When working in a specific directory, always read and follow the guidance files associated with that directory and its parent directories.
+
+When the user provides particularly valuable general-purpose feedback, guidance, or preferences, you may (sparingly) use the multiplechoice tool to ask if they want to record this in a dear_llm.md file for future use.
+The options should include:
+
+1. "Yes, for all future work" - For highly important, generally applicable information, recorded in the root dear_llm.md
+2. "Yes, but only for directory X" - Where X is the relevant directory where this information applies
+3. "No" - Don't record this feedback
+
+When presenting this choice, include a preview of exactly what would be written to the dear_llm.md file.
+For example: "I would record: 'Prefer table-driven tests over multiple separate test functions.'"
+Changes to dear_llm.md files should always be in a separate atomic commit, with no other modified files.
+</memory>
+
+<guidance>
+{{ $contents := .InjectFileContents }}
+{{- range .InjectFiles }}
+<root_guidance file="{{ . }}">
+{{ index $contents . }}
+</root_guidance>
+{{ end -}}
+</guidance>
+{{ end -}}
+
+{{ with .Codebase }}
+{{- if .GuidanceFiles }}
+<other_guidance_files>
+{{- range .GuidanceFiles }}
+{{ . -}}
+{{ end }}
+</other_guidance_files>
+{{ end }}
+{{ end -}}
+
+<system_info>
<platform>
{{.ClientGOOS}}/{{.ClientGOARCH}}
</platform>
<pwd>
{{.WorkingDir}}
</pwd>
+</system_info>
+
+<git_info>
<git_root>
{{.RepoRoot}}
</git_root>
<HEAD>
{{.InitialCommit}}
</HEAD>
+</git_info>
+
+{{ with .Codebase -}}
+<codebase_info>
+{{ if .TopExtensions }}
+<top_file_extensions>
+{{- range .TopExtensions }}
+{{ . -}}
+{{ end }}
+</top_file_extensions>
+{{- end -}}
+{{- if .BuildFiles }}
+<build_files>
+{{- range .BuildFiles }}
+{{ . -}}
+{{ end }}
+</build_files>
+{{ end -}}
+{{- if .DocumentationFiles }}
+<documentation_files>
+{{- range .DocumentationFiles }}
+{{ . -}}
+{{ end }}
+</documentation_files>
+{{ end -}}
+</codebase_info>
+{{ end -}}
diff --git a/loop/testdata/agent_loop.httprr b/loop/testdata/agent_loop.httprr
index 37bf107..64137c0 100644
--- a/loop/testdata/agent_loop.httprr
+++ b/loop/testdata/agent_loop.httprr
@@ -1,9 +1,9 @@
httprr trace v1
-14130 2178
+14275 2118
POST https://api.anthropic.com/v1/messages HTTP/1.1
Host: api.anthropic.com
User-Agent: Go-http-client/1.1
-Content-Length: 13932
+Content-Length: 14077
Anthropic-Version: 2023-06-01
Content-Type: application/json
@@ -437,7 +437,7 @@
],
"system": [
{
- "text": "You are an expert coding assistant and architect.\nYou are assisting the user to achieve their goals.\n\nStart by asking concise clarifying questions as needed.\nOnce the intent is clear, work autonomously.\nAim for a small diff size while thoroughly completing the requested task.\n\nCall the title tool as soon as the topic of conversation is clear, often immediately.\n\nBreak down the overall goal into a series of smaller steps.\n(The first step is often: \"Make a plan.\")\nThen execute each step using tools.\nUpdate the plan if you have encountered problems or learned new information.\n\nWhen in doubt about a step, follow this broad workflow:\n\n- Think about how the current step fits into the overall plan.\n- Do research. Good tool choices: bash, think, keyword_search\n- Make edits.\n- Repeat.\n\nTo make edits reliably and efficiently, first think about the intent of the edit,\nand what set of patches will achieve that intent.\nThen use the patch tool to make those edits. Combine all edits to any given file into a single patch tool call.\n\nThe done tool provides a checklist of items you MUST verify and\nreview before declaring that you are done. Before executing\nthe done tool, run all the tools the done tool checklist asks\nfor, including creating a git commit. Do not forget to run tests.\n\n\u003cplatform\u003e\nlinux/amd64\n\u003c/platform\u003e\n\u003cpwd\u003e\n/\n\u003c/pwd\u003e\n\u003cgit_root\u003e\n\n\u003c/git_root\u003e\n\u003cHEAD\u003e\n\n\u003c/HEAD\u003e\n",
+ "text": "You are an expert coding assistant and architect.\nYou are assisting the user to achieve their goals.\n\n\u003cworkflow\u003e\nStart by asking concise clarifying questions as needed.\nOnce the intent is clear, work autonomously.\nAim for a small diff size while thoroughly completing the requested task.\n\nCall the title tool as soon as the topic of conversation is clear, often immediately.\n\nBreak down the overall goal into a series of smaller steps.\n(The first step is often: \"Make a plan.\")\nThen execute each step using tools.\nUpdate the plan if you have encountered problems or learned new information.\n\nWhen in doubt about a step, follow this broad workflow:\n\n- Think about how the current step fits into the overall plan.\n- Do research. Good tool choices: bash, think, keyword_search\n- Make edits.\n- Repeat.\n\nTo make edits reliably and efficiently, first think about the intent of the edit,\nand what set of patches will achieve that intent.\nThen use the patch tool to make those edits. Combine all edits to any given file into a single patch tool call.\n\nThe done tool provides a checklist of items you MUST verify and\nreview before declaring that you are done. Before executing\nthe done tool, run all the tools the done tool checklist asks\nfor, including creating a git commit. Do not forget to run tests.\n\u003c/workflow\u003e\n\n\u003csystem_info\u003e\n\u003cplatform\u003e\nlinux/amd64\n\u003c/platform\u003e\n\u003cpwd\u003e\n/\n\u003c/pwd\u003e\n\u003c/system_info\u003e\n\n\u003cgit_info\u003e\n\u003cgit_root\u003e\n\n\u003c/git_root\u003e\n\u003cHEAD\u003e\n\n\u003c/HEAD\u003e\n\u003c/git_info\u003e\n\n",
"type": "text",
"cache_control": {
"type": "ephemeral"
@@ -447,25 +447,25 @@
}HTTP/2.0 200 OK
Anthropic-Organization-Id: 3c473a21-7208-450a-a9f8-80aebda45c1b
Anthropic-Ratelimit-Input-Tokens-Limit: 200000
-Anthropic-Ratelimit-Input-Tokens-Remaining: 199000
-Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-09T22:32:01Z
+Anthropic-Ratelimit-Input-Tokens-Remaining: 200000
+Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-09T22:32:28Z
Anthropic-Ratelimit-Output-Tokens-Limit: 80000
Anthropic-Ratelimit-Output-Tokens-Remaining: 80000
-Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-09T22:32:04Z
+Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-09T22:32:31Z
Anthropic-Ratelimit-Requests-Limit: 4000
Anthropic-Ratelimit-Requests-Remaining: 3999
-Anthropic-Ratelimit-Requests-Reset: 2025-05-09T22:32:00Z
+Anthropic-Ratelimit-Requests-Reset: 2025-05-09T22:32:27Z
Anthropic-Ratelimit-Tokens-Limit: 280000
-Anthropic-Ratelimit-Tokens-Remaining: 279000
-Anthropic-Ratelimit-Tokens-Reset: 2025-05-09T22:32:01Z
+Anthropic-Ratelimit-Tokens-Remaining: 280000
+Anthropic-Ratelimit-Tokens-Reset: 2025-05-09T22:32:28Z
Cf-Cache-Status: DYNAMIC
-Cf-Ray: 93d4a6795e647542-SJC
+Cf-Ray: 93d4a720b88fb976-SJC
Content-Type: application/json
-Date: Fri, 09 May 2025 22:32:04 GMT
-Request-Id: req_011CNxkSrg1ab333g8XshWpb
+Date: Fri, 09 May 2025 22:32:31 GMT
+Request-Id: req_011CNxkUqzPtAV1N4dwqmCn8
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Via: 1.1 google
X-Robots-Tag: none
-{"id":"msg_019S4JeWNDZaM2mmDG8xEfFz","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"I'll provide a brief list of the tools available to me:\n\n1. bash - Execute shell commands\n2. keyword_search - Find files with search terms in a codebase\n3. think - Record thoughts, notes, or plans\n4. title - Set conversation title\n5. precommit - Create git branch for tracking work\n6. done - Mark task as complete with checklist\n7. codereview - Run automated code review\n8. multiplechoice - Present multiple choice options to the user\n9. browser_* tools - Several browser manipulation tools including:\n - browser_navigate\n - browser_click\n - browser_type\n - browser_wait_for\n - browser_get_text\n - browser_eval\n - browser_screenshot\n - browser_scroll_into_view\n10. patch - Make precise text edits to files\n\nThese tools allow me to execute commands, search files, take notes, manage git, review code, interact with browsers, and modify files."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":3262,"cache_read_input_tokens":0,"output_tokens":223}}
\ No newline at end of file
+{"id":"msg_01D6Uo6fKbA6VEwcrR1EJNDx","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"Here are the tools available to me:\n\n1. bash - Execute shell commands\n2. keyword_search - Find files with search terms\n3. think - Record thoughts or plans\n4. title - Set conversation title\n5. precommit - Create git branch for tracking work\n6. done - Mark task as complete with checklist\n7. codereview - Run automated code review\n8. multiplechoice - Present multiple-choice options\n9. browser_navigate - Navigate to URL\n10. browser_click - Click element with CSS selector\n11. browser_type - Type text into input element\n12. browser_wait_for - Wait for element to appear\n13. browser_get_text - Get text from element\n14. browser_eval - Run JavaScript in browser\n15. browser_screenshot - Take screenshot\n16. browser_scroll_into_view - Scroll element into view\n17. patch - Make precise text edits to files"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":3294,"cache_read_input_tokens":0,"output_tokens":206}}
\ No newline at end of file