Add browse tool support.
I reviewed some MCPs (using OpenAI's deep research to help), and it
helped me choose chromedp as the relevant library and helped me come up
with an interface. This commit adds chrome to the Docker image which is
kind of big. (I've noticed that it's smaller on Ubuntu, where it doesn't
pull in X11.) go-playwright was a library contender as well.
Implement browser automation tooling using chromedp
This implementation adds browser automation capabilities to the system via the chromedp library,
enabling Claude to interact with web content effectively.
Key features include:
1. Core browser automation functionality:
- Created new browsertools package in claudetool/browser
- Implemented tools for navigating, clicking, typing, waiting for elements,
getting text, evaluating JavaScript, taking screenshots, and scrolling
- Added lazy browser initialization that defers until first use
- Integrated with the agent to expose these tools to Claude
2. Screenshot handling and display:
- Implemented screenshot storage with UUID-based IDs in /tmp/sketch-screenshots
- Added endpoint to serve screenshots via /screenshot/{id}
- Created dedicated UI component for displaying screenshots
- Ensured proper responsive design with loading states and error handling
- Fixed URL paths for proper rehomed URL support
- Modified tool calls component to auto-expand screenshot results
3. Error handling and reliability:
- Added graceful error handling for browser initialization failures
- Implemented proper cleanup of browser resources
The browser automation tools provide a powerful way for Claude to interact with web content,
making it possible to scrape data, test web applications, and automate web-based tasks.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/dockerimg/testdata/testcreatedockerfile_basic_repo_with_readme.httprr b/dockerimg/testdata/testcreatedockerfile_basic_repo_with_readme.httprr
index 23adfaa..e9f4989 100644
--- a/dockerimg/testdata/testcreatedockerfile_basic_repo_with_readme.httprr
+++ b/dockerimg/testdata/testcreatedockerfile_basic_repo_with_readme.httprr
@@ -1,9 +1,9 @@
httprr trace v1
-4024 1901
+4014 1797
POST https://api.anthropic.com/v1/messages HTTP/1.1
Host: api.anthropic.com
User-Agent: Go-http-client/1.1
-Content-Length: 3827
+Content-Length: 3817
Anthropic-Version: 2023-06-01
Content-Type: application/json
@@ -15,7 +15,7 @@
"content": [
{
"type": "text",
- "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/*\n\n# Strip out docs from debian.\nRUN rm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\nHere is the content of several files from the repository that may be relevant:\n\n"
+ "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim chromium \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/* \u0026\u0026 \\\n\trm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\nHere is the content of several files from the repository that may be relevant:\n\n"
},
{
"type": "text",
@@ -54,24 +54,24 @@
Anthropic-Organization-Id: 3c473a21-7208-450a-a9f8-80aebda45c1b
Anthropic-Ratelimit-Input-Tokens-Limit: 200000
Anthropic-Ratelimit-Input-Tokens-Remaining: 200000
-Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-05T19:55:29Z
+Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-06T03:30:56Z
Anthropic-Ratelimit-Output-Tokens-Limit: 80000
Anthropic-Ratelimit-Output-Tokens-Remaining: 80000
-Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-05T19:55:32Z
+Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-06T03:30:59Z
Anthropic-Ratelimit-Requests-Limit: 4000
Anthropic-Ratelimit-Requests-Remaining: 3999
-Anthropic-Ratelimit-Requests-Reset: 2025-05-05T19:55:28Z
+Anthropic-Ratelimit-Requests-Reset: 2025-05-06T03:30:55Z
Anthropic-Ratelimit-Tokens-Limit: 280000
Anthropic-Ratelimit-Tokens-Remaining: 280000
-Anthropic-Ratelimit-Tokens-Reset: 2025-05-05T19:55:29Z
+Anthropic-Ratelimit-Tokens-Reset: 2025-05-06T03:30:56Z
Cf-Cache-Status: DYNAMIC
-Cf-Ray: 93b2cbb08ab515f7-SJC
+Cf-Ray: 93b566d6690ace9c-SJC
Content-Type: application/json
-Date: Mon, 05 May 2025 19:55:32 GMT
-Request-Id: req_011CNpyGujGv6eewDja4g5Ee
+Date: Tue, 06 May 2025 03:30:59 GMT
+Request-Id: req_011CNqa1FVnavsQuEVK4FSjx
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Via: 1.1 google
X-Robots-Tag: none
-{"id":"msg_01FRKiayPrFbD6HCNFk6RhDS","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"Based on the provided information, I'll help you create a Dockerfile for this Go project using the dockerfile tool.\n\nSince this is a simple test Go project without any specific requirements beyond what's already in the template, I don't need to add many extra commands. The template already includes the essential Go development tools and environment setup."},{"type":"tool_use","id":"toolu_018ADQAYzXuVBBSvfwB3WKV6","name":"dockerfile","input":{"extra_cmds":"# No additional commands needed for this simple Go test project\n# The base Dockerfile already includes all necessary Go development tools"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1326,"cache_read_input_tokens":0,"output_tokens":147}}
\ No newline at end of file
+{"id":"msg_01XEdKoiC5EwZX9pS6uvifeA","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"Based on the provided information, you have a simple Go test project. Since the project seems very basic according to the README, and there's no indication of any Python dependencies or specialized tools needed, I'll create a minimal Dockerfile with just the essential setup."},{"type":"tool_use","id":"toolu_01MiuPNSrwSBCv6Q4s5dyg8n","name":"dockerfile","input":{"extra_cmds":"# No special commands needed for this simple Go test project\n# The base image already provides all necessary Go tools"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1321,"cache_read_input_tokens":0,"output_tokens":130}}
\ No newline at end of file