Add browse tool support.
I reviewed some MCPs (using OpenAI's deep research to help), and it
helped me choose chromedp as the relevant library and helped me come up
with an interface. This commit adds chrome to the Docker image which is
kind of big. (I've noticed that it's smaller on Ubuntu, where it doesn't
pull in X11.) go-playwright was a library contender as well.
Implement browser automation tooling using chromedp
This implementation adds browser automation capabilities to the system via the chromedp library,
enabling Claude to interact with web content effectively.
Key features include:
1. Core browser automation functionality:
- Created new browsertools package in claudetool/browser
- Implemented tools for navigating, clicking, typing, waiting for elements,
getting text, evaluating JavaScript, taking screenshots, and scrolling
- Added lazy browser initialization that defers until first use
- Integrated with the agent to expose these tools to Claude
2. Screenshot handling and display:
- Implemented screenshot storage with UUID-based IDs in /tmp/sketch-screenshots
- Added endpoint to serve screenshots via /screenshot/{id}
- Created dedicated UI component for displaying screenshots
- Ensured proper responsive design with loading states and error handling
- Fixed URL paths for proper rehomed URL support
- Modified tool calls component to auto-expand screenshot results
3. Error handling and reliability:
- Added graceful error handling for browser initialization failures
- Implemented proper cleanup of browser resources
The browser automation tools provide a powerful way for Claude to interact with web content,
making it possible to scrape data, test web applications, and automate web-based tasks.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/dockerimg/testdata/testcreatedockerfile_empty_repo.httprr b/dockerimg/testdata/testcreatedockerfile_empty_repo.httprr
index 6f5ec2d..5a4643f 100644
--- a/dockerimg/testdata/testcreatedockerfile_empty_repo.httprr
+++ b/dockerimg/testdata/testcreatedockerfile_empty_repo.httprr
@@ -1,9 +1,9 @@
httprr trace v1
-3777 2383
+3767 1928
POST https://api.anthropic.com/v1/messages HTTP/1.1
Host: api.anthropic.com
User-Agent: Go-http-client/1.1
-Content-Length: 3580
+Content-Length: 3570
Anthropic-Version: 2023-06-01
Content-Type: application/json
@@ -15,7 +15,7 @@
"content": [
{
"type": "text",
- "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/*\n\n# Strip out docs from debian.\nRUN rm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\n"
+ "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim chromium \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/* \u0026\u0026 \\\n\trm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\n"
},
{
"type": "text",
@@ -50,24 +50,24 @@
Anthropic-Organization-Id: 3c473a21-7208-450a-a9f8-80aebda45c1b
Anthropic-Ratelimit-Input-Tokens-Limit: 200000
Anthropic-Ratelimit-Input-Tokens-Remaining: 200000
-Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-05T19:55:42Z
+Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-06T03:31:08Z
Anthropic-Ratelimit-Output-Tokens-Limit: 80000
Anthropic-Ratelimit-Output-Tokens-Remaining: 80000
-Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-05T19:55:46Z
+Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-06T03:31:10Z
Anthropic-Ratelimit-Requests-Limit: 4000
Anthropic-Ratelimit-Requests-Remaining: 3999
-Anthropic-Ratelimit-Requests-Reset: 2025-05-05T19:55:40Z
+Anthropic-Ratelimit-Requests-Reset: 2025-05-06T03:31:07Z
Anthropic-Ratelimit-Tokens-Limit: 280000
Anthropic-Ratelimit-Tokens-Remaining: 280000
-Anthropic-Ratelimit-Tokens-Reset: 2025-05-05T19:55:42Z
+Anthropic-Ratelimit-Tokens-Reset: 2025-05-06T03:31:08Z
Cf-Cache-Status: DYNAMIC
-Cf-Ray: 93b2cbf9bcab15f7-SJC
+Cf-Ray: 93b56723daa0ce9c-SJC
Content-Type: application/json
-Date: Mon, 05 May 2025 19:55:46 GMT
-Request-Id: req_011CNpyHmZv42gfUEp4BiYvZ
+Date: Tue, 06 May 2025 03:31:11 GMT
+Request-Id: req_011CNqa2AVC9m56dtxX5MqbJ
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Via: 1.1 google
X-Robots-Tag: none
-{"id":"msg_01P8BMBGo1uG2pRH27YTQKRw","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"I'll help you create a Dockerfile for this Go project. Based on the template you provided, I need to generate appropriate extra commands for the Dockerfile that will set up a development environment.\n\nSince you mentioned this is primarily a Go project, I'll focus on creating a minimal set of extra commands that might be helpful for a Go development environment. I'll ensure Python commands have error handling, and apt-get install commands have the \"|| true\" suffix as requested."},{"type":"tool_use","id":"toolu_015rV7PbsVvcoTHGnkcYtC78","name":"dockerfile","input":{"extra_cmds":"# Install any Go tools that might be useful for development\ngo install github.com/golangci/golangci-lint/cmd/golangci-lint@latest || true\ngo install github.com/rakyll/gotest@latest || true\n\n# Install Python dependencies if needed (with error handling)\nif [ -f requirements.txt ]; then\n pip3 install -r requirements.txt || true\nfi\n\n# If Makefile exists, run make prepare or similar setup target\nif [ -f Makefile ]; then\n grep -q \"prepare:\" Makefile && make prepare || true\nfi"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1282,"cache_read_input_tokens":0,"output_tokens":289}}
\ No newline at end of file
+{"id":"msg_01UEUtJaKVQygbeUjy9apyfW","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"I'll call the dockerfile tool to create a Dockerfile for your Go project. Based on the template provided, I'll add minimal but useful extra commands."},{"type":"tool_use","id":"toolu_01GSRQTjFTpsavcRqaZCKaRP","name":"dockerfile","input":{"extra_cmds":"# Install any Go tools specific to development\nRUN go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest || true\n\n# Install any Python dependencies if needed (allow failures)\nRUN if [ -f requirements.txt ]; then pip3 install -r requirements.txt || true; fi\n\n# Pre-compile the project if possible\nRUN if [ -f go.mod ]; then go build -v ./... || true; fi"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1277,"cache_read_input_tokens":0,"output_tokens":195}}
\ No newline at end of file