Add browse tool support.
I reviewed some MCPs (using OpenAI's deep research to help), and it
helped me choose chromedp as the relevant library and helped me come up
with an interface. This commit adds chrome to the Docker image which is
kind of big. (I've noticed that it's smaller on Ubuntu, where it doesn't
pull in X11.) go-playwright was a library contender as well.
Implement browser automation tooling using chromedp
This implementation adds browser automation capabilities to the system via the chromedp library,
enabling Claude to interact with web content effectively.
Key features include:
1. Core browser automation functionality:
- Created new browsertools package in claudetool/browser
- Implemented tools for navigating, clicking, typing, waiting for elements,
getting text, evaluating JavaScript, taking screenshots, and scrolling
- Added lazy browser initialization that defers until first use
- Integrated with the agent to expose these tools to Claude
2. Screenshot handling and display:
- Implemented screenshot storage with UUID-based IDs in /tmp/sketch-screenshots
- Added endpoint to serve screenshots via /screenshot/{id}
- Created dedicated UI component for displaying screenshots
- Ensured proper responsive design with loading states and error handling
- Fixed URL paths for proper rehomed URL support
- Modified tool calls component to auto-expand screenshot results
3. Error handling and reliability:
- Added graceful error handling for browser initialization failures
- Implemented proper cleanup of browser resources
The browser automation tools provide a powerful way for Claude to interact with web content,
making it possible to scrape data, test web applications, and automate web-based tasks.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/dockerimg/testdata/testcreatedockerfile_python_misery.httprr b/dockerimg/testdata/testcreatedockerfile_python_misery.httprr
index b1ef4e7..7124d60 100644
--- a/dockerimg/testdata/testcreatedockerfile_python_misery.httprr
+++ b/dockerimg/testdata/testcreatedockerfile_python_misery.httprr
@@ -1,9 +1,9 @@
httprr trace v1
-4062 2058
+4052 2258
POST https://api.anthropic.com/v1/messages HTTP/1.1
Host: api.anthropic.com
User-Agent: Go-http-client/1.1
-Content-Length: 3865
+Content-Length: 3855
Anthropic-Version: 2023-06-01
Content-Type: application/json
@@ -15,7 +15,7 @@
"content": [
{
"type": "text",
- "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/*\n\n# Strip out docs from debian.\nRUN rm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\nHere is the content of several files from the repository that may be relevant:\n\n"
+ "text": "\nCall the dockerfile tool to create a Dockerfile.\nThe parameters to dockerfile fill out the From and ExtraCmds\ntemplate variables in the following Go template:\n\n```\nFROM golang:1.24-bookworm\n\n# Switch from dash to bash by default.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\n# attempt to keep package installs lean\nRUN printf '%s\\n' \\\n 'path-exclude=/usr/share/man/*' \\\n 'path-exclude=/usr/share/doc/*' \\\n 'path-exclude=/usr/share/doc-base/*' \\\n 'path-exclude=/usr/share/info/*' \\\n 'path-exclude=/usr/share/locale/*' \\\n 'path-exclude=/usr/share/groff/*' \\\n 'path-exclude=/usr/share/lintian/*' \\\n 'path-exclude=/usr/share/zoneinfo/*' \\\n \u003e /etc/dpkg/dpkg.cfg.d/01_nodoc\n\nRUN apt-get update; \\\n\tapt-get install -y --no-install-recommends \\\n\t\tgit jq sqlite3 npm nodejs gh ripgrep fzf python3 curl vim chromium \u0026\u0026 \\\n\tapt-get clean \u0026\u0026 \\\n\trm -rf /var/lib/apt/lists/* \u0026\u0026 \\\n\trm -rf /usr/share/{doc,doc-base,info,lintian,man,groff,locale,zoneinfo}/*\n\nENV PATH=\"$GOPATH/bin:$PATH\"\n\n# While these binaries install generally useful supporting packages,\n# the specific versions are rarely what a user wants so there is no\n# point polluting the base image module with them.\n\nRUN go install golang.org/x/tools/cmd/goimports@latest; \\\n\tgo install golang.org/x/tools/gopls@latest; \\\n\tgo install mvdan.cc/gofumpt@latest; \\\n\tgo clean -cache -testcache -modcache\n\nENV GOTOOLCHAIN=auto\nENV SKETCH=1\n\nRUN mkdir -p /root/.cache/sketch/webui\n\nARG GIT_USER_EMAIL\nARG GIT_USER_NAME\n\nRUN git config --global user.email \"$GIT_USER_EMAIL\" \u0026\u0026 \\\n git config --global user.name \"$GIT_USER_NAME\" \u0026\u0026 \\\n git config --global http.postBuffer 524288000\n\nLABEL sketch_context=\"{{.InitFilesHash}}\"\nCOPY . /app\nRUN rm -f /app/tmp-sketch-dockerfile\n\nWORKDIR /app{{.SubDir}}\nRUN if [ -f go.mod ]; then go mod download; fi\n\n# Switch to lenient shell so we are more likely to get past failing extra_cmds.\nSHELL [\"/bin/bash\", \"-uo\", \"pipefail\", \"-c\"]\n\n{{.ExtraCmds}}\n\n# Switch back to strict shell after extra_cmds.\nSHELL [\"/bin/bash\", \"-euxo\", \"pipefail\", \"-c\"]\n\nCMD [\"/bin/sketch\"]\n\n```\n\nIn particular:\n- Assume it is primarily a Go project.\n- Python env setup is challenging and often no required, so any RUN commands involving python tooling should be written to let docker build continue if there is a failure.\n- Include any tools particular to this repository that can be inferred from the given context.\n- Append || true to any apt-get install commands in case the package does not exist.\n- MINIMIZE the number of extra_cmds generated. Straightforward environments do not need any.\n- Do NOT expose any ports.\n- Do NOT generate any CMD or ENTRYPOINT extra commands.\nHere is the content of several files from the repository that may be relevant:\n\n"
},
{
"type": "text",
@@ -54,24 +54,24 @@
Anthropic-Organization-Id: 3c473a21-7208-450a-a9f8-80aebda45c1b
Anthropic-Ratelimit-Input-Tokens-Limit: 200000
Anthropic-Ratelimit-Input-Tokens-Remaining: 200000
-Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-05T19:55:47Z
+Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-06T03:31:12Z
Anthropic-Ratelimit-Output-Tokens-Limit: 80000
Anthropic-Ratelimit-Output-Tokens-Remaining: 80000
-Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-05T19:55:50Z
+Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-06T03:31:16Z
Anthropic-Ratelimit-Requests-Limit: 4000
Anthropic-Ratelimit-Requests-Remaining: 3999
-Anthropic-Ratelimit-Requests-Reset: 2025-05-05T19:55:47Z
+Anthropic-Ratelimit-Requests-Reset: 2025-05-06T03:31:11Z
Anthropic-Ratelimit-Tokens-Limit: 280000
Anthropic-Ratelimit-Tokens-Remaining: 280000
-Anthropic-Ratelimit-Tokens-Reset: 2025-05-05T19:55:47Z
+Anthropic-Ratelimit-Tokens-Reset: 2025-05-06T03:31:12Z
Cf-Cache-Status: DYNAMIC
-Cf-Ray: 93b2cc227ed515f7-SJC
+Cf-Ray: 93b5673b6e3bce9c-SJC
Content-Type: application/json
-Date: Mon, 05 May 2025 19:55:51 GMT
-Request-Id: req_011CNpyJFSiFJUtRqgiMPiG3
+Date: Tue, 06 May 2025 03:31:16 GMT
+Request-Id: req_011CNqa2SbZGPRZ3VpTEzQeT
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Via: 1.1 google
X-Robots-Tag: none
-{"id":"msg_01G9uUSv5BciWemp442LWMUE","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"Based on the README.md, this project requires Python 3.11 and the DVC (Data Version Control) tool. Since the dockerfile template already includes Python3, I'll add commands to ensure Python 3.11 is installed and to set up DVC."},{"type":"tool_use","id":"toolu_01PWZdG27vVMNfzgzmLV7Hi8","name":"dockerfile","input":{"extra_cmds":"# Install Python 3.11 (if not already the default version) and set it up\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends python3.11 python3.11-venv python3-pip || true && \\\n update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 || true && \\\n apt-get clean && \\\n rm -rf /var/lib/apt/lists/*\n\n# Install DVC tool as mentioned in README\nRUN pip3 install dvc || true"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1338,"cache_read_input_tokens":0,"output_tokens":254}}
\ No newline at end of file
+{"id":"msg_01G7Rd33sjrAA319QvFpukeZ","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"Based on the README.md file, I can see that this project requires Python 3.11 and the DVC tool. I'll create a Dockerfile with appropriate extra commands to set up these requirements."},{"type":"tool_use","id":"toolu_01MQ46ddwZ7x3VerhB9uRJiR","name":"dockerfile","input":{"extra_cmds":"# Install Python 3.11 (attempt to continue on failure)\nRUN apt-get update && apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev python3-pip || true\n\n# Set up Python 3.11 as default Python\nRUN if command -v python3.11 &> /dev/null; then \\\n update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 || true; \\\n fi\n\n# Install DVC tool\nRUN pip3 install dvc || true\n\n# Create and activate Python virtual environment if needed\nRUN if [ -f requirements.txt ]; then \\\n python3 -m venv .venv || true; \\\n source .venv/bin/activate || true; \\\n pip install -r requirements.txt || true; \\\n fi"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":1333,"cache_read_input_tokens":0,"output_tokens":317}}
\ No newline at end of file