Add browse tool support.
I reviewed some MCPs (using OpenAI's deep research to help), and it
helped me choose chromedp as the relevant library and helped me come up
with an interface. This commit adds chrome to the Docker image which is
kind of big. (I've noticed that it's smaller on Ubuntu, where it doesn't
pull in X11.) go-playwright was a library contender as well.
Implement browser automation tooling using chromedp
This implementation adds browser automation capabilities to the system via the chromedp library,
enabling Claude to interact with web content effectively.
Key features include:
1. Core browser automation functionality:
- Created new browsertools package in claudetool/browser
- Implemented tools for navigating, clicking, typing, waiting for elements,
getting text, evaluating JavaScript, taking screenshots, and scrolling
- Added lazy browser initialization that defers until first use
- Integrated with the agent to expose these tools to Claude
2. Screenshot handling and display:
- Implemented screenshot storage with UUID-based IDs in /tmp/sketch-screenshots
- Added endpoint to serve screenshots via /screenshot/{id}
- Created dedicated UI component for displaying screenshots
- Ensured proper responsive design with loading states and error handling
- Fixed URL paths for proper rehomed URL support
- Modified tool calls component to auto-expand screenshot results
3. Error handling and reliability:
- Added graceful error handling for browser initialization failures
- Implemented proper cleanup of browser resources
The browser automation tools provide a powerful way for Claude to interact with web content,
making it possible to scrape data, test web applications, and automate web-based tasks.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/loop/testdata/agent_loop.httprr b/loop/testdata/agent_loop.httprr
index 2183c46..ba483b9 100644
--- a/loop/testdata/agent_loop.httprr
+++ b/loop/testdata/agent_loop.httprr
@@ -1,9 +1,9 @@
httprr trace v1
-10862 1877
+14168 2249
POST https://api.anthropic.com/v1/messages HTTP/1.1
Host: api.anthropic.com
User-Agent: Go-http-client/1.1
-Content-Length: 10664
+Content-Length: 13970
Anthropic-Version: 2023-06-01
Content-Type: application/json
@@ -226,6 +226,156 @@
}
},
{
+ "name": "browser_navigate",
+ "description": "Navigate the browser to a specific URL and wait for page to load",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "url": {
+ "type": "string",
+ "description": "The URL to navigate to"
+ }
+ },
+ "required": [
+ "url"
+ ]
+ }
+ },
+ {
+ "name": "browser_click",
+ "description": "Click the first element matching a CSS selector",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the element to click"
+ },
+ "wait_visible": {
+ "type": "boolean",
+ "description": "Wait for the element to be visible before clicking"
+ }
+ },
+ "required": [
+ "selector"
+ ]
+ }
+ },
+ {
+ "name": "browser_type",
+ "description": "Type text into an input or textarea element",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the input element"
+ },
+ "text": {
+ "type": "string",
+ "description": "Text to type into the element"
+ },
+ "clear": {
+ "type": "boolean",
+ "description": "Clear the input field before typing"
+ }
+ },
+ "required": [
+ "selector",
+ "text"
+ ]
+ }
+ },
+ {
+ "name": "browser_wait_for",
+ "description": "Wait for an element to be present in the DOM",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the element to wait for"
+ },
+ "timeout_ms": {
+ "type": "integer",
+ "description": "Maximum time to wait in milliseconds (default: 30000)"
+ }
+ },
+ "required": [
+ "selector"
+ ]
+ }
+ },
+ {
+ "name": "browser_get_text",
+ "description": "Get the innerText of an element",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the element to get text from"
+ }
+ },
+ "required": [
+ "selector"
+ ]
+ }
+ },
+ {
+ "name": "browser_eval",
+ "description": "Evaluate JavaScript in the browser context",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "expression": {
+ "type": "string",
+ "description": "JavaScript expression to evaluate"
+ }
+ },
+ "required": [
+ "expression"
+ ]
+ }
+ },
+ {
+ "name": "browser_screenshot",
+ "description": "Take a screenshot of the page or a specific element",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the element to screenshot (optional)"
+ },
+ "format": {
+ "type": "string",
+ "description": "Output format ('base64' or 'png'), defaults to 'base64'",
+ "enum": [
+ "base64",
+ "png"
+ ]
+ }
+ }
+ }
+ },
+ {
+ "name": "browser_scroll_into_view",
+ "description": "Scroll an element into view if it's not visible",
+ "input_schema": {
+ "type": "object",
+ "properties": {
+ "selector": {
+ "type": "string",
+ "description": "CSS selector for the element to scroll into view"
+ }
+ },
+ "required": [
+ "selector"
+ ]
+ }
+ },
+ {
"name": "patch",
"description": "File modification tool for precise text edits.\n\nOperations:\n- replace: Substitute text with new content\n- append_eof: Append new text at the end of the file\n- prepend_bof: Insert new text at the beginning of the file\n- overwrite: Replace the entire file with new content (automatically creates the file)\n\nUsage notes:\n- All inputs are interpreted literally (no automatic newline or whitespace handling)\n- For replace operations, oldText must appear EXACTLY ONCE in the file",
"input_schema": {
@@ -286,25 +436,25 @@
}HTTP/2.0 200 OK
Anthropic-Organization-Id: 3c473a21-7208-450a-a9f8-80aebda45c1b
Anthropic-Ratelimit-Input-Tokens-Limit: 200000
-Anthropic-Ratelimit-Input-Tokens-Remaining: 184000
-Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-05T20:19:43Z
+Anthropic-Ratelimit-Input-Tokens-Remaining: 199000
+Anthropic-Ratelimit-Input-Tokens-Reset: 2025-05-06T02:52:06Z
Anthropic-Ratelimit-Output-Tokens-Limit: 80000
Anthropic-Ratelimit-Output-Tokens-Remaining: 80000
-Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-05T20:19:41Z
+Anthropic-Ratelimit-Output-Tokens-Reset: 2025-05-06T02:52:10Z
Anthropic-Ratelimit-Requests-Limit: 4000
Anthropic-Ratelimit-Requests-Remaining: 3999
-Anthropic-Ratelimit-Requests-Reset: 2025-05-05T20:19:37Z
+Anthropic-Ratelimit-Requests-Reset: 2025-05-06T02:52:05Z
Anthropic-Ratelimit-Tokens-Limit: 280000
-Anthropic-Ratelimit-Tokens-Remaining: 264000
-Anthropic-Ratelimit-Tokens-Reset: 2025-05-05T20:19:41Z
+Anthropic-Ratelimit-Tokens-Remaining: 279000
+Anthropic-Ratelimit-Tokens-Reset: 2025-05-06T02:52:06Z
Cf-Cache-Status: DYNAMIC
-Cf-Ray: 93b2ef104988d039-SJC
+Cf-Ray: 93b52df649282523-SJC
Content-Type: application/json
-Date: Mon, 05 May 2025 20:19:41 GMT
-Request-Id: req_011CNq17iEhcyfkeYYQUZ5tz
+Date: Tue, 06 May 2025 02:52:10 GMT
+Request-Id: req_011CNqX3XPUGXJBNMCWBpMaN
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Via: 1.1 google
X-Robots-Tag: none
-{"id":"msg_01KueCuaHxNHNG92tja2yhpo","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"I have access to the following tools:\n\n1. bash - Execute shell commands\n2. keyword_search - Search for files based on keywords\n3. think - Record thoughts and planning\n4. title - Set conversation title and create git branch\n5. done - Mark task as complete with a checklist\n6. codereview - Run automated code review\n7. multiplechoice - Present multiple choice questions to the user\n8. patch - Make precise edits to files\n\nThese tools allow me to help you with coding, research, file manipulation, and other development tasks, particularly with Go programming language."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":2602,"cache_read_input_tokens":0,"output_tokens":130}}
\ No newline at end of file
+{"id":"msg_01De9PhPGpXjkZks75fsrXew","type":"message","role":"assistant","model":"claude-3-7-sonnet-20250219","content":[{"type":"text","text":"I have the following tools available to me:\n\n1. `bash` - Execute shell commands\n2. `keyword_search` - Search for files in a codebase\n3. `think` - Record thoughts and plans\n4. `title` - Set conversation title and create git branch\n5. `done` - Mark task as complete with checklist\n6. `codereview` - Run automated code review\n7. `multiplechoice` - Present multiple choice options to user\n8. `browser_navigate` - Navigate to URL\n9. `browser_click` - Click element using CSS selector\n10. `browser_type` - Type text into input fields\n11. `browser_wait_for` - Wait for element to appear\n12. `browser_get_text` - Get text from element\n13. `browser_eval` - Run JavaScript in browser\n14. `browser_screenshot` - Take screenshot\n15. `browser_scroll_into_view` - Scroll to element\n16. `patch` - Make precise text edits to files\n\nThese tools allow me to help with coding tasks, navigate webpages, make file edits, and manage conversation state."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":4,"cache_creation_input_tokens":3303,"cache_read_input_tokens":0,"output_tokens":259}}
\ No newline at end of file