Add browse tool support.
I reviewed some MCPs (using OpenAI's deep research to help), and it
helped me choose chromedp as the relevant library and helped me come up
with an interface. This commit adds chrome to the Docker image which is
kind of big. (I've noticed that it's smaller on Ubuntu, where it doesn't
pull in X11.) go-playwright was a library contender as well.
Implement browser automation tooling using chromedp
This implementation adds browser automation capabilities to the system via the chromedp library,
enabling Claude to interact with web content effectively.
Key features include:
1. Core browser automation functionality:
- Created new browsertools package in claudetool/browser
- Implemented tools for navigating, clicking, typing, waiting for elements,
getting text, evaluating JavaScript, taking screenshots, and scrolling
- Added lazy browser initialization that defers until first use
- Integrated with the agent to expose these tools to Claude
2. Screenshot handling and display:
- Implemented screenshot storage with UUID-based IDs in /tmp/sketch-screenshots
- Added endpoint to serve screenshots via /screenshot/{id}
- Created dedicated UI component for displaying screenshots
- Ensured proper responsive design with loading states and error handling
- Fixed URL paths for proper rehomed URL support
- Modified tool calls component to auto-expand screenshot results
3. Error handling and reliability:
- Added graceful error handling for browser initialization failures
- Implemented proper cleanup of browser resources
The browser automation tools provide a powerful way for Claude to interact with web content,
making it possible to scrape data, test web applications, and automate web-based tasks.
Co-Authored-By: sketch <hello@sketch.dev>
diff --git a/dockerimg/testdata/testcreatedockerfile_python_misery.dockerfile b/dockerimg/testdata/testcreatedockerfile_python_misery.dockerfile
index 9e6ada1..b98c741 100644
--- a/dockerimg/testdata/testcreatedockerfile_python_misery.dockerfile
+++ b/dockerimg/testdata/testcreatedockerfile_python_misery.dockerfile
@@ -1,4 +1,4 @@
-FROM ghcr.io/boldsoftware/sketch:f5b4ebd9ca15d3dbd2cd08e6e7ab9548
+FROM ghcr.io/boldsoftware/sketch:99a2e4afe316b3c6cf138830dbfb7796
ARG GIT_USER_EMAIL
ARG GIT_USER_NAME
@@ -7,7 +7,7 @@
git config --global user.name "$GIT_USER_NAME" && \
git config --global http.postBuffer 524288000
-LABEL sketch_context="79b4c82f0c892e5f79900afb235c0ab50044626be5d6d43b774f6f5da9537800"
+LABEL sketch_context="7c933e98fc1d5fd35f964b6cf115bcf65b580c378069d078ab66723b2b1073c4"
COPY . /app
RUN rm -f /app/tmp-sketch-dockerfile
@@ -17,16 +17,24 @@
# Switch to lenient shell so we are more likely to get past failing extra_cmds.
SHELL ["/bin/bash", "-uo", "pipefail", "-c"]
-# Install Python 3.11 (if not already the default version) and set it up
-RUN apt-get update && \
- apt-get install -y --no-install-recommends python3.11 python3.11-venv python3-pip || true && \
- update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 || true && \
- apt-get clean && \
- rm -rf /var/lib/apt/lists/*
+# Install Python 3.11 (attempt to continue on failure)
+RUN apt-get update && apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev python3-pip || true
-# Install DVC tool as mentioned in README
+# Set up Python 3.11 as default Python
+RUN if command -v python3.11 &> /dev/null; then \
+ update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 || true; \
+ fi
+
+# Install DVC tool
RUN pip3 install dvc || true
+# Create and activate Python virtual environment if needed
+RUN if [ -f requirements.txt ]; then \
+ python3 -m venv .venv || true; \
+ source .venv/bin/activate || true; \
+ pip install -r requirements.txt || true; \
+ fi
+
# Switch back to strict shell after extra_cmds.
SHELL ["/bin/bash", "-euxo", "pipefail", "-c"]