CrewAI
Search, crawl and extract web data for CrewAI agents.
Available
Give your LangChain agents live web data
Add Geonode scraping tools to your agent. Discover pages, extract content and follow long-running jobs in your existing workflow.
What You'll Need
LangChain package
Install geonode-scraper-langchain in your Python environment.
Geonode API key
Set GEONODE_SCRAPER_API_KEY before you build the tools.
An agent or direct call
Invoke a tool directly, or give the tool list to your agent.
Build your tool set
Install the package and set your API key. Then call a StructuredTool directly.
import os
from geonode_scraper_langchain import build_langchain_tools
from geonode_scraper_tools_core import ScraperToolSettings
settings = ScraperToolSettings(
host="https://scraper.geonode.io",
api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)
tools = build_langchain_tools(settings=settings)
by_name = {tool.name: tool for tool in tools}
response = by_name["scraper_extract_content"].invoke({
"url": "https://example.com",
"formats": ["markdown"],
"processing_mode": "sync",
})
print(response["result"])Tools that fit your agent
Keep setup, invocation and returned data easy to inspect.
Shared settings
Configure the production host and your API key once.
ScraperToolSettings( host=host, api_key=api_key )StructuredTool interface
Pass named arguments as a dictionary to invoke.
tool.invoke({ "url": url, "formats": ["markdown"] })JSON-friendly results
Check the response envelope, then read result.
response["ok"] response["result"]
Developer documentation
Get the full setup guide, authentication details and technical reference.
Capabilities
Connect native tools to your LangChain agents to search for and collect source content.
Extract page content as Markdown or HTML, with optional JavaScript rendering.
Run batch extraction and bounded website crawls as background jobs.
Discover website URLs before choosing which pages to extract.
Search the web and retrieve or wait for background job results.
Common workflows
# Use by_name from the quick start
submitted = by_name["scraper_extract_content"].invoke({
"url": "https://example.com",
"formats": ["markdown"],
"processing_mode": "async",
})
job_id = submitted["result"]["job_id"]
response = by_name["scraper_wait_for_job"].invoke({
"job_id": job_id,
"timeout_seconds": 120,
})
print(response["result"])Runtime and access
Runtime
Python 3.10+
Account and access
Keep API keys on the server or in your client’s credential store. Usage is charged to your Geonode plan.
Installation
Shell
pip install geonode-scraper-langchain==0.4.3Troubleshooting
Tool not found
Check the names in the generated tool list.
Invalid API key
Verify the key passed to ScraperToolSettings.
Unexpected response
Read the payload from response["result"].
Wait timed out
Keep the job ID and check its status again.
Build a dependable workflow
Start with one tool
Validate a direct call before wiring an agent.
Give clear tool instructions
Tell the agent which sources and output you need.
Inspect the envelope
Check ok, operation and result.
Track async jobs
Keep job IDs in your workflow state.
FAQ
Yes. Each StructuredTool can be called directly with invoke. Validate a direct call first, then pass the same tool list to your agent or bind it to a chat model with bind_tools.
A JSON-friendly dictionary with ok, operation, attempts and result. The data you need, such as Markdown or a job ID, is inside response["result"].
Yes. The tool set includes tools to create batch and crawl jobs, check their status and wait for completion, such as scraper_create_batch and scraper_wait_for_batch. Use the operations argument to give an agent only the tools it needs.
The LangChain guide in the Geonode documentation lists every tool with example calls and live responses. Package versions and the changelog are on PyPI under geonode-scraper-langchain.
Related integrations
Available Available Geonode MCP Server
Web search and content collection for MCP-compatible agents.
Bring the live web into your agent.
Build a tool set and try a direct extraction.
