LangChain
Web research and content collection for LangChain agents.
Available
Give your CrewAI agents web research tools
Let your crew collect fresh web content with Geonode. Equip research agents with extraction, discovery and job-management tools.
What You'll Need
Python 3.10–3.13
Use a Python version supported by CrewAI.
CrewAI package
Install geonode-scraper-crewai in your project.
Geonode API key
Set GEONODE_SCRAPER_API_KEY in your environment.
Create tools for your crew
Build the tool list, then test extraction with run before attaching it to an Agent.
import os
from geonode_scraper_crewai import build_crewai_tools
from geonode_scraper_tools_core import ScraperToolSettings
settings = ScraperToolSettings(
host="https://scraper.geonode.io",
api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)
tools = build_crewai_tools(settings=settings)
by_name = {tool.name: tool for tool in tools}
response = by_name["scraper_extract_content"].run(
url="https://example.com",
formats=["markdown"],
processing_mode="sync",
)
print(response["result"])Configure your research workflow
Set up tools once and make the research task explicit.
Create the tools
Build the CrewAI tool list with shared settings.
tools = build_crewai_tools( settings=settings )Attach to an Agent
Give your agent the tools it needs for the task.
from crewai import Agent Agent( role="Web researcher", goal="Collect source content", backstory="Research assistant", tools=tools )Read the result
Direct tool calls return a dictionary.
response["ok"] response["result"]
Developer documentation
Get the full setup guide, authentication details and technical reference.
Capabilities
Connect native tools to your CrewAI agents to collect source content for research tasks.
Extract page content as Markdown or HTML, with optional JavaScript rendering.
Run batch extraction and bounded website crawls as background jobs.
Discover website URLs before choosing which pages to extract.
Search the web and retrieve or wait for background job results.
Common workflows
# Use by_name from the quick start
submitted = by_name["scraper_create_batch"].run(
urls=[
"https://example.com/a",
"https://example.com/b",
],
formats=["markdown"],
)
response = by_name["scraper_wait_for_batch"].run(
job_id=submitted["result"]["job_id"],
timeout_seconds=120,
)
print(response["result"])Runtime and access
Runtime
Python 3.10–3.13
Account and access
Keep API keys on the server or in your client’s credential store. Usage is charged to your Geonode plan.
Installation
Shell
pip install geonode-scraper-crewai==0.4.3Troubleshooting
Installation fails
Check the supported Python version and environment.
Agent cannot use tools
Confirm that the generated tools are attached to the Agent.
Invalid API key
Check the key in ScraperToolSettings.
Payload looks different
Read data inside response["result"].
Build a dependable workflow
Test a direct call
Validate run before launching a crew.
Define the research scope
Specify sources, limits and expected output.
Keep source references
Carry URLs into the next task.
Monitor long jobs
Use status and wait tools for async work.
FAQ
Yes. Call a tool directly with run and keyword arguments. Once the direct call works, attach the same tool list to your Agent.
CrewAI requires Python 3.10 through 3.13. Install geonode-scraper-crewai in an environment that uses one of these versions.
Extraction, batch, crawl, map and search, plus account tools for statistics, concurrency usage and health checks. Job-based groups include tools to create a job, check its status and wait for the result.
Yes. Each tool returns a JSON-friendly dictionary with ok, operation, attempts and result. Read the payload from response["result"].
Related integrations
Available Available Python SDK
Typed Python clients for web research and data pipelines.
Equip your crew with fresh sources.
Start with one tool call, then connect your Agent.
