Skip to content

Latest commit

 

History

History
85 lines (59 loc) · 2.2 KB

Readme.md

File metadata and controls

85 lines (59 loc) · 2.2 KB

Firecrawl Plugin

We've created a service wrapper for the Firecrawl API that provides two main functionalities: web scraping and crawling. Here's a breakdown of how it works:

Core Features

  1. Service Factory

    • Creates a service instance using an API key
    • Returns an object with two methods: getScrapeData and getCrawlData

  2. Web Scraping (getScrapeData)

    • Fetches and extracts content from a single webpage
    • Returns structured data including page content and metadata
    • Endpoint: https://api.firecrawl.dev/v1/scrape

  3. Web Searching (getSearchData)

    • Searches for data based on conversations
    • Endpoint: https://api.firecrawl.dev/v1/search

Configuration

The plugin requires minimal configuration. In your character file, simply add:

{
    "FIRECRAWL_API_KEY": "your-api-key-here"
}

Usage Examples

Web Scraping

The plugin recognizes various ways users might request web scraping:

// Single URL request
"Can you scrape the content from https://example.com?"
"Get the data from www.example.com/page"

// Two-step interaction
User: "I need to scrape some website data."
Agent: "I can help you scrape website data. Please share the URL you'd like me to process."
User: "example.com/products"

Web Search

The plugin handles different crawling request patterns:

// Direct search
"Find the latest news about SpaceX launches"
"Can you find details about the iPhone 16 release?"

Response Handling

The plugin automatically:

  • Validates URLs before processing
  • Handles both direct and conversational requests
  • Provides appropriate feedback during the scraping/crawling process
  • Returns structured data from the target website

Error Handling

The plugin includes built-in error handling for common scenarios:

  • Invalid or missing URLs
  • API authentication issues
  • Network failures
  • Malformed responses

Actions

The plugin provides two main actions:

  • FIRECRAWL_GET_SCRAPED_DATA: For single-page content extraction
  • WEB_SEARCH: Web search for any data

Security

  • API keys should be kept secure and never shared
  • All requests are made over HTTPS
  • Input validation is performed on all URLs before processing