Firecrawl Plugin

We've created a service wrapper for the Firecrawl API that provides two main functionalities: web scraping and crawling. Here's a breakdown of how it works:

Core Features

1. Service Factory

   • Creates a service instance using an API key
   • Returns an object with two methods: getScrapeData and getCrawlData

2. Web Scraping (getScrapeData)

   • Fetches and extracts content from a single webpage
   • Returns structured data including page content and metadata
   • Endpoint: https://api.firecrawl.dev/v1/scrape

3. Web Searching (getSearchData)

   • Searches for data based on conversations
   • Endpoint: https://api.firecrawl.dev/v1/search

Configuration

The plugin requires minimal configuration. In your character file, simply add:

{
    "FIRECRAWL_API_KEY": "your-api-key-here"
}

Usage Examples

Web Scraping

The plugin recognizes various ways users might request web scraping:

// Single URL request
"Can you scrape the content from https://example.com?"
"Get the data from www.example.com/page"

// Two-step interaction
User: "I need to scrape some website data."
Agent: "I can help you scrape website data. Please share the URL you'd like me to process."
User: "example.com/products"

Web Search

The plugin handles different crawling request patterns:

// Direct search
"Find the latest news about SpaceX launches"
"Can you find details about the iPhone 16 release?"

Response Handling

The plugin automatically:

• Validates URLs before processing
• Handles both direct and conversational requests
• Provides appropriate feedback during the scraping/crawling process
• Returns structured data from the target website

Error Handling

The plugin includes built-in error handling for common scenarios:

• Invalid or missing URLs
• API authentication issues
• Network failures
• Malformed responses

Actions

The plugin provides two main actions:

• FIRECRAWL_GET_SCRAPED_DATA: For single-page content extraction
• WEB_SEARCH: Web search for any data

Security

• API keys should be kept secure and never shared
• All requests are made over HTTPS
• Input validation is performed on all URLs before processing