Firecrawl is so expensive---Looking forward to alternatives

[GanQiao1990]

Firecrawl is so expensive---Looking forward to alternatives

[romor001]

Could Simon Willisons shot-scraper maybe be a solution?
https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/

[thadius83]

@GanQiao1990 check out my repo.

https://github.com/thadius83/firecrawl-customgoogle

Takes about 20 mins.

I’ll be adding multiple proxy support and bot detection soon.

[romor001]

Crawl4AI might also be an option (https://crawl4ai.com/mkdocs/ - https://github.com/unclecode/crawl4ai)

[loorisr]

I've developed a tool that you can use to integrate crawl4ai, self-hosted firecrawl, searxng, Tavily, ... with this awesome deep research tool!

https://github.com/loorisr/crawlrouter

[TomLucidor]

Seconding this between the different tools, why is CrawlRouter not more hot?

[loorisr]

I made a fork of firecrawl that uses SearXNG as a backend for searching. It is usable out of the box for scraping and searching.

https://github.com/loorisr/firecrawl-searxng

[GanQiao1990]

thanks for you kind. i think this also a good project for the information scrap https://github.com/Mintplex-Labs/anything-llm