Scrape Speed

[rickydaricky]

Is there a built-in sleep duration once we log in with credentials? Because I'm finding that my scrapes are slower once I log in, but I'm not sure how or why.

And as a follow up, what is the limiting factor in the speed of scraping? If I run the program on a faster machine, will I necessarily get faster results? I'm trying to get ~30000 posts from a page and am considering just putting it on a VPS.

Thanks!

[neon-ninja]

There's no built in sleeps, except in the case of an HTTPError (

facebook-scraper/facebook_scraper/page_iterators.py

Line 103 in 8397f43

time.sleep(sleep_duration)

)
Probably the reason it seems slower, is that Facebook also has to serve up elements specific to your account.
The main limiting factor is the response time of Facebook servers. It depends what you mean by faster machine. The closer to Facebook servers, the better in theory. Lower latency = faster page load time. However, I generally wouldn't recommend running it on a VPS, as it'll be pretty obvious that you're scraping. Most users don't browse Facebook from a VPS.

Perhaps try setting the "posts_per_page": 200 and "allow_extra_requests": False options, as those can greatly speed up post extraction