Suggestions for sleep values ?

[evildrome]

Hi,

I recently did a download over 8 sessions:

Session............ total downloaded............... Time.................... downloaded...................... time diff

1 .............................. 3414 ................................ 11:31.............................. 0
2 .............................. 4835 ................................ 12:37.............................. 1421............................. 01:06
3 .............................. 11418 .............................. 13:38.............................. 6583............................. 01:01
4 .............................. 11817 .............................. 15:09.............................. 399............................... 01:31
5 .............................. 11971 .............................. 16:16.............................. 154 .............................. 01:07
6 .............................. 12303 .............................. 20:21.............................. 332 .............................. 04:05
7 .............................. 12329 .............................. 21:22.............................. 26 ................................ 01:01
8 .............................. 12577 .............................. 09:03.............................. 248 ............................. 11:41

I did not use any sleep values.

The number of posts & comments downloaded in each session is highly variable.

Wait time between sessions also seems not to be relevant (see session 8).

Are other people seeing this crazy variability in the amount downloaded before being blocked?

And... can anyone recommend a strategy for setting sleep values & time to wait before restart after being blocked?

Cheers,

Wilson.

[neon-ninja]

What are these numbers? Post counts?
If you're getting temporary bans, you're scraping too much, too fast. Either scrape slower or less.

[evildrome]

Yes, the numbers are the posts & comments scraped.

I'm just trying to figure out the logic behind the temporary ban.

I'm used to this sort of throttling feature having downloaded YG! for 20 years.

Their throttle depended on the total bandwidth through the group.

So if you had a very large group with a lot of activity you could download proportionately more than a small group where your individual activity would represent a larger total of the group activity.

Also, their download volume limit was the same (more or less) per group.

Not like this where having downloaded approximately once per hour I got 1421, 6583, 399, 154, posts & comments.

I'd like to automate this & I'd like to set limits that won't hit the ban but I'm not setting it at 154 per hour, so... what to set it at I wonder.

[neon-ninja]

It's a lot more complex that that. Some requests are "more suspicious" than others. It's easier to scrape posts en mass than comments. Maybe try scrape the posts first, then come back for the comments later. See #637 (comment)

[evildrome]

Ahhh... I understand.

Thanks. I'll try getting just the posts initially.