Only scrapes few posts

[krinya]

Hi all,

First of all thanks for the quick reply. I really appreciate it. But again I ran into an issue.

I want to scrape many post of multiple facebook pages and save the data. It works fine for the 1st or the 2nd page when I start the script. But after some time I only can scrape 5 post of one page.

I am not a computer scientist I do not know the exact reason which can go bad.

I am using this function:

def get_page_data(page_name = 'fideszHU', page_lenght = 10, print_number = False, print_data = False):

# initialize pandas df:    
page_data = pd.DataFrame()


# get data from one facebook page
i = 1
if print_number:
    print("I am at this post of one page:")

for post in get_posts(page_name, pages = page_lenght, extra_info = False,
                     options={"allow_extra_requests": True, "posts_per_page": 50}):
    
    if print_number:
        print(i)
    
    if print_data:
        print(post)
    
    one_post_data = pd.DataFrame.from_dict(post, orient='index')
    one_post_data = one_post_data.transpose()
    
    one_post_data['download_datetime'] = datetime.now().strftime("%d/%m/%Y %H:%M:%S.%f")
    one_post_data['download_timestamp'] = time.time()
    
    page_data = page_data.append(one_post_data, ignore_index = True)
    
    i = i + 1

return(page_data)

And when I apply it first I get many results but then sometime later only the first 5 posts are scaped.

one_page_data = get_page_data(page_name= "karacsonygergely", page_lenght = 3, print_data = False, print_number = True)

Any idea what can go wrong?

Kind regards,
Kristof

[neon-ninja]

Do you get any errors or exceptions? Maybe try add some time.sleep statements

[5uru]

hello for me it's not an answer but a question because I encounter the same problem. how did you solve the problem?

[5uru]

hello for me it's not an answer but a question because I encounter the same problem. how did you solve the problem?

[neon-ninja]

Try pass cookies as per the readme

[5uru]

Hello
thank you for answering me I made the cookies according to readme but with get_post I always recover at most 6 posts help me please

  ` def post_scraper(page_name, page_numbers=1000):                            ] 
            post_df_full = pd.DataFrame()
            for post in get_posts(page_name, cookies="cookies.json", extra_info=True, page=page_numbers,encoding='utf8',options={"comments": True, "reactors": True}):
    post_entry = post
    fb_post_df = pd.DataFrame.from_dict(post_entry, orient='index')
    fb_post_df = fb_post_df.transpose()
    post_df_full = post_df_full.append(fb_post_df)

    return(post_df_full) 

  tests = post_scraper(page_name='BFMBusiness')

print(tests)`

[krinya]

Sorry I do not remember whether I was able to figure it out or not. I will check tomorrow.

[neon-ninja]

This page works fine for me, the code:

set_cookies("cookies.json")
posts = list(get_posts("BFMBusiness", options={"allow_extra_requests": False}))
print(f"Got {len(posts)} posts, with {sum(p['comments'] for p in posts)} total comments")

outputs:

Got 38 posts, with 3118 total comments

[5uru]

hi despite i used the same code as you i still have the same problem. Help me I need this for my studies.
the code :
posts = list(get_posts("BFMBusiness",cookies="cookies.json", options={"allow_extra_requests": False})) print(f"Got {len(posts)} posts, with {sum(p['comments'] for p in posts)} total comments")
outputs:
Got 3 posts, with 4 total comments
or it is the speed of the internet connection which is weak

[neon-ninja]

Try set your Facebook language to English, and re-export your cookies

[5uru]

Hi I don't know what to do anymore I tried everything you said but it's still 5 posts that I receive

[neon-ninja]

Maybe there's something wrong with your cookies

[krinya]

Try it on another computer or a VM.