post['time'] returns NoneType #662

micjgam · 2022-02-07T22:11:20Z

micjgam
Feb 7, 2022

Been running my script for a few days now (after a crash course in Docker and Docker Compose) with no issues...

Got one today, though

      delta = c_time - post['time']
 TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'

When would there ever be a case when post['time'] would be NoneType? delta and c_time are my own variables and I know what they are - so no issue there. I would think that ['time'] would always have a value with anyone posting and FB setting a timestamp to that post. Or, is that what ['post_time'] (type: time.time) would be?

Started the service back up to inspect the error. No issues now, it seems... maybe just a random bug...

neon-ninja · 2022-02-07T22:36:26Z

neon-ninja
Feb 7, 2022
Collaborator

What Page, Group or Profile did you have this problem with? Can you link to the problematic post?

1 reply

micjgam Feb 8, 2022
Author

GROUP ID = 971411739549035

I am not sure which specific post it was since I am filtering by keyword. By the time the error occurred, the keyword had already passed the initial gate. That particular line is to assess when the post was written and filter according to how many days old it is against the current time (c_time).

It's working fine again, maybe just attribute it to a random bug that may not be worth tracing down?

micjgam · 2022-02-11T22:28:48Z

micjgam
Feb 11, 2022
Author

same thing again today on the same group... can't find the offending post...

Wrote it out with this:

      try:
          delta = c_time - post['time']
      except TypeError:
          break
      delta = delta.total_seconds()

Script will skip the offending post and iterate to the next in line.

0 replies

micjgam · 2022-02-16T14:21:16Z

micjgam
Feb 16, 2022
Author

Now I am getting the same TypeError on post['text'] and post['post_text']... both are returning NoneType.

I ran a basic script to output the posts to the interpreter, but nothing is glaring out at me. All posts contained a text or post_text key/value pair.

      posts = get_posts(group=os.environ['FB_GROUP1'], cookies='.cooke.json',
                            options={'allow extra requests': False}, pages=3)

      for post in posts:
          with open('words.txt') as keyword:
              keyword = keyword.read().splitlines()
          for kw in keyword:
              if post['text'] in post is not None:
                  if kw in post['text']:
                      post_time_str = str(post['time'])
                      poster = post['username']
                      posting = sms.reft(post['text'][:75])
                      try:
                          delta = c_time - post['time']
                      except TypeError:
                          with open('status.txt', 'a') as c_status:
                              c_status.write('TypeError occurred: TIME was NoneType\n')
                          break
                      delta = delta.total_seconds()
                      if delta <= float(172800):
                          # write operation will not overwrite in r+ mode
                          with open('log.txt', 'r+') as log_rw:
                              log_post_time = log_rw.read().splitlines()
                              post_time_list = [post_time_str]
                              for post_time in post_time_list:
                                  if post_time in log_post_time:
                                      break
                                  else:
                                      sms.send(f'GYP post {post_time_str}\r\r{poster}\r\r{posting}',
                                               os.environ['SEND2'])
                                      log_rw.write(post_time_str + '\n')
              else:
                  with open('status.txt', 'a') as c_status:
                      c_status.write('TypeError occurred: POST_TEXT was NoneType\n')
                  break

UPDATE:

Changed the group and had no issues... so... the script is okay?

UPDATE:

I think I found the offending post. How does the scraper treat these types of posts? I have a feature built in that logs the datetime of results and omits them in subsequent re-scans, but this one kept coming up even though I had logged the date.

1 reply

neon-ninja Feb 16, 2022
Collaborator

options={'allow extra requests': False} won't do anything, the argument is options={'allow_extra_requests': False}. Perhaps you're making a lot more requests that you expected and inadvertently getting temp banned. This post extracts fine for me, here's what I get:

{'available': True,
 'comments': 10,
 'comments_full': None,
 'factcheck': None,
 'header': 'Bruce Foster posted in Greenville Young Professionals.',
 'image': None,
 'image_id': None,
 'image_ids': [],
 'image_lowquality': 'https://scontent.xx.fbcdn.net/m1/v/t6/An_UvxJXg9tdnLU3Y5qjPi0200MLilhzPXUgxzGjQzUMaNcmjdZA6anyrngvkdub33NZzZhd51fpCAEzNHFhko5aKRFP5fS1w_lKwYrzcNLupv27.png?ccb=10-5&oh=00_AT8r_DYfJnDLUo_JkXwxwAnEP1qECO5WK9fEQYr0gAqF5w&oe=6212A0B9&_nc_sid=55e238',
 'images': None,
 'images_description': None,
 'images_lowquality': ['https://scontent.xx.fbcdn.net/m1/v/t6/An_UvxJXg9tdnLU3Y5qjPi0200MLilhzPXUgxzGjQzUMaNcmjdZA6anyrngvkdub33NZzZhd51fpCAEzNHFhko5aKRFP5fS1w_lKwYrzcNLupv27.png?ccb=10-5&oh=00_AT8r_DYfJnDLUo_JkXwxwAnEP1qECO5WK9fEQYr0gAqF5w&oe=6212A0B9&_nc_sid=55e238',
                       'https://scontent.xx.fbcdn.net/m1/v/t6/An-SJYN61eefFdoaV8pa0G_5_APCa0prZaqkZGXpCFeUCLCg89UPOqSkSZxJkLy0hAKKpUIPEB91mo9yBBfcAqcwmpEu5jN_jmJufFtJoVJCUklu.png?ccb=10-5&oh=00_AT9hj-MLBJ3pk9dRG_iF81VSW_HTvIXNjwnLA8lTMxZ0Ww&oe=6211FCDF&_nc_sid=55e238'],
 'images_lowquality_description': ['Like', 'Love'],
 'is_live': False,
 'likes': 3,
 'link': None,
 'links': [],
 'original_text': None,
 'page_id': '971411739549035',
 'post_id': '4942008105822692',
 'post_text': "Hey GYP! I'm looking or someone who can take video content, "
              "that I've already captured, and edit it into short "
              'clips/stories for social media?\n'
              'Any suggestions or recommendations\n'
              '?',
 'post_url': 'https://m.facebook.com/groups/GreenvilleYoungProfessionals/permalink/4942008105822692/',
 'reaction_count': None,
 'reactions': None,
 'reactors': None,
 'shared_post_id': None,
 'shared_post_url': None,
 'shared_text': '',
 'shared_time': None,
 'shared_user_id': None,
 'shared_username': None,
 'sharers': None,
 'shares': 0,
 'text': "Hey GYP! I'm looking or someone who can take video content, that "
         "I've already captured, and edit it into short clips/stories for "
         'social media?\n'
         'Any suggestions or recommendations\n'
         '?',
 'time': datetime.datetime(2022, 2, 15, 5, 42, 53),
 'timestamp': 1644856973,
 'user_id': 1739154530,
 'user_url': 'https://facebook.com/FosterEmpire?groupid=971411739549035&refid=18&_ft_=qid.-6857243215025253633%3Amf_story_key.4942008105822692%3Aog_action_id.1128127288004916%3Atop_level_post_id.4942008105822692%3Atl_objid.4942008105822692%3Acontent_owner_id_new.1739154530%3Apage_id.971411739549035%3Atext_formatting.154977255088164%3Asrc.22%3Astory_location.6%3Afilter.GroupStoriesByActivityEntQuery%3Aott.AX9ggERK_fotN6LQ%3Atds_flgs.3%3Apage_insights.%7B%22971411739549035%22%3A%7B%22page_id%22%3A971411739549035%2C%22page_id_type%22%3A%22group%22%2C%22actor_id%22%3A1739154530%2C%22dm%22%3A%7B%22isShare%22%3A0%2C%22originalPostOwnerID%22%3A0%7D%2C%22psn%22%3A%22EntGroupMallPostCreationStory%22%2C%22post_context%22%3A%7B%22object_fbtype%22%3A657%2C%22publish_time%22%3A1644856973%2C%22story_name%22%3A%22EntGroupMallPostCreationStory%22%2C%22story_fbid%22%3A%5B4942008105822692%5D%7D%2C%22role%22%3A1%2C%22sl%22%3A6%7D%7D&__tn__=C-R',
 'username': 'Bruce Foster',
 'video': None,
 'video_duration_seconds': None,
 'video_height': None,
 'video_id': None,
 'video_quality': None,
 'video_size_MB': None,
 'video_thumbnail': None,
 'video_watches': None,
 'video_width': None,
 'w3_fb_url': None,
 'was_live': False,
 'with': [{'link': 'https://m.facebook.com/groups/GreenvilleYoungProfessionals/?refid=18&_ft_=qid.-6857243215025253633%3Amf_story_key.4942008105822692%3Aog_action_id.1128127288004916%3Atop_level_post_id.4942008105822692%3Atl_objid.4942008105822692%3Acontent_owner_id_new.1739154530%3Apage_id.971411739549035%3Atext_formatting.154977255088164%3Asrc.22%3Astory_location.6%3Afilter.GroupStoriesByActivityEntQuery%3Aott.AX9ggERK_fotN6LQ%3Atds_flgs.3%3Apage_insights.%7B%22971411739549035%22%3A%7B%22page_id%22%3A971411739549035%2C%22page_id_type%22%3A%22group%22%2C%22actor_id%22%3A1739154530%2C%22dm%22%3A%7B%22isShare%22%3A0%2C%22originalPostOwnerID%22%3A0%7D%2C%22psn%22%3A%22EntGroupMallPostCreationStory%22%2C%22post_context%22%3A%7B%22object_fbtype%22%3A657%2C%22publish_time%22%3A1644856973%2C%22story_name%22%3A%22EntGroupMallPostCreationStory%22%2C%22story_fbid%22%3A%5B4942008105822692%5D%7D%2C%22role%22%3A1%2C%22sl%22%3A6%7D%7D&__tn__=C-R',
           'name': 'Greenville Young Professionals'}]}

micjgam · 2022-02-17T19:57:32Z

micjgam
Feb 17, 2022
Author

Okay... fixed the options option, posts come through much faster.

I think I found the issue. Ran the script this morning and got an error first thing. Came back a few hours later and started it up again with no error. I ran the quick script as a baseline to see if there were any issues. An hour later, the script stopped again with the same error. I ran the quick script again and got a PytzUsageWarning in the middle of the posts. I have this error message filtered out in the script (as per #601), but I have not told the script to do anything else with it.

Post #4952038748152961 had metadata with it and then the next entry showed the same post with None for the fields. Why would this be the case? See the attached file and search by post number. File is CSV and I cannot highlight the erroneous post.

Book1.csv

Looking at the direct post, it appears that there is a post within a post and that requires translation as well.

1 reply

neon-ninja Feb 17, 2022
Collaborator

I see the issue, this one is reproducible with this code:

for post in get_posts(group=971411739549035, options={"allow_extra_requests": False}):
    if not post["text"]:
        pprint(post)
        break

Here's the HTML that caused this problem:

<a href="#" data-targetid="4952038748152961" data-ft="{&quot;qid&quot;:&quot;-6851431128931445952&quot;,&quot;mf_story_key&quot;:&quot;4952038748152961&quot;,&quot;top_level_post_id&quot;:&quot;4952038748152961&quot;,&quot;tl_objid&quot;:&quot;4952038748152961&quot;,&quot;content_owner_id_new&quot;:&quot;100046947553431&quot;,&quot;page_id&quot;:&quot;971411739549035&quot;,&quot;src&quot;:22,&quot;photo_attachments_list&quot;:[&quot;469757924599122&quot;,&quot;469757941265787&quot;],&quot;story_location&quot;:6,&quot;story_attachment_style&quot;:&quot;album&quot;,&quot;filter&quot;:&quot;GroupStoriesByActivityEntQuery&quot;,&quot;ott&quot;:&quot;AX_2-EA2TqdLuEZP&quot;,&quot;tds_flgs&quot;:3,&quot;page_insights&quot;:{&quot;971411739549035&quot;:{&quot;page_id&quot;:&quot;971411739549035&quot;,&quot;page_id_type&quot;:&quot;group&quot;,&quot;actor_id&quot;:&quot;100046947553431&quot;,&quot;dm&quot;:{&quot;isShare&quot;:0,&quot;originalPostOwnerID&quot;:0},&quot;psn&quot;:&quot;EntGroupMallPostCreationStory&quot;,&quot;post_context&quot;:{&quot;object_fbtype&quot;:657,&quot;publish_time&quot;:1645120455,&quot;story_name&quot;:&quot;EntGroupMallPostCreationStory&quot;,&quot;story_fbid&quot;:[&quot;4952038748152961&quot;]},&quot;role&quot;:1,&quot;sl&quot;:6}}}" data-sigil="m-see-translate-link">See Translation</a>

Note the presence of the data-ft attribute, containing top_level_post_id. This would be picked up by the selector in https://github.com/kevinzg/facebook-scraper/blob/v0.2.53/facebook_scraper/page_iterators.py#L151. So the "See Translation" link is interpreted as it's own post, which is why it's so empty. 6ac53d7 should fix this.

Also note that it's possible for a post to not contain any text (if say, it just contains an image). See https://m.facebook.com/groups/GreenvilleYoungProfessionals/permalink/4949797555043747/?_rdr for example. Extracts like this:

{'available': True,
 'comments': 0,
 'comments_full': None,
 'factcheck': None,
 'header': 'Kiani Gates\u200eGreenville Young Professionals',
 'image': None,
 'image_id': '140759278408502',
 'image_ids': ['140759278408502'],
 'image_lowquality': 'https://scontent.fakl8-1.fna.fbcdn.net/v/t39.30808-6/cp0/e15/q65/s320x320/273836660_140759275075169_5244096214075975523_n.jpg?_nc_cat=107&ccb=1-5&_nc_sid=5bac3a&efg=eyJpIjoidCJ9&_nc_ohc=iM3p3TPNZJUAX8lJtbb&_nc_ht=scontent.fakl8-1.fna&oh=00_AT9ocfoUqz_dxlmYr3WvoYbiXsLQNpgmgCGBVJ29eS9Bfg&oe=62142544',
 'images': None,
 'images_description': None,
 'images_lowquality': ['https://scontent.fakl8-1.fna.fbcdn.net/v/t39.30808-6/cp0/e15/q65/s320x320/273836660_140759275075169_5244096214075975523_n.jpg?_nc_cat=107&ccb=1-5&_nc_sid=5bac3a&efg=eyJpIjoidCJ9&_nc_ohc=iM3p3TPNZJUAX8lJtbb&_nc_ht=scontent.fakl8-1.fna&oh=00_AT9ocfoUqz_dxlmYr3WvoYbiXsLQNpgmgCGBVJ29eS9Bfg&oe=62142544'],
 'images_lowquality_description': ['May be an image of body of water and text '
                                   "that says 'Who needs a secured credit "
                                   'card? No credit check! No fees No minimum '
                                   "deposit GUARANTEED APPROVAL!'"],
 'is_live': False,
 'likes': 0,
 'link': None,
 'links': [],
 'original_text': None,
 'page_id': '971411739549035',
 'post_id': '4949797555043747',
 'post_text': '',
 'post_url': 'https://m.facebook.com/groups/GreenvilleYoungProfessionals/permalink/4949797555043747/',
 'reaction_count': None,
 'reactions': None,
 'reactors': None,
 'shared_post_id': None,
 'shared_post_url': None,
 'shared_text': None,
 'shared_time': None,
 'shared_user_id': None,
 'shared_username': None,
 'sharers': None,
 'shares': 0,
 'text': '',
 'time': datetime.datetime(2022, 2, 17, 14, 55, 32),
 'timestamp': 1645062932,
 'user_id': '100074232915639',
 'user_url': 'https://facebook.com/profile.php?id=100074232915639&groupid=971411739549035&refid=18&_ft_=qid.-6851370243586348938%3Amf_story_key.4949797555043747%3Atop_level_post_id.4949797555043747%3Atl_objid.4949797555043747%3Acontent_owner_id_new.100074232915639%3Apage_id.971411739549035%3Asrc.22%3Astory_location.6%3Astory_attachment_style.photo%3Afilter.GroupStoriesByActivityEntQuery%3Aott.AX-jGhHq9BL6xwMa%3Atds_flgs.3%3Apage_insights.%7B%22971411739549035%22%3A%7B%22page_id%22%3A971411739549035%2C%22page_id_type%22%3A%22group%22%2C%22actor_id%22%3A100074232915639%2C%22dm%22%3A%7B%22isShare%22%3A0%2C%22originalPostOwnerID%22%3A0%7D%2C%22psn%22%3A%22EntGroupMallPostCreationStory%22%2C%22post_context%22%3A%7B%22object_fbtype%22%3A657%2C%22publish_time%22%3A1645062932%2C%22story_name%22%3A%22EntGroupMallPostCreationStory%22%2C%22story_fbid%22%3A%5B4949797555043747%5D%7D%2C%22role%22%3A1%2C%22sl%22%3A6%7D%7D&__tn__=C-R',
 'username': 'Kiani Gates',
 'video': None,
 'video_duration_seconds': None,
 'video_height': None,
 'video_id': None,
 'video_quality': None,
 'video_size_MB': None,
 'video_thumbnail': None,
 'video_watches': None,
 'video_width': None,
 'w3_fb_url': None,
 'was_live': False,
 'with': [{'link': 'https://m.facebook.com/groups/GreenvilleYoungProfessionals/?refid=18&_ft_=qid.-6851370243586348938%3Amf_story_key.4949797555043747%3Atop_level_post_id.4949797555043747%3Atl_objid.4949797555043747%3Acontent_owner_id_new.100074232915639%3Apage_id.971411739549035%3Asrc.22%3Astory_location.6%3Astory_attachment_style.photo%3Afilter.GroupStoriesByActivityEntQuery%3Aott.AX-jGhHq9BL6xwMa%3Atds_flgs.3%3Apage_insights.%7B%22971411739549035%22%3A%7B%22page_id%22%3A971411739549035%2C%22page_id_type%22%3A%22group%22%2C%22actor_id%22%3A100074232915639%2C%22dm%22%3A%7B%22isShare%22%3A0%2C%22originalPostOwnerID%22%3A0%7D%2C%22psn%22%3A%22EntGroupMallPostCreationStory%22%2C%22post_context%22%3A%7B%22object_fbtype%22%3A657%2C%22publish_time%22%3A1645062932%2C%22story_name%22%3A%22EntGroupMallPostCreationStory%22%2C%22story_fbid%22%3A%5B4949797555043747%5D%7D%2C%22role%22%3A1%2C%22sl%22%3A6%7D%7D&__tn__=C-R',
           'name': 'Greenville Young Professionals'}]}

micjgam · 2022-02-18T00:14:07Z

micjgam
Feb 18, 2022
Author

I am new to commits so the one you mention above has not been tested? I made the manual change to the file you pointed to and it did not seem to work.

For now, this is how I am writing it out:

      for kw in kw:
          if post['text'] is None or post['time'] is None:
              break
          elif kw in post['text'].casefold():

In the final result, I am only needing these key:values anyway.

1 reply

neon-ninja Feb 18, 2022
Collaborator

pip install git+https://github.com/kevinzg/facebook-scraper.git should update your copy of facebook-scraper to the latest master branch, which contains this commit. I tested this commit, and it worked for me, but a second opinion is always welcome ;).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

post['time'] returns NoneType #662

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

post['time'] returns NoneType #662

micjgam Feb 7, 2022

Replies: 5 comments · 4 replies

neon-ninja Feb 7, 2022 Collaborator

micjgam Feb 8, 2022 Author

micjgam Feb 11, 2022 Author

micjgam Feb 16, 2022 Author

neon-ninja Feb 16, 2022 Collaborator

micjgam Feb 17, 2022 Author

neon-ninja Feb 17, 2022 Collaborator

micjgam Feb 18, 2022 Author

neon-ninja Feb 18, 2022 Collaborator

micjgam
Feb 7, 2022

Replies: 5 comments 4 replies

neon-ninja
Feb 7, 2022
Collaborator

micjgam Feb 8, 2022
Author

micjgam
Feb 11, 2022
Author

micjgam
Feb 16, 2022
Author

neon-ninja Feb 16, 2022
Collaborator

micjgam
Feb 17, 2022
Author

neon-ninja Feb 17, 2022
Collaborator

micjgam
Feb 18, 2022
Author

neon-ninja Feb 18, 2022
Collaborator