Pubtator vs user annotation offset differences

[gtsueng]

This may be contributing to issue #133. Basically, the user annotations are usually in line with the pubtator annotations in terms of the offset only for the first sentence worth of characters. Thereafter, there is usually a difference in the offsets of 1.

Eg. In group 5, doc: 17494702, the pubtator annotations are:
offset | text
68 | Fbx2
270 | Fbx2
463 | Fbx2
630 | Fbxo2

But user annotations are:
offset | text
68 | Fbx2
269 | Fbx2
462 | Fbx2
629 | Fbxo2

Notice that the offsets match in the first annotation, but are off by one thereafter. This is a consistent pattern observed in the pubtator vs user annotation data, so comparisions between user annotations and pubtator annotations will give poor matches. Also, since the user annotations are off by 1, you can imagine why the feedback highlight is always missing the single-character number in examples like 'N-glycanase 1'.

[JTFouquier]

Using the same example as @gtsueng used above,
User annotation offsets for document 17494702:
http://mark2cure.org/task/entity-recognition/m2c/17494702.json

Pubtator/BioC offset examples for document 17494702:
http://mark2cure.org/admin/document/pubtator/9902/?_changelist_filters=q%3D17494702

I'm still not really seeing why there is a difference, but these offsets should be the same. I also checked the offset and exact location that the first concept in each section (title and abstract) should have, and it actually seems like the user annotation .json file is correct (offset should be 125), but I need to look into this more.

[JTFouquier]

http://mark2cure.org/admin/document/pubtator/9903/?_changelist_filters=q%3D17494702
Looking here at the title this doc in the Pubtator admin, if you use a character counter, the length of the title is 125 but the offset is captured as 126.

http://www.lettercount.com/

[gtsueng]

Issue is mostly solved, but it looks like there are certain docs which the offsets start out fine, but get worse towards the mid-end of the doc. Looking at what these docs have in common, it looks like the source of the problem is the '%' sign. Basically, every time there's a percent sign, user offsets increase by 2, while pubtator offsets do not. This seems to be the cause of the problem.

[x0xMaximus]

Just saving @gtsueng notes in the Issue itself: Still problematic.xlsx

Current status is waiting to hear back from Pubtator team

[x0xMaximus]

This is an example how to compare documents that may be problematic

doc = Document.objects.get(document_id=9140401)
writer = Document.objects.as_writer(documents=[doc])

doc.section_set.last().text
writer.collection.documents[0].passages[1].text

The as_writer function must be expanded to support documents that don't have an existing pubtator result, required for "new" never seen before documents, or documents whose pubtator files were purposefully deleted.

[x0xMaximus]

e9d490a and 42c4a32 moved this to the production server but we're currently on hold waiting to hear back from them. Authentication (or some mechanism to serve 403 Forbidden) was added to the server without notice.

[x0xMaximus]

I'm waiting to close this until further testing with all docs having pubtator now and scheduled updates

[x0xMaximus]

Current analysis of broken Pubtator responses we've stored in the database