Assess the quality of user submitted annotations

[goodb]

The project needs a way to know how the user submitted annotations compare to expert and machine-generated annotations. In cases where a gold standard exists, it should be used to benchmark the data and make decisions about how the project should proceed. There are many that would be applicable e.g. from https://f1000research.com/articles/3-96

[x0xMaximus]

Potential Discussion Points:

• Variable sources of gold standard comparisons
• Sources for expert annotations: intra Mark2Cure environment or comparison to Expert annotations in other annotation projects
• Level of refinement for comparison user, is our users population subdivided and organized in specific ways
• Does "make decisions about how the project should proceed" imply automatic document import and task type distribution (in a multi task environment)