返回

MITRE 附录A：恢复结果2014年（13页）

ID：22738

阅读量：0

大小：0.30 MB

页数：13页

时间：2022-11-28

金币：15

上传者：战必胜
Appendix A: Re-Turking Results
Burger,J., Doughty,E., Khare,R., et al. Hybrid curation of gene-mutation relations
combining automated extraction and crowdsourcing. Database (2014) Vol. 2014: article
ID bau094; doi:10.1093/database/bau094
©2014-The MITRE Corporation. All rights reserved.
1
Appendix A: Re-Turking Results
We performed two sets of experiments to attempt to provide a ground truth set for surface
relation judgments.
Returking all Turker/Gold Standard Disagreements
In the first experiment, we selected all of the HITS where the aggregate Turker judgment
differed from the Gold Standard. Three of the authors (RK, JA, LH) “re-turked” 125
HITs where the aggregate Turker judgment differed from the gold standard (both false
positives and false negatives, but done using an earlier version of the gold standard).
These three authors performed 119 HITs in common. Response counts appear in Table
A1.
Table A1: Author responses to re-Turked HITs
X
Y
Z
Total
yes
81
97
109
287
no
23
14
8
45
inconsistent
15
8
2
25
Total
119
119
119
Pairwise raw agreement and Cohen’s Kappa for author re-turked HITs appear in Table
A2.
Table A2: Pairwise Agreement and Kappa for Re-Turked HITs
X-Y
% Agreement
0.765
Cohen’s Kappa
0.432
Although the agreement levels among the three authors (Table A2) was slightly higher
than for Turkers (Table 6 in the paper), the agreement levels were not particularly high in
either case. However, there were 83 HITs in the set (66%) where all three annotators
Approved for Public Release; Distribution Unlimited. 14-0249
资源描述：

当前文档最多预览五页，下载文档查看全文

此文档下载收益归作者所有

当前文档最多预览五页，下载文档查看全文
温馨提示：
1. 部分包含数学公式或PPT动画的文件，查看预览时可能会显示错乱或异常，文件下载后无此问题，请放心下载。
2. 本文档由用户上传，版权归属用户，天天文库负责整理代发布。如果您对本文档版权有争议请及时联系客服。
3. 下载前请仔细阅读文档内容，确认文档内容符合您的需求后进行下载，若出现内容与标题不符可向本站投诉处理。
4. 下载文档时可能由于网络波动等原因无法下载或下载错误，付费完成后未能成功下载的用户请联系客服处理。
最近更新
更多
大家都在看
近期热门

分享

购物车

关闭