Learning from Crowds by Modeling Common Confusions
Abstract:
Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low cost. However, the annotation quality of annotators varies considerably, which imposes new challenges in learning a high-quality model from the crowdsourced annotations. In this work, we provide a new perspective to decompose annotation noise into common noise and individual noise and differentiate the source of confusion based on instance difficulty and annotator expertise on a per-instance-annotator basis. We will realize this new crowdsourcing model by an end-to-end learning solution, which is believed to be both efficient and effective, with two types of noise adaptation layers: one is shared across annotators to capture their commonly shared confusions, and the other one is pertaining to each annotator to realize individual confusion. To recognize the source of noise in each annotation, we will use an auxiliary network to choose the two noise adaptation layers with respect to both instances and annotators.
Committee:
- Aidong Zhang, Committee Chair (The Department of Computer Science, UVA)
- Hongning Wang, Advisor (The Department of Computer Science, UVA)
- Vicente Ordóñez Román (The Department of Computer Science, UVA)
- Jundong Li (The Departments of CS & ECE, SEAS, DSI, UVA)