QANTA

Question Answering is Not a Trivial Activity

University of Maryland

What is QANTA?

QANTA (Question Aanswering is Not a Trivial Activity) is a question answering dataset composed of questions from Quizbowl - a trivia game that is challenging for both humans and machines. Each question contains 4-5 pyramidally arranged clues: obscure ones at the beginning and obvious ones at the end. Players of Quizbowl (humans and machines) compete to prove their superior mastery of knowledge by trying to answer using the least amount of information possible. More information on QANTA, including offline events, can be found at qanta.org.

New On Dec.15th University of Maryland will host a series of competitions including human vs. human, machine vs. human, and machine vs. machine. To find out more about the event and register (required for prizes), visit qanta.org.

Getting Started

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

To help you get started and demonstrate our API requirements, we provide a baseline system. The repo below contains code to: download data, train model in a docker container, evaluate model in a way that is identical to the CodaLab evaluation. You should be able to reproduce the "Baseline" entry on the leaderboard.

Submissions are done through CodaLab and the models will be tested with the same evaluation script.

Have Questions?

Please send your questions to our google group or at pedro@cs.umd.edu and shifeng@cs.umd.edu.

Acknowledgements

We thank the SQuAD team for allowing us to use their code and templates for generating this website.

Leaderboard

We evaluate each system with four metrics: accuracy at the end of the first sentence (first_acc) and at the end of the question (end_acc), and two new metrics: expected wins with system buzzer (EW) and with optimal buzzer (EW_OPT). Ranking is decided by EW.
RankModelfirst_accend_accEWEW_OPT

1

Dec 10, 2018
BitER_the_dusT

FYY

0.1190.6720.2910.618

2

Dec 10, 2018
SBQA

CMSC723 Technical Wizards

0.1040.5590.2710.589

3

Dec 10, 2018
DAN-TFIDF Buzzer

CMSC723 ForwardRethinking

0.06900.6090.2650.593

4

Dec 03, 2018
DAN

CMSC723 Technical Wizards

0.04680.5570.2140.546

5

Dec 10, 2018
GLOVE_300 DAN+TFIDF

CMSC723 ForwardRethinking

0.08770.6040.2010.594

6

Dec 11, 2018
TFIDF Guesser

CMSC723 Working Title

0.05340.4690.1930.513

7

Dec 01, 2018
TFIDF Buzzer

CMSC723 Technical Wizards

0.05950.4680.1830.514

8

Dec 01, 2018
TF-IDF Thresh

CMSC723 ForwardRethinking

0.03630.5580.1470.551

9

Nov 20, 2018
TF-IDF

CMSC723 FYY

0.04630.5450.1370.540

10

Dec 12, 2018
First submission

CMSC723 AmazingDH

0.05430.5400.1360.543

11

Dec 03, 2018
GLOVE_300 DAN

CMSC723 ForwardRethinking

0.05580.5030.1130.527

12

Nov 19, 2018
Less Than Adequate DAN

CMSC723 Iota

0.03170.4370.09490.496

13

Nov 21, 2018
TFIDF baseline

CMSC723 Technical Wizards

0.05340.4690.05090.513

14

Nov 14, 2018
DAN Wiki

University of Maryland

0.09230.5600.04870.582

15

Nov 21, 2018
Elmo DAN

University of Maryland

0.1020.5080.04530.567

16

Nov 11, 2018
DAN Baseline

University of Maryland

0.07360.4320.02070.529

17

Nov 09, 2018
Baseline submission

CMSC723 Team ROJA

0.05340.4690.002530.513

17

Nov 14, 2018
Baseline System

CMSC723 AmazingDH

0.05340.4690.002530.513

17

Nov 07, 2018
TFIDF Baseline

CMSC723 QAQA-Land

0.05340.4690.002530.513

18

Dec 05, 2018
Obscurity System

CMSC723 We Qanta Do This

https://arxiv.org/abs/something
0.05290.3940.000.490