The results of the Efficient Open-Domain Question Answering Competition were presented at the Conference on Neural Information Processing Systems（NeurIPS）2020 on December 12, 2020.
The new model Sōseki, developed by Studio Ousia in collaboration with Tohoku University, ranked #2 in the restricted 6GB track following Facebook. In the unrestricted track, the model ranked #3 following Microsoft and Facebook. Human evaluation found our model to have 62% accuracy rate.
reference: Efficient Open-Domain Question Answering
Systems under 6GB leaderboard
Unrestricted track leaderboard
In this competition, AI models attempted to answer 1,800 open-domain questions that were queried in Google's search engine. The Systems Under 6GB Track is designed to test practical AI systems. In this track, contestants had to contain the whole system, including the models and data used for question answering within 6GB. The models were restricted from accessing the internet, therefore all the data had to fit in the 6GB image size. The unrestricted track was free of these restrictions.
Sōseki consists of two neural network models: the Retriever that searches for candidate passages that are relevant to the given question, and the Reader that selects the most relevant passage and extracts an answer from it. Using these two models, Sōseki is able to find the Wikipedia passage that is likely to contain the answer, and extract the answer with high accuracy.
We plan to adopt Sōseki’s technology to manuals and internal business documents and release a new product.
Presentation at NeurIPS 2020