ParaphraseBench
Jump to navigation
Jump to search
A ParaphraseBench is a Benchmarking Task that evaluates the robustness of NLIDBs .
- Example(s):
- Counter-Example(s):
- See: DBPal, Neural Semantic Parser, User Interface, Natural Language Processing, Natural Language Understanding, Natural Language Generation, Question Answering Task, NLQ, SQL.
References
2019
- (DataManagementLab, 2019) ⇒ https://datamanagementlab.github.io/ParaphraseBench/ Retrieved: 2019-02-10.
- QUOTE: Current benchmarks like the GeoQuery benchmark to not explicitly test different linguistic variations which is important to understand the robustness of an NLIDB. For testing different linguistic variants in a principled manner, we therefore curated a new benchmark as part of our paper on DBPal that covers different linguistic variations for the user NL input and maps it to an expected SQL output.
The schema of our new benchmark models a medical database which contains only one table comprises of hospital’s patients attributes such as name, age, and disease. In total, the benchmark consists of 290 pairs of NL-SQL queries. The queries are grouped into one of the following categories depending on the linguistic variation that is used in the NL query: naıve, syntactic paraphrases, morphological paraphrases, and lexical paraphrases as well as a set of queries with missing information.
- QUOTE: Current benchmarks like the GeoQuery benchmark to not explicitly test different linguistic variations which is important to understand the robustness of an NLIDB. For testing different linguistic variants in a principled manner, we therefore curated a new benchmark as part of our paper on DBPal that covers different linguistic variations for the user NL input and maps it to an expected SQL output.