View source for Question answering

{{Distinguish|Query evaluation}}
{{Other uses|Question|Answer (disambiguation){{!}}Answer}}
{{short description|Computer science discipline}}
{{outdated|date=March 2024}}
'''Question answering''' ('''QA''') is a [[computer science]] discipline within the fields of [[information retrieval]] and [[natural language processing]] (NLP) that is concerned with building systems that automatically answer [[Question|questions]] that are posed by humans in a [[natural language]].<ref name="CimianoUnger2014">{{
cite book|author1=Philipp Cimiano|author2=Christina Unger|author3=John McCrae|title=Ontology-Based Interpretation of Natural Language|url=https://books.google.com/books?id=3GA-AwAAQBAJ|date=1 March 2014|publisher=Morgan & Claypool Publishers|isbn=978-1-60845-990-2}}</ref>

==Overview==
A question-answering implementation, usually a computer program, may construct its answers by querying a structured [[database]] of knowledge or information, usually a [[knowledge base]]. More commonly, question-answering systems can pull answers from an unstructured collection of natural language documents.

Some examples of natural language document collections used for question answering systems include:

* a {{clarify|text=local|reason=local to what?|date=April 2023}} collection of reference texts
* internal organization{{ambiguous|reason=adjective could mean either "about organizing" or "pertaining to some organization"|date=April 2023}} documents and web pages
* compiled [[newswire]] reports
* a set of [[Wikipedia]] pages<ref name=":2" />
* a subset of [[World Wide Web]] pages

==Types of question answering==
Question-answering research attempts to develop ways of answering a wide range of question types, including fact, list, [[definition]], how, why, hypothetical, semantically constrained, and cross-lingual questions.
* Answering questions related to an article in order to evaluate [[reading comprehension]] is one of the simpler form of question answering, since a given article is relatively short compared to the domains of other types of question-answering problems. An example of such a question is "What did Albert Einstein win the Nobel Prize for?" after an article about this subject is given to the system.
* ''Closed-book'' question answering is when a system has memorized some facts during training and can answer questions without explicitly being given a context. This is similar to humans taking closed-book exams. 
* ''Closed-domain'' question answering deals with questions under a specific domain (for example, medicine or automotive maintenance) and can exploit domain-specific knowledge frequently formalized in [[ontology (information science)|ontologies]]. Alternatively, "closed-domain" might refer to a situation where only a limited type of questions are accepted, such as questions asking for [[descriptive knowledge|descriptive]] rather than [[procedural knowledge|procedural]] information. Question answering systems {{vague|text=in the context of|date=April 2023}} machine reading applications have also been constructed in the medical domain, for instance {{vague|text=related to|date=April 2023}} Alzheimer's disease.<ref>Roser Morante, Martin Krallinger, Alfonso Valencia and  Walter Daelemans. [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.366.3461 Machine Reading of Biomedical Texts about Alzheimer's Disease]. CLEF 2012 Evaluation Labs and Workshop. September 17, 2012</ref>
* ''[[Open domain#References|Open-domain]]'' question answering deals with questions about nearly anything and can only rely on general ontologies and world knowledge. Systems designed for open-domain question answering usually have much more data available from which to extract the answer. An example of an open-domain question is "What did Albert Einstein win the Nobel Prize for?" while no article about this subject is given to the system.

Another way to categorize question-answering systems is by the technical approach used. There are a number of different types of QA systems, including 
* [[rule-based system]]s, 
* statistical systems, and 
* [[Hybrid intelligent system|hybrid systems]]. 

Rule-based systems use a set of rules to determine the correct answer to a question. Statistical systems use statistical methods to find the most likely answer to a question. Hybrid systems use a combination of rule-based and statistical methods.

==History==
Two early question answering systems were BASEBALL<ref>{{cite journal|last1=GREEN JR|first1=Bert F|title=Baseball: an automatic question-answerer|journal=Western Joint IRE-AIEE-ACM Computer Conference|date=1961|pages=219–224|display-authors=etal|url=https://web.stanford.edu/class/linguist289/p219-green.pdf}}</ref> and LUNAR.<ref>{{cite journal|last1=Woods|first1=William A|last2=Kaplan|first2=R.|title=Lunar rocks in natural English: Explorations in natural language question answering|journal=Linguistic Structures Processing 5|date=1977|volume=5|pages=521–569}}</ref> BASEBALL answered questions about Major League Baseball over a period of one year{{ambiguous|reason=it answered questions for a year, or it answered questions about a year's worth of data, or something else?|date=April 2023}}. LUNAR answered questions about the geological analysis of rocks returned by the Apollo Moon missions. Both question answering systems were very effective in their chosen domains. LUNAR was demonstrated at a lunar science convention in 1971 and it was able to answer 90% of the questions in its domain that were posed by people untrained on the system. Further restricted-domain question answering systems were developed in the following years. The common feature of all these systems is that they had a core database or knowledge system that was hand-written by experts of the chosen domain. The language abilities of BASEBALL and LUNAR used techniques similar to [[ELIZA]] and [[DOCTOR]], the first [[chatterbot]] programs.

[[SHRDLU]] was a successful question-answering program developed by [[Terry Winograd]] in the late 1960s and early 1970s. It simulated the operation of a robot in a toy world (the "blocks world"), and it offered the possibility of asking the robot questions about the state of the world. The strength of this system was the choice of a very specific domain and a very simple world with rules of physics that were easy to encode in a computer program.

In the 1970s, [[knowledge base]]s were developed that targeted narrower domains of knowledge. The question answering systems developed to interface with these [[expert system]]s produced {{clarify|text=more repeatable|date=April 2023}} and valid responses to questions within an area of knowledge. These expert systems closely resembled modern question answering systems except in their internal architecture. Expert systems rely heavily on expert-constructed and organized [[knowledge base]]s, whereas many modern question answering systems rely on statistical processing of a large, unstructured, natural language text corpus.

The 1970s and 1980s saw the development of comprehensive theories in [[computational linguistics]], which led to the development of ambitious projects in text comprehension and question answering. One example was the Unix Consultant (UC), developed by [[Robert Wilensky]] at [[University of California, Berkeley|U.C. Berkeley]] in the late 1980s. The system answered questions pertaining to the [[Unix]] operating system. It had a comprehensive, hand-crafted knowledge base of its domain, and it aimed at phrasing the answer to accommodate various types of users. Another project was LILOG, a [[natural-language understanding|text-understanding]] system that operated on the domain of tourism information in a German city. The systems developed in the UC and LILOG projects never went past the stage of simple demonstrations, but they helped the development of theories on computational linguistics and reasoning.

Specialized natural-language question answering systems have been developed, such as EAGLi for health and life scientists.<ref>{{Cite web|title=EAGLi platform - Question Answering in MEDLINE|url=https://candy.hesge.ch/EAGLi/|access-date=2021-12-02|website=candy.hesge.ch}}</ref>

==Applications==
QA systems are used in a variety of applications, including 
* [[Fact-checking]] if a fact is verified, by posing a question like: is fact ''X'' true or false? 
* customer service, 
* technical support,
* market research,
* [[natural language generation|generating reports]] or conducting research.

==Architecture==
{{As of|2001}}, question-answering systems typically included a ''question classifier'' module that determined the type of question and the type of answer.<ref>Hirschman, L. & Gaizauskas, R. (2001) [http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=96167 Natural Language Question Answering. The View from Here]. Natural Language Engineering (2001), 7:4:275-300 Cambridge University Press.</ref>

Different types of question-answering systems employ different architectures. For example, modern open-domain question answering systems may use a retriever-reader architecture. The retriever is aimed at retrieving relevant documents related to a given question, while the reader is used to infer the answer from the retrieved documents. Systems such as [[GPT-3]], T5,<ref>{{Cite arXiv|title=Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |eprint=1910.10683 |last1=Raffel |first1=Colin |last2=Shazeer |first2=Noam |last3=Roberts |first3=Adam |last4=Lee |first4=Katherine |last5=Narang |first5=Sharan |last6=Matena |first6=Michael |last7=Zhou |first7=Yanqi |last8=Li |first8=Wei |last9=Liu |first9=Peter J. |year=2019 |class=cs.LG }}</ref> and BART<ref>{{Cite arXiv|title=BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension |eprint=1910.13461 |last1=Lewis |first1=Mike |last2=Liu |first2=Yinhan |last3=Goyal |first3=Naman |last4=Ghazvininejad |first4=Marjan |last5=Mohamed |first5=Abdelrahman |last6=Levy |first6=Omer |last7=Stoyanov |first7=Ves |last8=Zettlemoyer |first8=Luke |year=2019 |class=cs.CL }}</ref> use an end-to-end{{jargon inline|date=April 2023}} architecture in which a transformer-based{{jargon inline|date=April 2023}} architecture stores large-scale textual data in the underlying parameters. Such models can answer questions without accessing any external knowledge sources.

==Question answering methods==
Question answering is dependent on a good search [[text corpus|corpus]]; without documents containing the answer, there is little any question answering system can do. Larger collections generally mean better question answering performance, unless the question domain is orthogonal to the collection. [[Data redundancy]] in massive collections, such as the web, means that nuggets of information are likely to be phrased in many different ways in differing contexts and documents,<ref>Lin, J. (2002). [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.457.6371&rep=rep1&type=pdf The Web as a Resource for Question Answering: Perspectives and Challenges]. In ''Proceedings of the Third International Conference on Language Resources and Evaluation'' (LREC 2002).</ref> leading to two benefits:

# If the right information appears in many forms, the question answering system needs to perform fewer complex NLP techniques to understand the text.
# Correct answers can be filtered from [[false positive]]s because the system can rely on versions of the correct answer appearing more times in the corpus than incorrect ones.

Some question answering systems rely heavily on [[automated reasoning]].<ref>Moldovan, Dan, et al. "[http://anthology.aclweb.org/N/N03/N03-1022.pdf Cogex: A logic prover for question answering] {{Webarchive|url=https://web.archive.org/web/20180127222828/http://anthology.aclweb.org/N/N03/N03-1022.pdf |date=2018-01-27 }}." Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics, 2003.</ref><ref>Furbach, Ulrich, Ingo Glöckner, and Björn Pelzer. "[https://web.archive.org/web/20170831043710/https://pdfs.semanticscholar.org/5d8c/a70a0448998162f91e3ce84fac92e10d4ccf.pdf An application of automated reasoning in natural language question answering]." Ai Communications 23.2-3 (2010): 241–265.</ref>

===Open domain question answering===
{{more citations needed section|date=January 2016}}
In [[information retrieval]], an open-domain question answering system tries to return an answer in response to the user's question. The returned answer is in the form of short texts rather than a list of relevant documents.<ref>{{cite book |last1=Sun |first1=Haitian |last2=Dhingra |first2=Bhuwan |last3=Zaheer |first3=Manzil |last4=Mazaitis |first4=Kathryn |last5=Salakhutdinov |first5=Ruslan |last6=Cohen |first6=William |title=  Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing|chapter=Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text |date=2018 |pages=4231–4242 |chapter-url=https://www.aclweb.org/anthology/D18-1455 |location=Brussels, Belgium|doi=10.18653/v1/D18-1455 |arxiv=1809.00782 |s2cid=52154304 }}</ref> The system finds answers by using a combination of techniques from [[computational linguistics]], [[information retrieval]], and [[knowledge representation]].

The system takes a [[natural language]] question as an input rather than a set of keywords, for example: "When is the national day of China?" It then transforms this input sentence into a query in its [[logical form]]. Accepting natural language questions makes the system more user-friendly, but harder to implement, as there are a variety of question types and the system will have to identify the correct one in order to give a sensible answer. Assigning a question type to the question is a crucial task; the entire answer extraction process relies on finding the correct question type and hence the correct answer type.

Keyword [[Data extraction|extraction]] is the first step in identifying the input question type.<ref>{{cite book |last1=Harabagiu |first1=Sanda |last2=Hickl |first2=Andrew |chapter=Methods for using textual entailment in open-domain question answering  |title= Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 |date=2006 |pages=905–912 |doi=10.3115/1220175.1220289 |chapter-url=https://www.aclweb.org/anthology/P06-1114 |doi-access=free }}</ref> In some cases, words clearly indicate the question type, e.g., "Who", "Where", "When", or "How many"—these words might suggest to the system that the answers should be of type "Person", "Location", "Date", or "Number", respectively. [[part-of-speech tagging|POS (part-of-speech) tagging]] and syntactic parsing techniques can also determine the answer type. In the example above, the subject is "Chinese National Day", the predicate is "is" and the adverbial modifier is "when", therefore the answer type is "Date". Unfortunately, some interrogative words like "Which", "What", or "How" do not correspond to unambiguous answer types: Each can represent more than one type. In situations like this, other words in the question need to be considered. A lexical dictionary such as [[WordNet]] can be used for understanding the context.

Once the system identifies the question type, it uses an [[information retrieval]] system to find a set of documents that contain the correct keywords. A [[Part-of-speech tagging|tagger]] and [[Phrase chunking|NP/Verb Group chunker]] can verify whether the correct entities and relations are mentioned in the found documents. For questions such as "Who" or "Where", a [[named-entity recognition|named-entity recogniser]] finds relevant "Person" and "Location" names from the retrieved documents. {{clarify|reason=who selects? who ranks? what does "ranking" mean in this context?|text=Only the relevant paragraphs are selected for ranking.|date=April 2023}}

A [[vector space model]] can classify the candidate answers. Check{{Who|date=April 2023}} if the answer is of the correct type as determined in the question type analysis stage. An inference technique can validate the candidate answers. A score is then given to each of these candidates according to the number of question words it contains and how close these words are to the candidate—the more and the closer the better. The answer is then translated by parsing into a compact and meaningful representation. In the previous example, the expected output answer is "1st Oct."

===Mathematical question answering===

An open-source, math-aware, question answering system called [[MathQA]], based on [[Ask Platypus]] and [[Wikidata]], was published in 2018.<ref name="SchubotzScharpf2018">{{cite journal|author1=Moritz Schubotz|author2=Philipp Scharpf|display-authors=et al.|title=Introducing MathQA: a Math-Aware question answering system|doi=10.1108/IDD-06-2018-0022| arxiv=1907.01642|date=12 September 2018|journal=Information Discovery and Delivery|volume=46|issue=4|pages=214–224|publisher=Emerald Publishing Limited|doi-access=free}}</ref> MathQA takes an English or Hindi natural language question as input and returns a mathematical formula retrieved from Wikidata as a succinct answer, translated into a computable form that allows the user to insert values for the variables. The system retrieves names and values of variables and common constants from Wikidata if those are available. It is claimed that the system outperforms a commercial computational mathematical knowledge engine on a test set.<ref name="SchubotzScharpf2018"/> MathQA is hosted by Wikimedia at https://mathqa.wmflabs.org/. In 2022, it was extended to answer 15 math question types.<ref>Scharpf, P. Schubotz, M. Gipp, B. [https://www.gipp.com/wp-content/papercite-data/pdf/scharpf2022.pdf Mining Mathematical Documents for Question Answering via Unsupervised Formula Labeling] ACM/IEEE Joint Conference on Digital Libraries, 2022.</ref>

MathQA methods need to combine natural and formula language. One possible approach is to perform supervised annotation via [[Entity Linking]]. The "ARQMath Task" at [[CLEF]] 2020<ref name=":1">{{Citation|last1=Zanibbi|first1=Richard|date=2020|url=http://dx.doi.org/10.1007/978-3-030-58219-7_15|pages=169–193|place=Cham|publisher=Springer International Publishing|isbn=978-3-030-58218-0|access-date=2021-06-09|last2=Oard|first2=Douglas W.|last3=Agarwal|first3=Anurag|last4=Mansouri|first4=Behrooz|title=Experimental IR Meets Multilinguality, Multimodality, and Interaction |chapter=Overview of ARQMath 2020: CLEF Lab on Answer Retrieval for Questions on Math |series=Lecture Notes in Computer Science |volume=12260 |doi=10.1007/978-3-030-58219-7_15|s2cid=221351064 |url-access=subscription}}</ref> was launched to address the problem of linking newly posted questions from the platform Math [[Stack Exchange]] to existing ones that were already answered by the community. Providing hyperlinks to already answered, semantically related questions helps users to get answers earlier but is a challenging problem because semantic relatedness is not trivial.<ref name=":0">{{Cite book |last=Scharpf |display-authors=etal |title=ARQMath Lab: An Incubator for Semantic Formula Search in zbMATH Open? |date=2020-12-04 |oclc=1228449497}}</ref> The lab was motivated by the fact that 20% of mathematical queries in general-purpose search engines are expressed as well-formed questions.<ref>{{Cite book|last1=Mansouri|first1=Behrooz|last2=Zanibbi|first2=Richard|last3=Oard|first3=Douglas W.|title=2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL) |chapter=Characterizing Searches for Mathematical Concepts |date=June 2019|chapter-url=http://dx.doi.org/10.1109/jcdl.2019.00019|pages=57–66|publisher=IEEE|doi=10.1109/jcdl.2019.00019|isbn=978-1-7281-1547-4|s2cid=198972305}}</ref> The challenge contained two separate sub-tasks. Task 1: "Answer retrieval" matching old post answers to newly posed questions, and Task 2: "Formula retrieval" matching old post formulae to new questions. Starting with the domain of mathematics, which involves formula language, the goal is to later extend the task to other domains (e.g., STEM disciplines, such as chemistry, biology, etc.), which employ other types of special notation (e.g., chemical formulae).<ref name=":1" /><ref name=":0" />

The inverse of mathematical question answering—mathematical question generation—has also been researched. The PhysWikiQuiz physics question generation and test engine retrieves mathematical formulae from Wikidata together with semantic information about their constituting identifiers (names and values of variables).<ref>{{Cite journal |last1=Scharpf |first1=Philipp |last2=Schubotz |first2=Moritz |last3=Spitz |first3=Andreas |last4=Greiner-Petter |first4=Andre |last5=Gipp |first5=Bela |date=2022 |title=Collaborative and AI-aided Exam Question Generation using Wikidata in Education |doi=10.13140/RG.2.2.30988.18568 |arxiv=2211.08361|s2cid=253270181 }}</ref> The formulae are then rearranged to generate a set of formula variants. Subsequently, the variables are substituted with random values to generate a large number of different questions suitable for individual student tests. PhysWikiquiz is hosted by Wikimedia at https://physwikiquiz.wmflabs.org/.

==Progress==
Question answering systems have been extended in recent{{current event inline|date=April 2023}} years to encompass additional domains of knowledge<ref>{{Cite journal | doi=10.1162/089120105774321055|title = Book Review ''New'' Directions in Question Answering Mark T. Maybury (editor) (MITRE Corporation) Menlo Park, CA: AAAI Press and Cambridge, MA: The MIT Press, 2004, xi+336 pp; paperbound, ISBN 0-262-63304-3, $40.00, £25.95| journal=Computational Linguistics| volume=31| issue=3| pages=413–417|year = 2005|last1 = Paşca|first1 = Marius|s2cid = 12705839|doi-access=free}}</ref> For example, systems have been developed to automatically answer temporal and geospatial questions, questions of definition and terminology, biographical questions, multilingual questions, and questions about the content of audio, images,<ref name="visual question answering"/> and video.<ref>{{cite arXiv | eprint=1511.04670 | last1=Zhu | first1=Linchao | last2=Xu | first2=Zhongwen | last3=Yang | first3=Yi | last4=Hauptmann | first4=Alexander G. | title=Uncovering Temporal Context for Video Question and Answering | year=2015 | class=cs.CV }}</ref> Current question answering research topics include:

* interactivity—clarification of questions or answers{{elucidate|date=April 2023}}<ref>Quarteroni, Silvia, and Suresh Manandhar. "[https://www.researchgate.net/publication/231992433_Designing_an_interactive_open-domain_question_answering_system Designing an interactive open-domain question answering system]." Natural Language Engineering 15.1 (2009): 73–95.</ref>
* answer reuse or caching<ref>Light, Marc, et al. "[https://www.aaai.org/Papers/Symposia/Spring/2003/SS-03-07/SS03-07-016.pdf Reuse in Question Answering: A Preliminary Study]." New Directions in Question Answering. 2003.</ref>
* [[semantic parsing]]<ref>Yih, Wen-tau, Xiaodong He, and Christopher Meek. "[https://www.aclweb.org/anthology/P14-2105 Semantic parsing for single-relation question answering]." Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2014.</ref>
* answer presentation{{elucidate|date=April 2023}}<ref>Perera, R., Nand, P. and Naeem, A. 2017. [https://link.springer.com/article/10.1007/s13748-017-0113-9 Utilizing typed dependency subtree patterns for answer sentence generation in question answering systems.]</ref>
* [[knowledge representation and reasoning|knowledge representation]] and semantic [[entailment (linguistics)|entailment]]<ref>de Salvo Braz, Rodrigo, et al. "[https://www.aaai.org/Papers/AAAI/2005/AAAI05-165.pdf An inference model for semantic entailment in natural language]." Machine Learning Challenges Workshop. Springer, Berlin, Heidelberg, 2005.</ref>
* social media analysis{{elucidate|date=April 2023}} with question answering systems
* [[sentiment analysis]]<ref>{{cite web |url=http://totalgood.com/bitcrawl/ |title=BitCrawl by Hobson Lane |access-date=2012-05-29 |url-status=bot: unknown |archive-url=https://web.archive.org/web/20121027153311/http://totalgood.com/bitcrawl/ |archive-date=October 27, 2012 }}</ref>
* utilization of thematic roles<ref>Perera, R. and Perera, U. 2012. [http://rivinduperera.com/publications/qacd_coling2012.html Towards a thematic role based target identification model for question answering.] {{Webarchive|url=https://web.archive.org/web/20160304111643/http://rivinduperera.com/publications/qacd_coling2012.html |date=2016-03-04 }}</ref>
* [[Image captioning]] for visual question answering<ref name="visual question answering">Anderson, Peter, et al. "[http://openaccess.thecvf.com/content_cvpr_2018/papers/Anderson_Bottom-Up_and_Top-Down_CVPR_2018_paper.pdf Bottom-up and top-down attention for image captioning and visual question answering]." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.</ref>
* [[Embodied agent|Embodied]] question answering<ref>Das, Abhishek, et al. "[https://openaccess.thecvf.com/content_cvpr_2018/papers/Das_Embodied_Question_Answering_CVPR_2018_paper.pdf Embodied question answering]." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.</ref>

In 2011, [[Watson (computer)|Watson]], a question answering computer system developed by [[IBM]], competed in two exhibition matches of ''[[Jeopardy!]]'' against [[Brad Rutter]] and [[Ken Jennings]], winning by a significant margin.<ref>{{Cite news | url=https://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?_r=0 | title=On 'Jeopardy!' Watson Win is All but Trivial| newspaper=The New York Times| date=2011-02-16| last1=Markoff| first1=John}}</ref>
[[Facebook Research]] made their [[DrQA]] system<ref>{{Cite web | url=https://research.fb.com/downloads/drqa/ | title=DrQA}}</ref> available under an [[open source license]]. This system uses [[Wikipedia]] as knowledge source.<ref name=":2">{{cite arXiv| eprint=1704.00051| last1=Chen| first1=Danqi| title=Reading Wikipedia to Answer Open-Domain Questions| last2=Fisch| first2=Adam| last3=Weston| first3=Jason| last4=Bordes| first4=Antoine| class=cs.CL| year=2017}}</ref> The [[open source]] framework Haystack by [[deepset]] combines open-domain question answering with generative question answering and supports the {{clarify|text=domain adaptation|reason=what's "domain adaptation"|date=April 2023}} of the {{clarify|text=underlying|reason=what does it mean for a language model to underlie? what does it underlie?|date=April 2023}} [[Language model|language models]] for {{vague|text=industry use cases|date=April 2023}}.
<ref>{{cite book |last1=Tunstall |first1=Lewis |title=Natural Language Processing with Transformers: Building Language Applications with Hugging Face |date=5 July 2022 |publisher=O'Reilly UK Ltd. |isbn=978-1098136796 |page=Chapter 7 |edition=2nd |url=https://www.oreilly.com/library/view/natural-language-processing/9781098136789/ }}</ref><ref>{{cite web |title=Haystack documentation |url=https://docs.haystack.deepset.ai/docs/intro |publisher=deepset |access-date=4 November 2022}}</ref>

Large Language Models (LLMs)<sup>[[Large language model|[36]]]</sup> like GPT-4<sup>[[GPT-4o|[37]]]</sup>, Gemini<sup>[[Gemini (language model)|[38]]]</sup> are examples of successful QA systems that are enabling more sophisticated understanding and generation of text. When coupled with Multimodal<sup>[[Multimodal learning|[39]]]</sup> QA Systems, which can process and understand information from various modalities like text, images, and audio, LLMs significantly improve the capabilities of QA systems.

==References==
{{reflist|30em}}

==Further reading==
* Dragomir R. Radev, John Prager, and Valerie Samn. [http://clair.si.umich.edu/~radev/papers/anlp00.pdf Ranking suspected answers to natural language questions using predictive annotation] {{Webarchive|url=https://web.archive.org/web/20110826180039/http://clair.si.umich.edu/~radev/papers/anlp00.pdf |date=2011-08-26 }}. In Proceedings of the 6th Conference on Applied Natural Language Processing, Seattle, WA, May 2000.
* John Prager, Eric Brown, Anni Coden, and Dragomir Radev. [http://clair.si.umich.edu/~radev/papers/sigir00.pdf Question-answering by predictive annotation] {{Webarchive|url=https://web.archive.org/web/20110823221004/http://clair.si.umich.edu/~radev/papers/sigir00.pdf |date=2011-08-23 }}. In Proceedings, 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece, July 2000.
*{{cite book | last = Hutchins | first = W. John | author-link = W. John Hutchins | author2 = Harold L. Somers | year = 1992 | title = An Introduction to Machine Translation | url = https://archive.org/details/introductiontoma0000hutc | publisher = Academic Press | location = London | isbn = 978-0-12-362830-5 | url-access = registration }}
* L. Fortnow, Steve Homer (2002/2003).   [http://people.cs.uchicago.edu/~fortnow/papers/history.pdf A Short History of Computational Complexity].  In D. van Dalen, J. Dawson, and A. Kanamori, editors, ''The History of Mathematical Logic''. North-Holland, Amsterdam.
*{{cite book |last1=Tunstall |first1=Lewis |title=Natural Language Processing with Transformers: Building Language Applications with Hugging Face |date=5 July 2022 |publisher=O'Reilly UK Ltd. |isbn=978-1098136796 |page=Chapter 7 |edition=2nd |url=https://www.oreilly.com/library/view/natural-language-processing/9781098136789/ }}

==External links==
* [https://trec.nist.gov/data/qamain.html Question Answering Evaluation at TREC]
* [http://nlp.uned.es/clef-qa/ Question Answering Evaluation at CLEF]

{{Natural Language Processing}}

[[Category:Tasks of natural language processing]]
[[Category:Computational linguistics]]
[[Category:Information retrieval genres]]
[[Category:Deep learning]]