New Framework Improves AI Question Answering in Industries
Researchers at Chinese academic institutions have developed a new framework to boost the performance of large language models when answering technical questions.
In industries like manufacturing, equipment maintenance, and fault diagnosis, experts rely heavily on knowledge-based inquiries. However, current LLMs often produce fluent but incorrect outputs when handling specialized content – a problem these researchers set out to solve.
The new framework is designed specifically for accurately answering industrial-related questions. To achieve this goal, it’s built around four core technical contributions: creating an industry-specific knowledge base, improving text classification, enhancing knowledge retrieval, and refining question-answering capabilities.
One key component of the framework involves building a massive database from various sources like journal articles, dissertations, patents, and production documents. This data is cleaned up, processed, and organized to retain its meaning in segments no longer than 256 characters – long enough for most search queries.
The unified knowledge base is then split into five independent sub-databases for chemical engineering, petroleum, metallurgy, electricity, and building materials sectors. By partitioning the data this way, searches can focus on relevant areas, leading to more accurate results in industries that need precision.
A BERT-based text classifier acts as a filter within the framework’s modules. It performs two classification tasks: screening out non-industrial questions and assigning valid ones to their matching sub-domains based on semantic features extracted from token embeddings – essentially categorizing queries by relevance.
Tests show that this BERT classifier delivers consistent performance with low result variance across repeated training runs, which is crucial for knowledge-based applications like question answering in industry settings. Its strength lies in its ability to handle diverse industrial queries effectively and efficiently.
For knowledge retrieval, the researchers employ a GTE-DPR dual-encoder dense retrieval architecture using the GTE-large-zh embedding model. This system generates 1024-dimensional text vectors that allow it to match query embeddings with precomputed chunk embeddings quickly – making relevant information easier to locate within large databases.
According to metrics, the Domain + GTE–DPR setup outperforms general-purpose DPR and domain-only DPR alternatives in semantic matching precision for industrial text. Its consistent results across test trials demonstrate its effectiveness in retrieving accurate knowledge chunks from a large-scale industrial base – making it stand out among competitors.
After filtering and ranking relevant knowledge chunks, the framework combines them with original user prompts through prompt engineering before passing the information to two open-source LLMs: ChatGLM2-6B and LLaMA2-7B. These models generate domain-grounded answers based on validated documents, reducing reliance on incomplete or inaccurate information – a critical improvement in AI tools for businesses.
Experiments have shown that the modular RAG system outperforms standalone LLM inference in question answering tasks when properly aligned with human evaluation standards of answer quality. This advancement comes from carefully integrating various components rather than relying solely on one model’s capabilities.
Despite its improvements, however, the framework still faces challenges posed by niche industrial terminology and misalignment between current metrics and the nuanced understanding humans use to judge the accuracy of an AI-generated response – a limitation that future versions aim to address.