feat(server): faster reranking based on confidence (#12957)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Improved document reranking with a more streamlined and accurate
scoring system.
* Enhanced support for binary ("yes"/"no") document relevance judgments.

* **Improvements**
* Simplified user prompts and output formats for reranking tasks, making
results easier to interpret.
  * Increased reliability and consistency in document ranking results.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
DarkSky
2025-06-28 11:41:53 +08:00
committed by GitHub
parent e6f91cced6
commit 9b881eb59a
5 changed files with 81 additions and 80 deletions
@@ -342,57 +342,11 @@ Convert a multi-speaker audio recording into a structured JSON format by transcr
messages: [
{
role: 'system',
content: `Evaluate and rank search results based on their relevance and quality to the given query by assigning a score from 1 to 10, where 10 denotes the highest relevance.
Consider various factors such as content alignment with the query, source credibility, timeliness, and user intent.
# Steps
1. **Read the Query**: Understand the main intent and specific details of the search query.
2. **Review Each Result**:
- Analyze the content's relevance to the query.
- Assess the credibility of the source or website.
- Consider the timeliness of the information, ensuring it's current and relevant.
- Evaluate the alignment with potential user intent based on the query.
3. **Scoring**:
- Assign a score from 1 to 10 based on the overall relevance and quality, with 10 being the most relevant.
- Each chunk returns a score and should not be mixed together.
# Output Format
Return a JSON object for each result in the following format in raw:
{
"scores": [
{
"reason": "[Reasoning behind the score in 20 words]",
"chunk": "[chunk]",
"targetId": "[targetId]",
"score": [1-10]
}
]
}
# Notes
- Be aware of the potential biases or inaccuracies in the sources.
- Consider if the content is comprehensive and directly answers the query.
- Pay attention to the nuances of user intent that might influence relevance.`,
content: `Judge whether the Document meets the requirements based on the Query and the Instruct provided. The answer must be "yes" or "no".`,
},
{
role: 'user',
content: `
<query>{{query}}</query>
<results>
{{#results}}
<result>
<targetId>{{targetId}}</targetId>
<chunk>{{chunk}}</chunk>
<content>
{{content}}
</content>
</result>
{{/results}}
</results>`,
content: `<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: {query}\n<Document>: {doc}`,
},
],
},