bug: bm25_query_to_svector has an additional norm

[kemingy]

pg_bestmatch.rs/src/lib.rs

Lines 216 to 221 in 3126173

	// https://github.com/pinecone-io/pinecone-text/issues/69
	let sum = x.values().copied().sum::<f32>();
	let mut result = "{".to_string();
	for (index, value) in x.into_iter() {
	result.push_str(&format!("{}:{}, ", index + 1, value / sum));
	}

This is for:

$$score(D, Q) = \sum^{D\land Q}_t (TF_{norm} * IDF /\sum^Q_t IDF)$$

But if the score (BM25) is pre-computed on the document level, it will become:

$$score(D, Q) = \sum^{D\land Q}_t (TF_{norm} * IDF /\sum^D_t IDF)$$

This means that for each query, the score is not only affected by token $\in D \land Q$, but also token $\in D \notin Q$.

solutions

1. create another function that computes the BM25 score directly instead of using the element-wise multiplication of document/query sparse vector
2. or delete the query IDF norm