2.19.0.0

Version 2.19.0.0 Release Notes

Compatible with OpenSearch 2.19.0

Features

• Add Support for Multi Values in innerHit for Nested k-NN Fields in Lucene and FAISS #2283
• Add binary index support for Lucene engine. #2292
• Add expand_nested_docs Parameter support to NMSLIB engine #2331
• Add a new build mode, FAISS_OPT_LEVEL=avx512_spr, which enables the use of advanced AVX-512 instructions introduced with Intel[R] Sapphire Rapids #2404
• Add cosine similarity support for faiss engine #2376
• Add concurrency optimizations with native memory graph loading and force eviction #2265
• Add derived source feature for vector fields #2449

Enhancements

• Introduced a writing layer in native engines where relies on the writing interface to process IO. #2241
• Allow method parameter override for training based indices #2290
• Optimizes lucene query execution to prevent unnecessary rewrites #2305
• Added more detailed error messages for KNN model training #2378
• Add check to directly use ANN Search when filters match all docs. #2320
• Use one formula to calculate cosine similarity #2357
• Make the build work for M series MacOS without manual code changes and local JAVA_HOME config #2397
• Remove DocsWithFieldSet reference from NativeEngineFieldVectorsWriter #2408
• Remove skip building graph check for quantization use case #2430
• Removing redundant type conversions for script scoring for hamming space with binary vectors #2351
• Update default to 0 to always build graph as default behavior #2452
• Enabled concurrent graph creation for Lucene engine with index thread qty settings#2480

Bug Fixes

• Fixing the bug when a segment has no vector field present for disk based vector search #2282
• Fixing the bug where search fails with "fields" parameter for an index with a knn_vector field #2314
• Fix for NPE while merging segments after all the vector fields docs are deleted #2365
• Allow validation for non knn index only after 2.17.0 #2315
• Fixing the bug to prevent updating the index.knn setting after index creation#2348
• Release query vector memory after execution #2346
• Fix shard level rescoring disabled setting flag #2352
• Fix filter rewrite logic which was resulting in getting inconsistent / incorrect results for cases where filter was getting rewritten for shards #2359
• Fixing it to retrieve space_type from index setting when both method and top level don't have the value. #2374
• Fixing the bug where setting rescore as false for on_disk knn_vector query is a no-op #2399
• Fixing the bug to prevent index.knn setting from being modified or removed on restore snapshot #2445
• Fix Faiss byte vector efficient filter bug #2448
• Fixing bug where mapping accepts both dimension and model-id #2410
• Add version check for full field name validation #2477
• Update engine for version 2.19 or above #2501

Infrastructure

• Updated C++ version in JNI from c++11 to c++17 #2259
• Upgrade bytebuddy and objenesis version to match OpenSearch core and, update github ci runner for macos #2279

Documentation

Maintenance

• Select index settings based on cluster version2236
• Added periodic cache maintenance for QuantizationStateCache and NativeMemoryCache #2308
• Added null checks for fieldInfo in ExactSearcher to avoid NPE while running exact search for segments with no vector field #2278
• Added Lucene BWC tests #2313
• Upgrade jsonpath from 2.8.0 to 2.9.02325
• Bump Faiss commit from 1f42e81 to 0cbc2a8 to accelerate hamming distance calculation using _mm512_popcnt_epi64 intrinsic and also add avx512-fp16 instructions to boost performance #2381
• Deprecate nmslib engine #2427
• Add spotless mirror repo for fixing builds #2453

2.18.0.0

Version 2.18.0.0 Release Notes

Compatible with OpenSearch 2.18.0

Features

• Add AVX512 support to k-NN for FAISS library #2069

Enhancements

• Introducing a loading layer in FAISS #2033
• Add short circuit if no live docs are in segments #2059
• Optimize reduceToTopK in ResultUtil by removing pre-filling and reducing peek calls #2146
• Update Default Rescore Context based on Dimension #2149
• KNNIterators should support with and without filters #2155
• Adding Support to Enable/Disble Share level Rescoring and Update Oversampling Factor#2172
• Add support to build vector data structures greedily and perform exact search when there are no engine files #1942
• Add CompressionLevel Calculation for PQ #2200
• Remove FSDirectory dependency from native engine constructing side and deprecated FileWatcher #2182
• Update approximate_threshold to 15K documents #2229
• Update default engine to FAISS #2221

Bug Fixes

• Add DocValuesProducers for releasing memory when close index #1946
• KNN80DocValues should only be considered for BinaryDocValues fields #2147
• Score Fix for Binary Quantized Vector and Setting Default value in case of shard level rescoring is disabled for oversampling factor#2183
• Java Docs Fix For 2.x#2190

Documentation

• Fix sed command in DEVELOPER_GUIDE.md to append a new line character '\n'. #2181

Maintenance

• Remove benchmarks folder from k-NN repo #2127
• Fix lucene codec after lucene version bumped to 9.12. #2195

Refactoring

• Does not create additional KNNVectorValues in NativeEngines990KNNVectorWriter when quantization is not needed #2133
• Minor refactoring and refactored some unit test #2167

2.17.0.0

Version 2.17.0.0 Release Notes

Compatible with OpenSearch 2.17.0

Features

• Integrate Lucene Vector field with native engines to use KNNVectorFormat during segment creation #1945
• k-NN query rescore support for native engines #1984
• Add support for byte vector with Faiss Engine HNSW algorithm #1823
• Add support for byte vector with Faiss Engine IVF algorithm #2002
• Add mode/compression configuration support for disk-based vector search #2034
• Add spaceType as a top level optional parameter while creating vector field. #2044

Enhancements

• Adds iterative graph build capability into a faiss index to improve the memory footprint during indexing and Integrates KNNVectorsFormat for native engines#1950
• Add model version to model metadata and change model metadata reads to be from cluster metadata #2005

Bug Fixes

• Corrected search logic for scenario with non-existent fields in filter #1874
• Add script_fields context to KNNAllowlist [#1917] (#1917)
• Fix graph merge stats size calculation #1844
• Disallow a vector field to have an invalid character for a physical file name. #1936
• Fix memory overflow caused by cache behavior #2015
• Use correct type for binary vector in ivf training #2086
• Switch MINGW32 to MINGW64 #2090

Infrastructure

• Parallelize make to reduce build time [#2006] (#2006)

Maintenance

• Fix a flaky unit test:testMultiFieldsKnnIndex, which was failing due to inconsistent merge behaviors #1924

Refactoring

• Introduce KNNVectorValues interface to iterate on different types of Vector values during indexing and search #1897
• Integrate KNNVectorValues with vector ANN Search flow #1952
• Clean up parsing for query #1824
• Refactor engine package structure #1913
• Refactor method structure and definitions #1920
• Refactor KNNVectorFieldType from KNNVectorFieldMapper to a separate class for better readability. #1931
• Generalize lib interface to return context objects #1925
• Restructure mappers to better handle null cases and avoid branching in parsing #1939
• Added Quantization Framework and implemented 1Bit and multibit quantizer#1889
• Encapsulate dimension, vector data type validation/processing inside Library #1957
• Add quantization state cache #1960
• Add quantization state reader and writer #1997

2.16.0.0

Version 2.16.0.0 Release Notes

Compatible with OpenSearch 2.16.0

Features

• Adds dynamic query parameter ef_search #1783
• Adds dynamic query parameter ef_search in radial search faiss engine #1790
• Adds dynamic query parameter nprobes #1792
• Add binary format support with HNSW method in Faiss Engine #1781
• Add script scoring support for knn field with binary data type #1826
• Add painless script support for hamming with binary vector data type #1839
• Add binary format support with IVF method in Faiss Engine #1784
• Add support for Lucene inbuilt Scalar Quantizer #1848

Enhancements

• Switch from byte stream to byte ref for serde #1825

Bug Fixes

• Fixing the arithmetic to find the number of vectors to stream from java to jni layer.#1804
• Fixed LeafReaders casting errors to SegmentReaders when segment replication is enabled during search.#1808
• Release memory properly for an array type #1820
• FIX Same Suffix Cause Recall Drop to zero #1802

Infrastructure

• Apply custom patch only once by comparing the last patch id #1833

Documentation

• Update dev guide to fix clang linking issue on arm #1746

Maintenance

• Bump faiss commit to 33c0ba5 #1796

2.15.0.0

Version 2.15.0.0 Release Notes

Compatible with OpenSearch 2.15.0

Features

• Use the Lucene Distance Calculation Function in Script Scoring for doing exact search #1699

Enhancements

• Add KnnCircuitBreakerException and modify exception message #1688
• Add stats for radial search #1684
• Support script score when doc value is disabled and fix misusing DISI #1696
• Add validation for pq m parameter before training starts #1713
• Block delete model requests if an index uses the model #1722

Bug Fixes

• Block commas in model description #1692
• Update threshold value after new result is added #1715

2.14.0.0

Version 2.14.0.0 Release Notes

Compatible with OpenSearch 2.14.0

Features

• Add k-NN clear cache api #740
• Support radial search in k-NN plugin #1617
• Support filter and nested field in faiss engine radial search #1652

Enhancements

• Make the HitQueue size more appropriate for exact search #1549
• Implement the Streaming Feature to stream vectors from Java to JNI layer to enable creation of larger segments for vector indices #1604
• Remove unnecessary toString conversion of vector field and added some minor optimization in KNNCodec 1613
• Serialize all models into cluster metadata #1499

Bug Fixes

• Add stored fields for knn_vector type #1630
• Enable script score to work with model based indices #1649

Infrastructure

• Add micro-benchmark module in k-NN plugin for benchmark streaming vectors to JNI layer functionality. #1583
• Add arm64 check when SIMD is disabled #1618
• Skip rebuild from scratch after cmake is run #1636

2.13.0.0

Version 2.13.0.0 Release Notes

Compatible with OpenSearch 2.13.0

Enhancements

• Optize Faiss Query With Filters: Reduce iteration and memory for id filter #1402
• Detect AVX2 Dynamically on the System #1502
• Validate zero vector when using cosine metric #1501
• Persist model definition in model metadata [#1527] (#1527)
• Added Inner Product Space type support for Lucene Engine #1551
• Add Range Validation for Faiss SQFP16 #1493
• SQFP16 Range Validation for Faiss IVF Models #1557

Bug Fixes

• Disable sdc table for HNSWPQ read-only indices #1518
• Switch SpaceType.INNERPRODUCT's vector similarity function to MAXIMUM_INNER_PRODUCT #1532
• Add patch to fix arm segfault in nmslib during ingestion #1541
• Share ivfpq-l2 table allocations across indices on load #1558

Infrastructure

• Manually install zlib for win CI #1513
• Update k-NN build artifact script to enable SIMD on ARM for Faiss #1543

Maintenance

• Bump faiss lib commit to 32f0e8cf92cd2275b60364517bb1cce67aa29a55 #1443
• Fix FieldInfo Parameters Mismatch #1490
• Upgrade faiss to 12b92e9 #1509

2.12.0.0

Version 2.12.0.0 Release Notes

Compatible with OpenSearch 2.12.0

Features

• Add parent join support for lucene knn #1182
• Add parent join support for faiss hnsw #1398

Enhancements

• Increase Lucene max dimension limit to 16,000 #1346
• Tuned default values for ef_search and ef_construction for better indexing and search performance for vector search #1353
• Enabled Filtering on Nested Vector fields with top level filters #1372
• Throw proper exception to invalid k-NN query #1380

Bug Fixes

• Fix use-after-free case on nmslib search path #1305
• Allow nested knn field mapping when train model #1318
• Properly designate model state for actively training models when nodes crash or leave cluster #1317
• Fix script score queries not getting cached #1367
• Fix KNNScorer to apply boost #1403
• Fix equals and hashCode methods for KNNQuery and KNNQueryBuilder #1397
• Pass correct value on IDSelectorBitmap initialization #1444

Infrastructure

• Upgrade gradle to 8.4 1289
• Refactor security testing to install from individual components #1307
• Refactor integ tests that access model index #1423
• Fix flaky model tests #1429

Maintenance

• Update developer guide to include M1 Setup #1222
• Upgrade urllib to 1.26.17 #1278
• Upgrade urllib to 1.26.18 #1319
• Upgrade guava to 32.1.3 #1319
• Bump lucene codec to 99 #1383
• Update spotless and eclipse dependencies #1450

2.11.0.0

Version 2.11.0.0 Release Notes

Compatible with OpenSearch 2.11.0

Enhancements

• Added support for ignore_unmapped in KNN queries. #1071
• Add graph creation stats to the KNNStats API. #1141

Maintenance

• Update bytebuddy to 1.14.7 #1135

2.10.0.0

Version 2.10.0.0 Release Notes

Compatible with OpenSearch 2.10.0

Features

• Add Clear Cache API (#740)

Enhancements

• Enabled the IVF algorithm to work with Filters of K-NN Query. (#1013)
• Improved the logic to switch to exact search for restrictive filters search for better recall. (#1059)
• Added max distance computation logic to enhance the switch to exact search in filtered Nearest Neighbor Search. (#1066)

Bug Fixes

• Update Faiss parameter construction to allow HNSW+PQ to work (#1074)

Maintenance

• Update Guava Version to 32.0.1 (#1019)

Refactoring

• Fix TransportAddress Refactoring Changes in Core (#1020)