feat(oss-opensearch): Add Scalar Quantization support

[Akhil-Pathivada]

Summary

Adds support for Lucene Scalar Quantization (SQ) and FAISS 16-bit Scalar Quantization (FP16) for OSS OpenSearch, enabling users to reduce memory usage for in-memory vector indexes.

Changes

Backend

• Refactored quantization enum: fp32/fp16 → None/LuceneSQ/FaissSQfp16
• Added new fields:
  • confidence_interval (float, optional): For Lucene SQ quantile calculation
  • clip (bool, optional): For FAISS FP16 out-of-range value handling
• Implemented validator: Converts UI/CLI string values to enum with backward compatibility
• Updated encoder logic: Engine-aware configuration for Lucene SQ and FAISS FP16

Frontend

• Engine-aware UI: Separate quantization dropdowns for Lucene and FAISS engines
  • Lucene: ["None", "LuceneSQ"]
  • FAISS: ["None", "FaissSQfp16"]
• Progressive disclosure: Optional parameters appear only when relevant
  • confidence_interval shown for Lucene SQ
  • clip shown for FAISS FP16
• Prevents invalid combinations: UI logic ensures engine/quantization compatibility

CLI

• Updated options: --quantization-type now accepts None, LuceneSQ, FaissSQfp16
• New parameters: --confidence-interval and --clip for fine-tuning

Technical Details

Lucene SQ

• Converts 32-bit float vectors to 7-bit integers (4x memory reduction)
• Optional confidence_interval (0.0-1.0) controls quantile calculation
• OpenSearch API: encoder: {name: "sq", parameters: {confidence_interval: <value>}}

FAISS FP16

• Converts 32-bit float vectors to 16-bit floats (2x memory reduction)
• Optional clip parameter handles out-of-range values ([-65504, 65504])
• OpenSearch API: encoder: {name: "sq", parameters: {type: "fp16", clip: true}}

Testing

• ✅ All linter checks passed
• ✅ Backward compatible with existing configs
• ✅ Engine-aware UI prevents invalid configurations
• ✅ CLI and UI both functional

Screenshots

[Akhil-Pathivada]

/assign @alwayslove2013

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Akhil-Pathivada, alwayslove2013
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment