From 646ce9271a5c1c88a23b1f71e7cd0cac78f20271 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 16 Jan 2026 09:10:13 +0800 Subject: [PATCH 1/8] feat: added doc for migrate es to os --- ...igrate_from_Elasticsearch_to_OpenSearch.md | 510 ++++++++++++++++++ 1 file changed, 510 insertions(+) create mode 100644 docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md new file mode 100644 index 0000000..6f858e6 --- /dev/null +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -0,0 +1,510 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +--- + +# How to Migrate from Elasticsearch to OpenSearch + +:::info +Applicable Version: OpenSearch Operator ~= 2.8.*, OpenSearch ~= 2.x / 3.x +::: + +This document provides detailed guidance for migrating from Elasticsearch (ES) to OpenSearch, covering two primary scenarios: + +1. **Elasticsearch 7.10 → OpenSearch 2.x → OpenSearch 3.x**: Using Snapshot & Restore (two-phase migration) +2. **Elasticsearch 8.x/9.x → OpenSearch 3.x**: Using Reindex from Remote + +It also includes client migration guides for Java, Python, and Golang. + +## Migration Strategy Overview + +| Source Version | Target Version | Migration Method | Notes | +| :--- | :--- | :--- | :--- | +| **ES 7.10** | **OS 2.x** | Snapshot & Restore | ✅ Direct restore supported | +| **ES 7.10** | **OS 3.x** | Snapshot & Restore → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | +| **ES 8.x/9.x** | **OS 3.x** | Reindex from Remote | ✅ Direct migration supported | + +:::warning Key Compatibility Note + +- **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. +- ES 7.10 snapshots must be restored to OpenSearch 2.x first, then upgrade the cluster to OS 3.x. +- ES 8.x/9.x use incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. + +::: + +### Migration Path Diagram + +```mermaid +flowchart LR + ES710["ES 7.10"] -->|"Snapshot & Restore"| OS2["OS 2.x"] -->|"Upgrade"| OS3a["OS 3.x"] + ES8["ES 8.x/9.x"] -->|"Reindex from Remote"| OS3b["OS 3.x"] +``` + +## Migrate from ES 7.10 to OpenSearch 2.x to 3.x + +This migration requires a **two-phase approach**: + +1. **Phase 1**: Restore ES 7.10 snapshot to OpenSearch 2.x +2. **Phase 2**: Upgrade OpenSearch 2.x to 3.x + +### Prerequisites + +- A shared storage backend (e.g., S3 Bucket, GCS Bucket, or NFS) accessible by both source and target clusters. +- The `repository-s3` plugin (or corresponding storage backend plugin) installed on both clusters. + +#### Check if Plugin is Installed + +```bash +curl -X GET "http://localhost:9200/_cat/plugins?v" -u ":" +``` + +#### Install repository-s3 Plugin + +Plugin download URLs: + +| Version | Download URL | +| :--- | :--- | +| ES 7.10.2 | `https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.10.2.zip` | +| OpenSearch 2.19.3 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip` | +| OpenSearch 3.3.1 | `https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip` | + +:::note +The plugin version **must exactly match** the Elasticsearch/OpenSearch version. For example, OpenSearch 3.3.1 requires `repository-s3-3.3.1.zip`. +::: + +:::warning Air-Gapped Environments + +If your Kubernetes cluster does not have external network access, download the plugin zip files first and host them on an internal HTTP server (e.g., Nexus, Artifactory, or Nginx). Then replace the download URLs in the configurations below with your internal accessible URLs. + +::: + +**ES 7.10 (Helm Chart):** + +In **Application Container Platform** > **Applications** > **Applications** page: + +- Find the elasticsearch instance +- Click **Update** +- Switch to **YAML** edit page + +Update `values.yaml` on the **Custom** input textarea with below content: + +```yaml +masterNodes: + config: + elasticsearch.yml: | + s3.client.default.endpoint: "http://minio.example.com:9000" + s3.client.default.region: "us-east-1" + s3.client.default.path_style_access: true # Required for MinIO + +extraInitContainers: + - name: install-plugins + image: harbor.alauda.cn/middleware/elasticsearch:v7.10.2 + command: + - sh + - -c + - | + bin/elasticsearch-plugin install --batch https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.10.2.zip + volumeMounts: + - name: plugins + mountPath: /usr/share/elasticsearch/plugins + +extraVolumes: + - name: plugins + emptyDir: {} + +extraVolumeMounts: + - name: plugins + mountPath: /usr/share/elasticsearch/plugins +``` + +**OpenSearch:** + +In your `OpenSearchCluster` CR: + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: my-cluster +spec: + bootstrap: + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip + general: + additionalConfig: + s3.client.default.endpoint: "http://minio.example.com:9000" + s3.client.default.region: "us-east-1" + s3.client.default.path_style_access: "true" + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip +``` + +:::note +All three approaches will trigger a rolling restart of nodes to load the newly installed plugin. +::: + +### Procedure + +#### Step 1: Configure S3 Credentials + +For security reasons, avoid including access keys directly in API request bodies. Use the keystore instead. + +**On Elasticsearch 7.10:** + +1. Add S3 credentials to keystore (secure settings): + + ```bash + bin/elasticsearch-keystore add s3.client.default.access_key + bin/elasticsearch-keystore add s3.client.default.secret_key + ``` + + Or use non-interactive mode: + + ```bash + echo "YOUR_ACCESS_KEY" | bin/elasticsearch-keystore add --stdin s3.client.default.access_key + echo "YOUR_SECRET_KEY" | bin/elasticsearch-keystore add --stdin s3.client.default.secret_key + ``` + +2. Reload the secure settings: + + ```bash + curl -X POST "http://localhost:9200/_nodes/reload_secure_settings" + ``` + +**On OpenSearch:** + +Use the Operator's declarative configuration: + +1. Create a Secret containing the credentials and endpoint: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: s3-secret + stringData: + s3.client.default.access_key: "YOUR_ACCESS_KEY" + s3.client.default.secret_key: "YOUR_SECRET_KEY" + ``` + + :::note S3 Endpoint Configuration + + - For AWS S3: Omit the `endpoint` field, or set it to `s3.amazonaws.com` + - For S3-compatible services (MinIO, Ceph, etc.): Set the endpoint to your server address + - For path-style access: Add `s3.client.default.path_style_access: "true"` (required for MinIO) + + ::: + +2. Reference the Secret in the `OpenSearchCluster` CR: + + ```yaml + spec: + general: + keystore: + - secret: + name: s3-secret + ``` + + > The Operator will automatically mount the secret and reload the secure settings. + +#### Step 2: Register Snapshot Repository on Source Cluster (ES 7.10) + +```bash +curl -X PUT "http://localhost:9200/_snapshot/migration_repo" -H 'Content-Type: application/json' -d' +{ + "type": "s3", + "settings": { + "bucket": "my-migration-bucket", + "base_path": "es_710_backup" + } +}' +``` + +#### Step 3: Create a Full Snapshot + +```bash +curl -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" -H 'Content-Type: application/json' -d' +{ + "indices": "*", + "ignore_unavailable": true, + "include_global_state": true +}' +``` + +### Phase 1: Restore to OpenSearch 2.x + +#### Step 4: Deploy OpenSearch 2.x Cluster + +Deploy a new OpenSearch **2.x** cluster using the OpenSearch Operator: + +```yaml +apiVersion: opensearch.opster.io/v1 +kind: OpenSearchCluster +metadata: + name: my-cluster +spec: + general: + version: 2.19.3 + additionalConfig: + s3.client.default.endpoint: "http://minio.example.com:9000" + s3.client.default.region: "us-east-1" + s3.client.default.path_style_access: "true" + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/2.19.3/repository-s3-2.19.3.zip + keystore: + - secret: + name: s3-secret + snapshotRepositories: + - name: migration_repo + type: s3 + settings: + bucket: my-migration-bucket + base_path: es_710_backup + readonly: "true" +``` + +#### Step 5: Restore the Snapshot + +Exclude system indices to avoid conflicts with OpenSearch's internal indices: + +```bash +curl -k -X POST "https://localhost:9200/_snapshot/migration_repo/snapshot_1/_restore" \ + -u "admin:admin" \ + -H 'Content-Type: application/json' -d' +{ + "indices": "-.kibana*,-.security*,-.monitoring*", + "include_global_state": false +}' +``` + +#### Step 6: Verification + +Verify the index count and document count match the source cluster: + +```bash +# Check indices +curl -X GET "https://localhost:9200/_cat/indices?v" -u "admin:admin" -k + +# Check document count +curl -X GET "https://localhost:9200//_count" -u "admin:admin" -k +``` + +### Phase 2: Reindex and Upgrade to OpenSearch 3.x + +:::warning Critical Step +Indices restored from ES 7.10 snapshots retain their original version metadata (`7.10.2`). OpenSearch 3.x requires indices to have version `2.0.0+`. You **MUST reindex** all restored indices within OpenSearch 2.x before upgrading. +::: + +#### Step 7: Reindex All Restored Indices + +For each restored index, create a new index and reindex the data: + +```bash +# 1. Get the original index mapping and extract the mappings object using sed +curl -s -X GET "https://localhost:9200/migration_test/_mapping" -u "admin:admin" -k | \ + sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json + +# 2. Create a new index with the same mapping (add suffix _v2) +curl -X PUT "https://localhost:9200/migration_test_v2" \ + -u "admin:admin" -k \ + -H 'Content-Type: application/json' \ + -d @mapping.json + +# 3. Reindex data from old index to new index +curl -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ + -u "admin:admin" -k \ + -H 'Content-Type: application/json' -d' +{ + "source": { "index": "migration_test" }, + "dest": { "index": "migration_test_v2" } +}' + +# 4. Delete old index and create alias (or rename) +curl -X DELETE "https://localhost:9200/migration_test" -u "admin:admin" -k +curl -X POST "https://localhost:9200/_aliases" \ + -u "admin:admin" -k \ + -H 'Content-Type: application/json' -d' +{ + "actions": [ + { "add": { "index": "migration_test_v2", "alias": "migration_test" } } + ] +}' +``` + +Repeat for all restored indices. After reindexing, verify the new index version: + +```bash +curl -X GET "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" -u "admin:admin" -k +``` + +The `version.created` should show an OpenSearch 2.x internal version number (e.g., `136408127` for OS 2.19.x). ES 7.10.2 indices show `7102099`. If you see a number starting with `136` or higher, the reindex was successful. + +#### Step 8: Upgrade OpenSearch Cluster + +Update the `OpenSearchCluster` CR to upgrade the version: + +```yaml +spec: + general: + version: 3.3.1 # Upgrade to OpenSearch 3.x + pluginsList: + - https://artifacts.opensearch.org/releases/plugins/repository-s3/3.3.1/repository-s3-3.3.1.zip +``` + +The Operator will perform a rolling upgrade automatically. + +#### Step 9: Post-Upgrade Verification + +Verify all indices are accessible after upgrade: + +```bash +curl -X GET "https://localhost:9200/_cat/indices?v" -u "admin:admin" -k +curl -X GET "https://localhost:9200/_cluster/health?pretty" -u "admin:admin" -k +``` + +## Migrate from ES 8.x/9.x to OpenSearch 3.x + +Elasticsearch 8.x uses a newer Lucene version with incompatible metadata protocols, making snapshots unreadable by OpenSearch. Use **Reindex from Remote** instead. + +### Prerequisites + +- **Network Connectivity**: The OpenSearch cluster must be able to reach the ES 8.x cluster's HTTP/REST port (typically 9200). + +### Deploy ES 8.x Using ECK Operator + +Deploy an Elasticsearch 8.x cluster using ECK Operator: + +```yaml +apiVersion: elasticsearch.k8s.elastic.co/v1 +kind: Elasticsearch +metadata: + name: es-cluster +spec: + http: + service: + spec: + type: NodePort + version: 8.17.5 + nodeSets: + - name: default + count: 3 + config: + podTemplate: + spec: + containers: + - name: elasticsearch + resources: + limits: + cpu: "2" + memory: 4Gi + requests: + cpu: "1" + memory: 4Gi + volumeClaimTemplates: + - metadata: + name: elasticsearch-data + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +### Procedure + +#### Step 1: Configure the Remote Whitelist in OpenSearch + +Modify `opensearch.yml` (via Helm ConfigMap or Operator configuration) to allow connections to the ES 8.x host: + +```yaml +reindex.remote.whitelist: ["es8-cluster-host:9200"] +``` + +> **Note**: OpenSearch nodes must be restarted after this configuration change. + +#### Step 2: Create Index Templates (Optional but Recommended) + +If your ES 8.x indices rely on specific settings or mappings, it is recommended to manually create the corresponding Index Templates or Mappings in OpenSearch beforehand. + +#### Step 3: Execute Reindex (Asynchronous Mode) + +Initiate the reindex request from the OpenSearch cluster. Set `wait_for_completion=false` to run asynchronously. + +```bash +curl -X POST "http://localhost:9200/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d' +{ + "source": { + "remote": { + "host": "http://es8-cluster-host:9200", + "username": "elastic", + "password": "password" + }, + "index": "my-index-001" + }, + "dest": { + "index": "my-index-001" + } +}' +``` + +**Example Response:** + +```json +{ + "task": "N6q0j8s-T0m0j8s-T0m0j8:123456" +} +``` + +#### Step 4: Monitor Reindex Progress + +Use the Task ID from the previous step to check the task status: + +```bash +curl -X GET "http://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" +``` + +You can also view all reindex tasks: + +```bash +curl -X GET "http://localhost:9200/_tasks?actions=*reindex&detailed=true" +``` + +> Monitor the `response.created` (documents created) and `response.total` (total documents) fields in the response. + +## Client Migration Guide + +Regardless of the source ES version, **it is strongly recommended to switch to the official OpenSearch clients**. + +:::warning Compatibility Note + +- Elasticsearch OSS 7.10.2 clients may work with OpenSearch 1.x, but latest ES clients include license/version checks that break compatibility. +- **For OpenSearch 2.0 and later, no Elasticsearch clients are fully compatible with OpenSearch.** +- Using OpenSearch clients for OpenSearch clusters is strongly recommended. +::: + +### OpenSearch Official Clients + +| Language | Client | Documentation | +| :--- | :--- | :--- | +| **Python** | opensearch-py | [High-level](https://docs.opensearch.org/latest/clients/python-high-level/), [Low-level](https://docs.opensearch.org/latest/clients/python-low-level/) | +| **Java** | opensearch-java | [Java Client](https://docs.opensearch.org/latest/clients/java/) | +| **JavaScript** | @opensearch-project/opensearch | [Node.js Client](https://docs.opensearch.org/latest/clients/javascript/index) | +| **Go** | opensearch-go | [Go Client](https://docs.opensearch.org/latest/clients/go/) | +| **Ruby** | opensearch-ruby | [Ruby Client](https://docs.opensearch.org/latest/clients/ruby/) | +| **PHP** | opensearch-php | [PHP Client](https://docs.opensearch.org/latest/clients/php/) | +| **.NET** | OpenSearch.Client | [.NET Clients](https://docs.opensearch.org/latest/clients/dot-net/) | +| **Rust** | opensearch-rs | [Rust Client](https://docs.opensearch.org/latest/clients/rust/) | +| **Hadoop** | opensearch-hadoop | [GitHub](https://github.com/opensearch-project/opensearch-hadoop) | + +For detailed migration instructions, refer to the [OpenSearch Clients Documentation](https://docs.opensearch.org/latest/clients/). + +## References + +- [OpenSearch Migration Guide](https://opensearch.org/docs/latest/upgrade-or-migrate/) +- [Snapshot and Restore](https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +- [Reindex API](https://opensearch.org/docs/latest/api-reference/document-apis/reindex/) +- [Keystore Management](https://opensearch.org/docs/latest/security/configuration/security-settings/) +- [OpenSearch Clients](https://opensearch.org/docs/latest/clients/) From c75f0ddacbd098d3cbdcf8dd2829bfa4923e0246 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 16 Jan 2026 12:23:40 +0800 Subject: [PATCH 2/8] docs: update ES to OpenSearch migration guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add two-phase migration path: ES 7.10 → OS 2.x → OS 3.x - Add ES 8.x/9.x → OS 3.x reindex from remote migration - Fix version compatibility matrix (ES 7.10 cannot restore directly to OS 3.x) - Add reindex step in OS 2.x before upgrading to OS 3.x - Add SSL verification disable config for remote reindex - Update OpenSearch clients documentation - Use reindex.remote.allowlist for OS 3.x (breaking change from whitelist) --- ...igrate_from_Elasticsearch_to_OpenSearch.md | 49 +++++++++++++------ 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 6f858e6..f10d765 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -270,8 +270,7 @@ spec: Exclude system indices to avoid conflicts with OpenSearch's internal indices: ```bash -curl -k -X POST "https://localhost:9200/_snapshot/migration_repo/snapshot_1/_restore" \ - -u "admin:admin" \ +curl -k -u "admin:admin" -X POST "https://localhost:9200/_snapshot/migration_repo/snapshot_1/_restore" \ -H 'Content-Type: application/json' -d' { "indices": "-.kibana*,-.security*,-.monitoring*", @@ -285,10 +284,10 @@ Verify the index count and document count match the source cluster: ```bash # Check indices -curl -X GET "https://localhost:9200/_cat/indices?v" -u "admin:admin" -k +curl -k -u "admin:admin" -X GET "https://localhost:9200/_cat/indices?v" # Check document count -curl -X GET "https://localhost:9200//_count" -u "admin:admin" -k +curl -k -u "admin:admin" -X GET "https://localhost:9200//_count" ``` ### Phase 2: Reindex and Upgrade to OpenSearch 3.x @@ -415,15 +414,21 @@ spec: ### Procedure -#### Step 1: Configure the Remote Whitelist in OpenSearch +#### Step 1: Configure OpenSearch for Remote Reindex -Modify `opensearch.yml` (via Helm ConfigMap or Operator configuration) to allow connections to the ES 8.x host: +Add the following configurations to `OpenSearchCluster` CR's `additionalConfig`: ```yaml -reindex.remote.whitelist: ["es8-cluster-host:9200"] +spec: + general: + additionalConfig: + # Allow connections to ES 8.x host (OpenSearch 3.x uses 'allowlist') + reindex.remote.allowlist: "es8-cluster-host:9200" + # Disable SSL verification for self-signed certificates + reindex.ssl.verification_mode: "none" ``` -> **Note**: OpenSearch nodes must be restarted after this configuration change. +> **Note**: Nodes will be restarted after applying this configuration change. #### Step 2: Create Index Templates (Optional but Recommended) @@ -434,18 +439,18 @@ If your ES 8.x indices rely on specific settings or mappings, it is recommended Initiate the reindex request from the OpenSearch cluster. Set `wait_for_completion=false` to run asynchronously. ```bash -curl -X POST "http://localhost:9200/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d' +curl -k -u "admin:admin" -X POST "https://localhost:9200/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d' { "source": { "remote": { - "host": "http://es8-cluster-host:9200", + "host": "https://es8-cluster-host:9200", "username": "elastic", - "password": "password" + "password": "" }, - "index": "my-index-001" + "index": "migration_test" }, "dest": { - "index": "my-index-001" + "index": "migration_test" } }' ``` @@ -463,16 +468,28 @@ curl -X POST "http://localhost:9200/_reindex?wait_for_completion=false" -H 'Cont Use the Task ID from the previous step to check the task status: ```bash -curl -X GET "http://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" +curl -k -u "admin:admin" -X GET "https://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" ``` You can also view all reindex tasks: ```bash -curl -X GET "http://localhost:9200/_tasks?actions=*reindex&detailed=true" +curl -k -u "admin:admin" -X GET "https://localhost:9200/_tasks?actions=*reindex&detailed=true" ``` -> Monitor the `response.created` (documents created) and `response.total` (total documents) fields in the response. +> If the response shows `{"nodes":{}}`, it means no reindex tasks are currently running. The task may have completed or failed. Check the target index document count to verify. + +#### Step 5: Verify Reindex Completion + +Verify that the index was created and contains data: + +```bash +# Check if index exists and document count +curl -k -u "admin:admin" -X GET "https://localhost:9200/migration_test/_count" + +# Compare with source ES 8.x cluster +curl -k -u "elastic:" -X GET "https://es8-cluster-host:9200/migration_test/_count" +``` ## Client Migration Guide From d4b9ef410639add1ba8e4d47d721250653756191 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 16 Jan 2026 14:28:37 +0800 Subject: [PATCH 3/8] docs: refine ES to OpenSearch migration guide - Standardize curl command parameter order (options before -X METHOD) - Add missing authentication flags for ES/OpenSearch commands - Use 'elastic:' for ES commands, 'admin:admin' for OpenSearch - Fix YAML syntax error in ECK Elasticsearch CR config - Remove ES 9.x references (only ES 8.x supported) - Remove NFS from shared storage options - Simplify reindex step title and remove verbose task monitoring section --- ...igrate_from_Elasticsearch_to_OpenSearch.md | 58 +++++++------------ 1 file changed, 21 insertions(+), 37 deletions(-) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index f10d765..59b1fd9 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -14,7 +14,7 @@ Applicable Version: OpenSearch Operator ~= 2.8.*, OpenSearch ~= 2.x / 3.x This document provides detailed guidance for migrating from Elasticsearch (ES) to OpenSearch, covering two primary scenarios: 1. **Elasticsearch 7.10 → OpenSearch 2.x → OpenSearch 3.x**: Using Snapshot & Restore (two-phase migration) -2. **Elasticsearch 8.x/9.x → OpenSearch 3.x**: Using Reindex from Remote +2. **Elasticsearch 8.x → OpenSearch 3.x**: Using Reindex from Remote It also includes client migration guides for Java, Python, and Golang. @@ -24,24 +24,16 @@ It also includes client migration guides for Java, Python, and Golang. | :--- | :--- | :--- | :--- | | **ES 7.10** | **OS 2.x** | Snapshot & Restore | ✅ Direct restore supported | | **ES 7.10** | **OS 3.x** | Snapshot & Restore → Upgrade | ⚠️ Must restore to OS 2.x first, then upgrade | -| **ES 8.x/9.x** | **OS 3.x** | Reindex from Remote | ✅ Direct migration supported | +| **ES 8.x** | **OS 3.x** | Reindex from Remote | ✅ Direct migration supported | :::warning Key Compatibility Note - **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. - ES 7.10 snapshots must be restored to OpenSearch 2.x first, then upgrade the cluster to OS 3.x. -- ES 8.x/9.x use incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. +- ES 8.x use incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. ::: -### Migration Path Diagram - -```mermaid -flowchart LR - ES710["ES 7.10"] -->|"Snapshot & Restore"| OS2["OS 2.x"] -->|"Upgrade"| OS3a["OS 3.x"] - ES8["ES 8.x/9.x"] -->|"Reindex from Remote"| OS3b["OS 3.x"] -``` - ## Migrate from ES 7.10 to OpenSearch 2.x to 3.x This migration requires a **two-phase approach**: @@ -51,13 +43,13 @@ This migration requires a **two-phase approach**: ### Prerequisites -- A shared storage backend (e.g., S3 Bucket, GCS Bucket, or NFS) accessible by both source and target clusters. +- A shared storage backend (e.g., S3 Bucket, GCS Bucket) accessible by both source and target clusters. - The `repository-s3` plugin (or corresponding storage backend plugin) installed on both clusters. #### Check if Plugin is Installed ```bash -curl -X GET "http://localhost:9200/_cat/plugins?v" -u ":" +curl -u "elastic:" -X GET "http://localhost:9200/_cat/plugins?v" ``` #### Install repository-s3 Plugin @@ -170,7 +162,7 @@ For security reasons, avoid including access keys directly in API request bodies 2. Reload the secure settings: ```bash - curl -X POST "http://localhost:9200/_nodes/reload_secure_settings" + curl -u "elastic:" -X POST "http://localhost:9200/_nodes/reload_secure_settings" ``` **On OpenSearch:** @@ -212,7 +204,8 @@ Use the Operator's declarative configuration: #### Step 2: Register Snapshot Repository on Source Cluster (ES 7.10) ```bash -curl -X PUT "http://localhost:9200/_snapshot/migration_repo" -H 'Content-Type: application/json' -d' +curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_repo" \ + -H 'Content-Type: application/json' -d' { "type": "s3", "settings": { @@ -225,7 +218,8 @@ curl -X PUT "http://localhost:9200/_snapshot/migration_repo" -H 'Content-Type: a #### Step 3: Create a Full Snapshot ```bash -curl -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" -H 'Content-Type: application/json' -d' +curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_repo/snapshot_1?wait_for_completion=true" \ + -H 'Content-Type: application/json' -d' { "indices": "*", "ignore_unavailable": true, @@ -263,6 +257,7 @@ spec: bucket: my-migration-bucket base_path: es_710_backup readonly: "true" + ... ``` #### Step 5: Restore the Snapshot @@ -302,18 +297,16 @@ For each restored index, create a new index and reindex the data: ```bash # 1. Get the original index mapping and extract the mappings object using sed -curl -s -X GET "https://localhost:9200/migration_test/_mapping" -u "admin:admin" -k | \ +curl -s -u "admin:admin" -k -X GET "https://localhost:9200/migration_test/_mapping" | \ sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json # 2. Create a new index with the same mapping (add suffix _v2) -curl -X PUT "https://localhost:9200/migration_test_v2" \ - -u "admin:admin" -k \ +curl -u "admin:admin" -k -X PUT "https://localhost:9200/migration_test_v2" \ -H 'Content-Type: application/json' \ -d @mapping.json # 3. Reindex data from old index to new index -curl -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ - -u "admin:admin" -k \ +curl -u "admin:admin" -k -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ -H 'Content-Type: application/json' -d' { "source": { "index": "migration_test" }, @@ -321,9 +314,8 @@ curl -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ }' # 4. Delete old index and create alias (or rename) -curl -X DELETE "https://localhost:9200/migration_test" -u "admin:admin" -k -curl -X POST "https://localhost:9200/_aliases" \ - -u "admin:admin" -k \ +curl -u "admin:admin" -k -X DELETE "https://localhost:9200/migration_test" +curl -u "admin:admin" -k -X POST "https://localhost:9200/_aliases" \ -H 'Content-Type: application/json' -d' { "actions": [ @@ -335,7 +327,7 @@ curl -X POST "https://localhost:9200/_aliases" \ Repeat for all restored indices. After reindexing, verify the new index version: ```bash -curl -X GET "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" -u "admin:admin" -k +curl -u "admin:admin" -k -X GET "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" ``` The `version.created` should show an OpenSearch 2.x internal version number (e.g., `136408127` for OS 2.19.x). ES 7.10.2 indices show `7102099`. If you see a number starting with `136` or higher, the reindex was successful. @@ -359,8 +351,8 @@ The Operator will perform a rolling upgrade automatically. Verify all indices are accessible after upgrade: ```bash -curl -X GET "https://localhost:9200/_cat/indices?v" -u "admin:admin" -k -curl -X GET "https://localhost:9200/_cluster/health?pretty" -u "admin:admin" -k +curl -u "admin:admin" -k -X GET "https://localhost:9200/_cat/indices?v" +curl -u "admin:admin" -k -X GET "https://localhost:9200/_cluster/health?pretty" ``` ## Migrate from ES 8.x/9.x to OpenSearch 3.x @@ -389,7 +381,7 @@ spec: nodeSets: - name: default count: 3 - config: + config: {} podTemplate: spec: containers: @@ -434,7 +426,7 @@ spec: If your ES 8.x indices rely on specific settings or mappings, it is recommended to manually create the corresponding Index Templates or Mappings in OpenSearch beforehand. -#### Step 3: Execute Reindex (Asynchronous Mode) +#### Step 3: Execute Reindex Initiate the reindex request from the OpenSearch cluster. Set `wait_for_completion=false` to run asynchronously. @@ -471,14 +463,6 @@ Use the Task ID from the previous step to check the task status: curl -k -u "admin:admin" -X GET "https://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" ``` -You can also view all reindex tasks: - -```bash -curl -k -u "admin:admin" -X GET "https://localhost:9200/_tasks?actions=*reindex&detailed=true" -``` - -> If the response shows `{"nodes":{}}`, it means no reindex tasks are currently running. The task may have completed or failed. Check the target index document count to verify. - #### Step 5: Verify Reindex Completion Verify that the index was created and contains data: From d66456894ef3d08e087797216a2216a55828d2b5 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 16 Jan 2026 14:36:22 +0800 Subject: [PATCH 4/8] refine: standardize curl commands and remove redundant introduction in OS migration guide --- ...igrate_from_Elasticsearch_to_OpenSearch.md | 40 +++++++++---------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 59b1fd9..eba7430 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -11,12 +11,7 @@ kind: Applicable Version: OpenSearch Operator ~= 2.8.*, OpenSearch ~= 2.x / 3.x ::: -This document provides detailed guidance for migrating from Elasticsearch (ES) to OpenSearch, covering two primary scenarios: - -1. **Elasticsearch 7.10 → OpenSearch 2.x → OpenSearch 3.x**: Using Snapshot & Restore (two-phase migration) -2. **Elasticsearch 8.x → OpenSearch 3.x**: Using Reindex from Remote - -It also includes client migration guides for Java, Python, and Golang. +This document provides detailed guidance for migrating from Elasticsearch (ES) to OpenSearch. ## Migration Strategy Overview @@ -30,7 +25,7 @@ It also includes client migration guides for Java, Python, and Golang. - **ES 7.10 → OS 3.x direct restore is NOT supported**. OpenSearch 3.x requires indices to be created with OpenSearch 2.0.0+. - ES 7.10 snapshots must be restored to OpenSearch 2.x first, then upgrade the cluster to OS 3.x. -- ES 8.x use incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. +- ES 8.x uses incompatible Lucene versions, so Snapshot & Restore is not available; use Reindex from Remote instead. ::: @@ -49,7 +44,7 @@ This migration requires a **two-phase approach**: #### Check if Plugin is Installed ```bash -curl -u "elastic:" -X GET "http://localhost:9200/_cat/plugins?v" +curl -u "elastic:" "http://localhost:9200/_cat/plugins?v" ``` #### Install repository-s3 Plugin @@ -279,10 +274,10 @@ Verify the index count and document count match the source cluster: ```bash # Check indices -curl -k -u "admin:admin" -X GET "https://localhost:9200/_cat/indices?v" +curl -k -u "admin:admin" "https://localhost:9200/_cat/indices?v" # Check document count -curl -k -u "admin:admin" -X GET "https://localhost:9200//_count" +curl -k -u "admin:admin" "https://localhost:9200//_count" ``` ### Phase 2: Reindex and Upgrade to OpenSearch 3.x @@ -297,16 +292,19 @@ For each restored index, create a new index and reindex the data: ```bash # 1. Get the original index mapping and extract the mappings object using sed -curl -s -u "admin:admin" -k -X GET "https://localhost:9200/migration_test/_mapping" | \ + +curl -s -k -u "admin:admin" "https://localhost:9200/migration_test/_mapping" | \ sed 's/^{"migration_test"://' | sed 's/}$//' > mapping.json # 2. Create a new index with the same mapping (add suffix _v2) -curl -u "admin:admin" -k -X PUT "https://localhost:9200/migration_test_v2" \ + +curl -k -u "admin:admin" -X PUT "https://localhost:9200/migration_test_v2" \ -H 'Content-Type: application/json' \ -d @mapping.json # 3. Reindex data from old index to new index -curl -u "admin:admin" -k -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ + +curl -k -u "admin:admin" -X POST "https://localhost:9200/_reindex?wait_for_completion=true" \ -H 'Content-Type: application/json' -d' { "source": { "index": "migration_test" }, @@ -314,8 +312,8 @@ curl -u "admin:admin" -k -X POST "https://localhost:9200/_reindex?wait_for_compl }' # 4. Delete old index and create alias (or rename) -curl -u "admin:admin" -k -X DELETE "https://localhost:9200/migration_test" -curl -u "admin:admin" -k -X POST "https://localhost:9200/_aliases" \ +curl -k -u "admin:admin" -X DELETE "https://localhost:9200/migration_test" +curl -k -u "admin:admin" -X POST "https://localhost:9200/_aliases" \ -H 'Content-Type: application/json' -d' { "actions": [ @@ -327,7 +325,7 @@ curl -u "admin:admin" -k -X POST "https://localhost:9200/_aliases" \ Repeat for all restored indices. After reindexing, verify the new index version: ```bash -curl -u "admin:admin" -k -X GET "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" +curl -k -u "admin:admin" "https://localhost:9200/migration_test_v2/_settings?filter_path=**.version" ``` The `version.created` should show an OpenSearch 2.x internal version number (e.g., `136408127` for OS 2.19.x). ES 7.10.2 indices show `7102099`. If you see a number starting with `136` or higher, the reindex was successful. @@ -351,8 +349,8 @@ The Operator will perform a rolling upgrade automatically. Verify all indices are accessible after upgrade: ```bash -curl -u "admin:admin" -k -X GET "https://localhost:9200/_cat/indices?v" -curl -u "admin:admin" -k -X GET "https://localhost:9200/_cluster/health?pretty" +curl -k -u "admin:admin" "https://localhost:9200/_cat/indices?v" +curl -k -u "admin:admin" "https://localhost:9200/_cluster/health?pretty" ``` ## Migrate from ES 8.x/9.x to OpenSearch 3.x @@ -460,7 +458,7 @@ curl -k -u "admin:admin" -X POST "https://localhost:9200/_reindex?wait_for_compl Use the Task ID from the previous step to check the task status: ```bash -curl -k -u "admin:admin" -X GET "https://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" +curl -k -u "admin:admin" "https://localhost:9200/_tasks/N6q0j8s-T0m0j8s-T0m0j8:123456" ``` #### Step 5: Verify Reindex Completion @@ -469,10 +467,10 @@ Verify that the index was created and contains data: ```bash # Check if index exists and document count -curl -k -u "admin:admin" -X GET "https://localhost:9200/migration_test/_count" +curl -k -u "admin:admin" "https://localhost:9200/migration_test/_count" # Compare with source ES 8.x cluster -curl -k -u "elastic:" -X GET "https://es8-cluster-host:9200/migration_test/_count" +curl -k -u "elastic:" "https://es8-cluster-host:9200/migration_test/_count" ``` ## Client Migration Guide From f7cfac4e422ff68a5d910744975a770129f39239 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 16 Jan 2026 14:48:37 +0800 Subject: [PATCH 5/8] fix: fix doc links --- .../How_to_Migrate_from_Elasticsearch_to_OpenSearch.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index eba7430..a7dce6e 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -502,8 +502,7 @@ For detailed migration instructions, refer to the [OpenSearch Clients Documentat ## References -- [OpenSearch Migration Guide](https://opensearch.org/docs/latest/upgrade-or-migrate/) -- [Snapshot and Restore](https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) -- [Reindex API](https://opensearch.org/docs/latest/api-reference/document-apis/reindex/) -- [Keystore Management](https://opensearch.org/docs/latest/security/configuration/security-settings/) -- [OpenSearch Clients](https://opensearch.org/docs/latest/clients/) +- [OpenSearch Migration Guide](https://docs.opensearch.org/latest/upgrade-or-migrate/) +- [Snapshot and Restore](https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/) +- [Reindex API](https://docs.opensearch.org/latest/api-reference/document-apis/reindex/) +- [Keystore Management](https://docs.opensearch.org/latest/security/configuration/opensearch-keystore/) From 41259db8d7965143a5bac86e807a6d65099e44e3 Mon Sep 17 00:00:00 2001 From: hfxia Date: Fri, 23 Jan 2026 18:34:05 +0800 Subject: [PATCH 6/8] docs: add note about S3 config for data nodes in ES to OS migration guide --- .../How_to_Migrate_from_Elasticsearch_to_OpenSearch.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index a7dce6e..8d45902 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -106,6 +106,10 @@ extraVolumeMounts: mountPath: /usr/share/elasticsearch/plugins ``` +:::note +The above configuration only sets S3 configs for master nodes. If you have dedicated data nodes, add the same S3 config to `dataNodes` as well. +::: + **OpenSearch:** In your `OpenSearchCluster` CR: From cda3c369a4a24ae01031d846981ec0a0f0b5751e Mon Sep 17 00:00:00 2001 From: hfxia Date: Mon, 26 Jan 2026 16:11:21 +0800 Subject: [PATCH 7/8] docs: add notes to exclude system indices and clarify example index name in ES to OS migration guide --- ...ow_to_Migrate_from_Elasticsearch_to_OpenSearch.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 8d45902..4f8f370 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -222,10 +222,14 @@ curl -u "elastic:" -X PUT "http://localhost:9200/_snapshot/migration_r { "indices": "*", "ignore_unavailable": true, - "include_global_state": true + "include_global_state": true }' ``` +:::note Excluding System Indices +It is recommended to exclude system indices (`.kibana*`, `.security*`, `.monitoring*`, `apm*`, `.apm*`) during snapshot creation. These indices are Elasticsearch-specific and will conflict with OpenSearch's internal indices during restore. By excluding them at snapshot time, you reduce snapshot size and avoid potential restore issues. +::: + ### Phase 1: Restore to OpenSearch 2.x #### Step 4: Deploy OpenSearch 2.x Cluster @@ -267,7 +271,7 @@ Exclude system indices to avoid conflicts with OpenSearch's internal indices: curl -k -u "admin:admin" -X POST "https://localhost:9200/_snapshot/migration_repo/snapshot_1/_restore" \ -H 'Content-Type: application/json' -d' { - "indices": "-.kibana*,-.security*,-.monitoring*", + "indices": "-.kibana*,-.security*,-.monitoring*,-apm*,-.apm*", "include_global_state": false }' ``` @@ -294,6 +298,10 @@ Indices restored from ES 7.10 snapshots retain their original version metadata ( For each restored index, create a new index and reindex the data: +:::note +The examples below use `migration_test` as the index name. Replace `migration_test` with your actual index name when executing these commands. +::: + ```bash # 1. Get the original index mapping and extract the mappings object using sed From 88c477344051f6df384296971571ab921de158bb Mon Sep 17 00:00:00 2001 From: hfxia Date: Tue, 27 Jan 2026 15:42:23 +0800 Subject: [PATCH 8/8] docs: add note about http access when TLS disabled in ES to OS migration guide --- ..._to_Migrate_from_Elasticsearch_to_OpenSearch.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md index 4f8f370..fedd9c8 100644 --- a/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md +++ b/docs/en/solutions/ecosystem/opensearch/How_to_Migrate_from_Elasticsearch_to_OpenSearch.md @@ -414,6 +414,20 @@ spec: storage: 5Gi ``` +:::note TLS Configuration +If you disable TLS by setting: + +```yaml +spec: + http: + tls: + selfSignedCertificate: + disabled: true +``` + +You must use `http://` instead of `https://` when accessing the Elasticsearch API. +::: + ### Procedure #### Step 1: Configure OpenSearch for Remote Reindex