- Column exclusion, computed columns, and partitioned tables are now supported in table migrations with MOLT Fetch. They are supported via a new transformations framework that allows the user to specify a JSON file with instructions on how MOLT Fetch should treat certain schemas, tables, or underlying columns. + Column exclusion, computed columns, and partitioned tables are now supported in table migrations with MOLT Fetch. They are supported via a new transformations framework that allows the user to specify a JSON file with instructions on how MOLT Fetch should treat certain schemas, tables, or underlying columns.
**Default:** `false` | +| `--cleanup` | Whether to delete intermediate files after moving data using [cloud or local storage]({% link molt/molt-fetch.md %}#define-intermediate-storage). **Note:** Cleanup does not occur on [continuation]({% link molt/molt-fetch.md %}#continue-molt-fetch-after-interruption). | +| `--compression` | Compression method for data when using [`IMPORT INTO`]({% link molt/molt-fetch.md %}#import-into-vs-copy-from) (`gzip`/`none`).
**Default:** `gzip` | +| `--continuation-file-name` | Restart fetch at the specified filename if the task encounters an error. `--fetch-id` must be specified. For details, see [Fetch continuation]({% link molt/molt-fetch.md %}#continue-molt-fetch-after-interruption). | +| `--continuation-token` | Restart fetch at a specific table, using the specified continuation token, if the task encounters an error. `--fetch-id` must be specified. For details, see [Fetch continuation]({% link molt/molt-fetch.md %}#continue-molt-fetch-after-interruption). | +| `--crdb-pts-duration` | The duration for which each timestamp used in data export from a CockroachDB source is protected from garbage collection. This ensures that the data snapshot remains consistent. For example, if set to `24h`, each timestamp is protected for 24 hours from the initiation of the export job. This duration is extended at regular intervals specified in `--crdb-pts-refresh-interval`.
**Default:** `24h0m0s` | +| `--crdb-pts-refresh-interval` | The frequency at which the protected timestamp's validity is extended. This interval maintains protection of the data snapshot until data export from a CockroachDB source is completed. For example, if set to `10m`, the protected timestamp's expiration will be extended by the duration specified in `--crdb-pts-duration` (e.g., `24h`) every 10 minutes while export is not complete.
**Default:** `10m0s` | +| `--direct-copy` | Enables [direct copy]({% link molt/molt-fetch.md %}#direct-copy), which copies data directly from source to target without using an intermediate store. | +| `--export-concurrency` | Number of shards to export at a time per table, each on a dedicated thread. This controls how many shards are created for each individual table during the [data export phase]({% link molt/molt-fetch.md %}#data-export-phase) and is distinct from `--table-concurrency`, which controls how many tables are processed simultaneously. The total number of concurrent threads is the product of `--export-concurrency` and `--table-concurrency`. Tables can be sharded with a range-based or stats-based mechanism. For details, refer to [Table sharding]({% link molt/molt-fetch.md %}#shard-tables-for-concurrent-export).
**Default:** `4` | +| `--export-retry-max-attempts` | Maximum number of retry attempts for source export queries when connection failures occur. Only supported for PostgreSQL and CockroachDB sources.
**Default:** `3` | +| `--export-retry-max-duration` | Maximum total duration for retrying source export queries. If `0`, no time limit is enforced. Only supported for PostgreSQL and CockroachDB sources.
**Default:** `5m0s` | +| `--filter-path` | Path to a JSON file defining row-level filters for the [data import phase]({% link molt/molt-fetch.md %}#data-import-phase). Refer to [Selective data movement]({% link molt/molt-fetch.md %}#select-data-to-migrate). | +| `--fetch-id` | Restart fetch task corresponding to the specified ID. If `--continuation-file-name` or `--continuation-token` are not specified, fetch restarts for all failed tables. | +| `--flush-rows` | Number of rows before the source data is flushed to intermediate files. **Note:** If `--flush-size` is also specified, the fetch behavior is based on the flag whose criterion is met first. | +| `--flush-size` | Size (in bytes) before the source data is flushed to intermediate files. **Note:** If `--flush-rows` is also specified, the fetch behavior is based on the flag whose criterion is met first. | +| `--ignore-replication-check` | Skip querying for replication checkpoints such as `pg_current_wal_insert_lsn()` on PostgreSQL, `gtid_executed` on MySQL, and `CURRENT_SCN` on Oracle. This option is intended for use during bulk load migrations or when doing a one-time data export from a read replica. | +| `--import-batch-size` | The number of files to be imported at a time to the target database during the [data import phase]({% link molt/molt-fetch.md %}#data-import-phase). This applies only when using [`IMPORT INTO`]({% link molt/molt-fetch.md %}#import-into-vs-copy-from) for data movement. **Note:** Increasing this value can improve the performance of full-scan queries on the target database shortly after fetch completes, but very high values are not recommended. If any individual file in the import batch fails, you must [retry]({% link molt/molt-fetch.md %}#continue-molt-fetch-after-interruption) the entire batch.
**Default:** `1000` | +| `--import-region` | The region of the [cloud storage]({% link molt/molt-fetch.md %}#bucket-path) bucket. This applies only to [Amazon S3 buckets]({% link molt/molt-fetch.md %}#bucket-path). Set this flag only if you need to specify an `AWS_REGION` explicitly when using [`IMPORT INTO`]({% link molt/molt-fetch.md %}#import-into-vs-copy-from) for data movement. For example, `--import-region=ap-south-1`. | +| `--local-path` | The path within the [local file server]({% link molt/molt-fetch.md %}#local-path) where intermediate files are written (e.g., `data/migration/cockroach`). `--local-path-listen-addr` must be specified. | +| `--local-path-crdb-access-addr` | Address of a [local file server]({% link molt/molt-fetch.md %}#local-path) that is **publicly accessible**. This flag is only necessary if CockroachDB cannot reach the local address specified with `--local-path-listen-addr` (e.g., when moving data to a CockroachDB {{ site.data.products.cloud }} deployment). `--local-path` and `--local-path-listen-addr` must be specified.
**Default:** Value of `--local-path-listen-addr`. | +| `--local-path-listen-addr` | Write intermediate files to a [local file server]({% link molt/molt-fetch.md %}#local-path) at the specified address (e.g., `'localhost:3000'`). `--local-path` must be specified. | +| `--log-file` | Write messages to the specified log filename. If no filename is provided, messages write to `fetch-{datetime}.log`. If `"stdout"` is provided, messages write to `stdout`. | +| `--logging` | Level at which to log messages (`trace`/`debug`/`info`/`warn`/`error`/`fatal`/`panic`).
**Default:** `info` | +| `--metrics-listen-addr` | Address of the Prometheus metrics endpoint, which has the path `{address}/metrics`. For details on important metrics to monitor, refer to [Monitoring]({% link molt/molt-fetch-monitoring.md %}).
**Default:** `'127.0.0.1:3030'` | +| `--mode` | Configure the MOLT Fetch behavior: `data-load`, `export-only`, or `import-only`. For details, refer to [Fetch mode]({% link molt/molt-fetch.md %}#define-fetch-mode).
**Default:** `data-load` | +| `--non-interactive` | Run the fetch task without interactive prompts. This is recommended **only** when running `molt fetch` in an automated process (i.e., a job or continuous integration). | +| `--pprof-listen-addr` | Address of the pprof endpoint.
**Default:** `'127.0.0.1:3031'` | +| `--row-batch-size` | Number of rows per shard to export at a time. For details on sharding, refer to [Table sharding]({% link molt/molt-fetch.md %}#shard-tables-for-concurrent-export). See also [Best practices]({% link molt/molt-fetch-best-practices.md %}).
**Default:** `100000` | +| `--schema-filter` | Move schemas that match a specified [regular expression](https://wikipedia.org/wiki/Regular_expression).
**Default:** `'.*'` | +| `--skip-pk-check` | Skip primary-key matching to allow data load when source or target tables have missing or mismatched primary keys. Disables sharding and bypasses `--export-concurrency` and `--row-batch-size` settings. Refer to [Skip primary key matching]({% link molt/molt-fetch.md %}#skip-primary-key-matching).
**Default:** `false` | +| `--table-concurrency` | Number of tables to export at a time. The number of concurrent threads is the product of `--export-concurrency` and `--table-concurrency`.
**Default:** `4` | +| `--table-exclusion-filter` | Exclude tables that match a specified [POSIX regular expression](https://wikipedia.org/wiki/Regular_expression).
This value **cannot** be set to `'.*'`, which would cause every table to be excluded.
**Default:** Empty string | +| `--table-filter` | Move tables that match a specified [POSIX regular expression](https://wikipedia.org/wiki/Regular_expression).
**Default:** `'.*'` | +| `--table-handling` | How tables are initialized on the target database (`none`/`drop-on-target-and-recreate`/`truncate-if-exists`). For details, see [Target table handling]({% link molt/molt-fetch.md %}#handle-target-tables).
**Default:** `none` | +| `--transformations-file` | Path to a JSON file that defines transformations to be performed on the target schema during the fetch task. Refer to [Transformations]({% link molt/molt-fetch.md %}#define-transformations). | +| `--type-map-file` | Path to a JSON file that contains explicit type mappings for automatic schema creation, when enabled with `--table-handling drop-on-target-and-recreate`. For details on the JSON format and valid type mappings, see [type mapping]({% link molt/molt-fetch.md %}#type-mapping). | +| `--use-console-writer` | Use the console writer, which has cleaner log output but introduces more latency.
**Default:** `false` (log as structured JSON) | +| `--use-copy` | Use [`COPY FROM`]({% link molt/molt-fetch.md %}#import-into-vs-copy-from) to move data. This makes tables queryable during data load, but is slower than using `IMPORT INTO`. For details, refer to [Data movement]({% link molt/molt-fetch.md %}#import-into-vs-copy-from). | +| `--use-implicit-auth` | Use [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}) for [cloud storage]({% link molt/molt-fetch.md %}#bucket-path) URIs. | +| `--use-stats-based-sharding` | Enable statistics-based sharding for PostgreSQL sources. This allows sharding of tables with primary keys of any data type and can create more evenly distributed shards compared to the default numerical range sharding. Requires PostgreSQL 11+ and access to `pg_stats`. For details, refer to [Table sharding]({% link molt/molt-fetch.md %}#shard-tables-for-concurrent-export). | + + +### `tokens list` flags + +| Flag | Description | +|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------| +| `--conn-string` | (Required) Connection string for the target database. For details, see [List active continuation tokens]({% link molt/molt-fetch.md %}#list-active-continuation-tokens). | +| `-n`, `--num-results` | Number of results to return. | + +## See also + +- [MOLT Fetch]({% link molt/molt-fetch.md %}) +- [MOLT Fetch Best Practices]({% link molt/molt-fetch-best-practices.md %}) +- [MOLT Fetch Monitoring]({% link molt/molt-fetch-monitoring.md %}) +- [MOLT Fetch Troubleshooting]({% link molt/molt-fetch-troubleshooting.md %}) +- [Migration Overview]({% link molt/migration-overview.md %}) diff --git a/src/current/molt/molt-fetch-installation.md b/src/current/molt/molt-fetch-installation.md new file mode 100644 index 00000000000..fee592f94a5 --- /dev/null +++ b/src/current/molt/molt-fetch-installation.md @@ -0,0 +1,52 @@ +--- +title: MOLT Fetch Installation +summary: Learn how to install MOLT Fetch and configure prerequisites for data migration. +toc: true +docs_area: migrate +--- + +## Prerequisites + +### Supported databases + +The following source databases are supported: + +- PostgreSQL 11-16 +- MySQL 5.7, 8.0 and later +- Oracle Database 19c (Enterprise Edition) and 21c (Express Edition) + +### Database configuration + +Ensure that the source and target schemas are identical, unless you enable automatic schema creation with the [`drop-on-target-and-recreate`]({% link molt/molt-fetch.md %}#handle-target-tables) option. If you are creating the target schema manually, review the behaviors in [Mismatch handling]({% link molt/molt-fetch.md %}#mismatch-handling). + +{{site.data.alerts.callout_info}} +MOLT Fetch does not support migrating sequences. If your source database contains sequences, refer to the [guidance on indexing with sequential keys]({% link {{site.current_cloud_version}}/sql-faqs.md %}#how-do-i-generate-unique-slowly-increasing-sequential-numbers-in-cockroachdb). If a sequential key is necessary in your CockroachDB table, you must create it manually. After using MOLT Fetch to load the data onto the target, but before cutover, make sure to update each sequence's current value using [`setval()`]({% link {{site.current_cloud_version}}/functions-and-operators.md %}#sequence-functions) so that new inserts continue from the correct point. +{{site.data.alerts.end}} + +If you plan to use cloud storage for the data migration, follow [Cloud storage security]({% link molt/molt-fetch-best-practices.md %}#cloud-storage-security) best practices. + +### User permissions + +The SQL user running MOLT Fetch requires specific privileges on both the source and target databases: + +| Database | Required Privileges | Details | +|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| +| PostgreSQL source |
- `CONNECT` on database.
- `USAGE` on schema.
- `SELECT` on tables to migrate.
- `SELECT` on tables to migrate.
- `CONNECT` and `CREATE SESSION`.
- `SELECT` and `FLASHBACK` on tables to migrate.
- `SELECT` on metadata views (`ALL_USERS`, `DBA_USERS`, `DBA_OBJECTS`, `DBA_SYNONYMS`, `DBA_TABLES`).
- `ALL` on target database.
- `CREATE` on schema.
- `SELECT`, `INSERT`, `UPDATE`, `DELETE` on target tables.
- For `IMPORT INTO`: `SELECT`, `INSERT`, `DROP` on target tables. Optionally `EXTERNALIOIMPLICITACCESS` for implicit cloud storage authentication.
- For `COPY FROM`: `admin` role.
`molt_fetch_rows_exported{table="public.users"}` | +| `molt_fetch_rows_imported` | Number of rows that have been imported from a table. For example:
`molt_fetch_rows_imported{table="public.users"}` | +| `molt_fetch_table_export_duration_ms` | Duration (in milliseconds) of a table's export. For example:
`molt_fetch_table_export_duration_ms{table="public.users"}` | +| `molt_fetch_table_import_duration_ms` | Duration (in milliseconds) of a table's import. For example:
`molt_fetch_table_import_duration_ms{table="public.users"}` | + +You can also use the [sample Grafana dashboard](https://molt.cockroachdb.com/molt/cli/grafana_dashboard.json) to view the preceding metrics. + +## See also + +- [MOLT Fetch]({% link molt/molt-fetch.md %}) +- [MOLT Fetch Best Practices]({% link molt/molt-fetch-best-practices.md %}) +- [Migration Overview]({% link molt/migration-overview.md %}) +- [MOLT Replicator]({% link molt/molt-replicator.md %}) diff --git a/src/current/molt/molt-fetch-troubleshooting.md b/src/current/molt/molt-fetch-troubleshooting.md new file mode 100644 index 00000000000..59555ce78be --- /dev/null +++ b/src/current/molt/molt-fetch-troubleshooting.md @@ -0,0 +1,21 @@ +--- +title: MOLT Fetch Troubleshooting +summary: Troubleshoot common issues when using MOLT Fetch for data migration. +toc: true +docs_area: migrate +--- + +
+
+
+
+
+
+{% include molt/molt-troubleshooting-fetch.md %}
+
+## See also
+
+- [MOLT Fetch]({% link molt/molt-fetch.md %})
+- [Migration Overview]({% link molt/migration-overview.md %})
+- [MOLT Replicator]({% link molt/molt-replicator.md %})
+- [MOLT Verify]({% link molt/molt-verify.md %})
diff --git a/src/current/molt/molt-fetch.md b/src/current/molt/molt-fetch.md
index fe4318bf145..ed7c3497f15 100644
--- a/src/current/molt/molt-fetch.md
+++ b/src/current/molt/molt-fetch.md
@@ -7,158 +7,55 @@ docs_area: migrate
MOLT Fetch moves data from a source database into CockroachDB as part of a [database migration]({% link molt/migration-overview.md %}).
-MOLT Fetch uses [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}) or [`COPY FROM`]({% link {{site.current_cloud_version}}/copy.md %}) to move the source data to cloud storage (Google Cloud Storage, Amazon S3, or Azure Blob Storage), a local file server, or local memory. Once the data is exported, MOLT Fetch loads the data into a target CockroachDB database. For details, refer to [Migration phases](#migration-phases).
+MOLT Fetch uses [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}) or [`COPY FROM`]({% link {{site.current_cloud_version}}/copy.md %}) to move the source data to cloud storage (Google Cloud Storage, Amazon S3, or Azure Blob Storage), a local file server, or local memory. Once the data is exported, MOLT Fetch loads the data into a target CockroachDB database.
-## Terminology
+
-## Prerequisites
+## How it works
-### Supported databases
+MOLT Fetch operates in two distinct phases to move data from the source databases to CockroachDB. The [data export phase](#data-export-phase) moves data to intermediate storage (either cloud storage or a local file server). The [data import phase](#data-import-phase) moves data from that intermediate storage to the CockroachDB cluster. For details on available modes, refer to [Define fetch mode](#define-fetch-mode).
-The following source databases are supported:
-
-- PostgreSQL 11-16
-- MySQL 5.7, 8.0 and later
-- Oracle Database 19c (Enterprise Edition) and 21c (Express Edition)
-
-### Database configuration
-
-Ensure that the source and target schemas are identical, unless you enable automatic schema creation with the [`drop-on-target-and-recreate`](#target-table-handling) option. If you are creating the target schema manually, review the behaviors in [Mismatch handling](#mismatch-handling).
-
-{{site.data.alerts.callout_info}}
-MOLT Fetch does not support migrating sequences. If your source database contains sequences, refer to the [guidance on indexing with sequential keys]({% link {{site.current_cloud_version}}/sql-faqs.md %}#how-do-i-generate-unique-slowly-increasing-sequential-numbers-in-cockroachdb). If a sequential key is necessary in your CockroachDB table, you must create it manually. After using MOLT Fetch to load the data onto the target, but before cutover, make sure to update each sequence's current value using [`setval()`]({% link {{site.current_cloud_version}}/functions-and-operators.md %}#sequence-functions) so that new inserts continue from the correct point.
-{{site.data.alerts.end}}
-
-If you plan to use cloud storage for the data migration, follow the steps in [Cloud storage security](#cloud-storage-security).
+
+
+
-### User permissions
+### Data export phase
-The SQL user running MOLT Fetch requires specific privileges on both the source and target databases:
+In this first phase, MOLT Fetch connects to the source database and exports table data to intermediate storage.
-| Database | Required Privileges | Details |
-|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
-| PostgreSQL source |
+- `CONNECT` on database.
- `USAGE` on schema.
- `SELECT` on tables to migrate.
- `SELECT` on tables to migrate.
- `CONNECT` and `CREATE SESSION`.
- `SELECT` and `FLASHBACK` on tables to migrate.
- `SELECT` on metadata views (`ALL_USERS`, `DBA_USERS`, `DBA_OBJECTS`, `DBA_SYNONYMS`, `DBA_TABLES`).
- `ALL` on target database.
- `CREATE` on schema.
- `SELECT`, `INSERT`, `UPDATE`, `DELETE` on target tables.
- For `IMPORT INTO`: `SELECT`, `INSERT`, `DROP` on target tables. Optionally `EXTERNALIOIMPLICITACCESS` for implicit cloud storage authentication.
- For `COPY FROM`: `admin` role.
**Default:** `false` | -| `--cleanup` | Whether to delete intermediate files after moving data using [cloud or local storage](#data-path). **Note:** Cleanup does not occur on [continuation](#fetch-continuation). | -| `--compression` | Compression method for data when using [`IMPORT INTO`](#data-load-mode) (`gzip`/`none`).
**Default:** `gzip` | -| `--continuation-file-name` | Restart fetch at the specified filename if the task encounters an error. `--fetch-id` must be specified. For details, see [Fetch continuation](#fetch-continuation). | -| `--continuation-token` | Restart fetch at a specific table, using the specified continuation token, if the task encounters an error. `--fetch-id` must be specified. For details, see [Fetch continuation](#fetch-continuation). | -| `--crdb-pts-duration` | The duration for which each timestamp used in data export from a CockroachDB source is protected from garbage collection. This ensures that the data snapshot remains consistent. For example, if set to `24h`, each timestamp is protected for 24 hours from the initiation of the export job. This duration is extended at regular intervals specified in `--crdb-pts-refresh-interval`.
**Default:** `24h0m0s` | -| `--crdb-pts-refresh-interval` | The frequency at which the protected timestamp's validity is extended. This interval maintains protection of the data snapshot until data export from a CockroachDB source is completed. For example, if set to `10m`, the protected timestamp's expiration will be extended by the duration specified in `--crdb-pts-duration` (e.g., `24h`) every 10 minutes while export is not complete.
**Default:** `10m0s` | -| `--direct-copy` | Enables [direct copy](#direct-copy), which copies data directly from source to target without using an intermediate store. | -| `--export-concurrency` | Number of shards to export at a time per table, each on a dedicated thread. This controls how many shards are created for each individual table during the [data export phase](#data-export-phase) and is distinct from `--table-concurrency`, which controls how many tables are processed simultaneously. The total number of concurrent threads is the product of `--export-concurrency` and `--table-concurrency`. Tables can be sharded with a range-based or stats-based mechanism. For details, refer to [Table sharding](#table-sharding).
**Default:** `4` | -| `--export-retry-max-attempts` | Maximum number of retry attempts for source export queries when connection failures occur. Only supported for PostgreSQL and CockroachDB sources.
**Default:** `3` | -| `--export-retry-max-duration` | Maximum total duration for retrying source export queries. If `0`, no time limit is enforced. Only supported for PostgreSQL and CockroachDB sources.
**Default:** `5m0s` | -| `--filter-path` | Path to a JSON file defining row-level filters for the [data import phase](#data-import-phase). Refer to [Selective data movement](#selective-data-movement). | -| `--fetch-id` | Restart fetch task corresponding to the specified ID. If `--continuation-file-name` or `--continuation-token` are not specified, fetch restarts for all failed tables. | -| `--flush-rows` | Number of rows before the source data is flushed to intermediate files. **Note:** If `--flush-size` is also specified, the fetch behavior is based on the flag whose criterion is met first. | -| `--flush-size` | Size (in bytes) before the source data is flushed to intermediate files. **Note:** If `--flush-rows` is also specified, the fetch behavior is based on the flag whose criterion is met first. | -| `--ignore-replication-check` | Skip querying for replication checkpoints such as `pg_current_wal_insert_lsn()` on PostgreSQL, `gtid_executed` on MySQL, and `CURRENT_SCN` on Oracle. This option is intended for use during bulk load migrations or when doing a one-time data export from a read replica. | -| `--import-batch-size` | The number of files to be imported at a time to the target database during the [data import phase](#data-import-phase). This applies only when using [`IMPORT INTO`](#data-load-mode) for data movement. **Note:** Increasing this value can improve the performance of full-scan queries on the target database shortly after fetch completes, but very high values are not recommended. If any individual file in the import batch fails, you must [retry](#fetch-continuation) the entire batch.
**Default:** `1000` | -| `--import-region` | The region of the [cloud storage](#bucket-path) bucket. This applies only to [Amazon S3 buckets](#bucket-path). Set this flag only if you need to specify an `AWS_REGION` explicitly when using [`IMPORT INTO`](#data-load-mode) for data movement. For example, `--import-region=ap-south-1`. | -| `--local-path` | The path within the [local file server](#local-path) where intermediate files are written (e.g., `data/migration/cockroach`). `--local-path-listen-addr` must be specified. | -| `--local-path-crdb-access-addr` | Address of a [local file server](#local-path) that is **publicly accessible**. This flag is only necessary if CockroachDB cannot reach the local address specified with `--local-path-listen-addr` (e.g., when moving data to a CockroachDB {{ site.data.products.cloud }} deployment). `--local-path` and `--local-path-listen-addr` must be specified.
**Default:** Value of `--local-path-listen-addr`. | -| `--local-path-listen-addr` | Write intermediate files to a [local file server](#local-path) at the specified address (e.g., `'localhost:3000'`). `--local-path` must be specified. | -| `--log-file` | Write messages to the specified log filename. If no filename is provided, messages write to `fetch-{datetime}.log`. If `"stdout"` is provided, messages write to `stdout`. | -| `--logging` | Level at which to log messages (`trace`/`debug`/`info`/`warn`/`error`/`fatal`/`panic`).
**Default:** `info` | -| `--metrics-listen-addr` | Address of the Prometheus metrics endpoint, which has the path `{address}/metrics`. For details on important metrics to monitor, refer to [Monitoring](#monitoring).
**Default:** `'127.0.0.1:3030'` | -| `--mode` | Configure the MOLT Fetch behavior: `data-load`, `export-only`, or `import-only`. For details, refer to [Fetch mode](#fetch-mode).
**Default:** `data-load` | -| `--non-interactive` | Run the fetch task without interactive prompts. This is recommended **only** when running `molt fetch` in an automated process (i.e., a job or continuous integration). | -| `--pglogical-replication-slot-name` | Name of a PostgreSQL replication slot that will be created before taking a snapshot of data. Must match the slot name specified with `--slotName` in the [MOLT Replicator command]({% link molt/molt-replicator.md %}#replication-checkpoints). For details, refer to [Load before replication](#load-before-replication). | -| `--pglogical-publication-and-slot-drop-and-recreate` | Drop the PostgreSQL publication and replication slot if they exist, then recreate them. Creates a publication named `molt_fetch` and the replication slot specified with `--pglogical-replication-slot-name`. For details, refer to [Load before replication](#load-before-replication).
**Default:** `false` | -| `--pprof-listen-addr` | Address of the pprof endpoint.
**Default:** `'127.0.0.1:3031'` | -| `--row-batch-size` | Number of rows per shard to export at a time. For details on sharding, refer to [Table sharding](#table-sharding). See also [Best practices](#best-practices).
**Default:** `100000` | -| `--schema-filter` | Move schemas that match a specified [regular expression](https://wikipedia.org/wiki/Regular_expression). Not used with MySQL sources. For Oracle sources, this filter is case-insensitive.
**Default:** `'.*'` | -| `--skip-pk-check` | Skip primary-key matching to allow data load when source or target tables have missing or mismatched primary keys. Disables sharding and bypasses `--export-concurrency` and `--row-batch-size` settings. Refer to [Skip primary key matching](#skip-primary-key-matching).
**Default:** `false` | -| `--table-concurrency` | Number of tables to export at a time. The number of concurrent threads is the product of `--export-concurrency` and `--table-concurrency`.
**Default:** `4` | -| `--table-exclusion-filter` | Exclude tables that match a specified [POSIX regular expression](https://wikipedia.org/wiki/Regular_expression).
This value **cannot** be set to `'.*'`, which would cause every table to be excluded.
**Default:** Empty string | -| `--table-filter` | Move tables that match a specified [POSIX regular expression](https://wikipedia.org/wiki/Regular_expression).
**Default:** `'.*'` | -| `--table-handling` | How tables are initialized on the target database (`none`/`drop-on-target-and-recreate`/`truncate-if-exists`). For details, see [Target table handling](#target-table-handling).
**Default:** `none` | -| `--transformations-file` | Path to a JSON file that defines transformations to be performed on the target schema during the fetch task. Refer to [Transformations](#transformations). | -| `--type-map-file` | Path to a JSON file that contains explicit type mappings for automatic schema creation, when enabled with `--table-handling drop-on-target-and-recreate`. For details on the JSON format and valid type mappings, see [type mapping](#type-mapping). | -| `--use-console-writer` | Use the console writer, which has cleaner log output but introduces more latency.
**Default:** `false` (log as structured JSON) | -| `--use-copy` | Use [`COPY FROM`](#data-load-mode) to move data. This makes tables queryable during data load, but is slower than using `IMPORT INTO`. For details, refer to [Data load mode](#data-load-mode). | -| `--use-implicit-auth` | Use [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}) for [cloud storage](#bucket-path) URIs. | -| `--use-stats-based-sharding` | Enable statistics-based sharding for PostgreSQL sources. This allows sharding of tables with primary keys of any data type and can create more evenly distributed shards compared to the default numerical range sharding. Requires PostgreSQL 11+ and access to `pg_stats`. For details, refer to [Table sharding](#table-sharding). | - - -### `tokens list` flags - -| Flag | Description | -|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| `--conn-string` | (Required) Connection string for the target database. For details, see [List active continuation tokens](#list-active-continuation-tokens). | -| `-n`, `--num-results` | Number of results to return. | - - -## Usage - -The following sections describe how to use the `molt fetch` [flags](#flags). - -### Source and target databases +### Specify source and target databases {{site.data.alerts.callout_success}} -Follow the recommendations in [Connection security](#connection-security). +Follow the recommendations in [Connection security]({% link molt/molt-fetch-best-practices.md %}#connection-security). {{site.data.alerts.end}} -`--source` specifies the connection string of the source database. +[`--source`]({% link molt/molt-fetch-commands-and-flags.md %}#source) specifies the connection string of the source database. PostgreSQL or CockroachDB connection string: @@ -181,7 +78,7 @@ Oracle connection string: --source 'oracle://{username}:{password}@{host}:{port}/{service_name}' ~~~ -For Oracle Multitenant databases, `--source-cdb` specifies the container database (CDB) connection. `--source` specifies the pluggable database (PDB): +For Oracle Multitenant databases, [`--source-cdb`]({% link molt/molt-fetch-commands-and-flags.md %}#source-cdb) specifies the container database (CDB) connection. [`--source`]({% link molt/molt-fetch-commands-and-flags.md %}#source) specifies the pluggable database (PDB): {% include_cached copy-clipboard.html %} ~~~ @@ -189,16 +86,16 @@ For Oracle Multitenant databases, `--source-cdb` specifies the container databas --source-cdb 'oracle://{username}:{password}@{host}:{port}/{cdb_service_name}' ~~~ -`--target` specifies the [CockroachDB connection string]({% link {{site.current_cloud_version}}/connection-parameters.md %}#connect-using-a-url): +[`--target`]({% link molt/molt-fetch-commands-and-flags.md %}#target) specifies the [CockroachDB connection string]({% link {{site.current_cloud_version}}/connection-parameters.md %}#connect-using-a-url): {% include_cached copy-clipboard.html %} ~~~ --target 'postgresql://{username}:{password}@{host}:{port}/{database}' ~~~ -### Fetch mode +### Define fetch mode -`--mode` specifies the MOLT Fetch behavior. +[`--mode`]({% link molt/molt-fetch-commands-and-flags.md %}#mode) specifies the MOLT Fetch behavior. `data-load` (default) instructs MOLT Fetch to load the source data into CockroachDB: @@ -221,29 +118,108 @@ For Oracle Multitenant databases, `--source-cdb` specifies the container databas --mode import-only ~~~ -### Data load mode +### Select data to migrate -MOLT Fetch can use either [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}) or [`COPY FROM`]({% link {{site.current_cloud_version}}/copy.md %}) to load data into CockroachDB. +By default, MOLT Fetch moves all data from the [`--source`]({% link molt/molt-fetch-commands-and-flags.md %}#source) database to CockroachDB. Use the following flags to move a subset of data. -By default, MOLT Fetch uses `IMPORT INTO`: +#### Schema and table selection -- `IMPORT INTO` achieves the highest throughput, but [requires taking the CockroachDB tables **offline**]({% link {{site.current_cloud_version}}/import-into.md %}#considerations) to achieve its import speed. Tables are taken back online once an [import job]({% link {{site.current_cloud_version}}/import-into.md %}#view-and-control-import-jobs) completes successfully. See [Best practices](#best-practices). -- `IMPORT INTO` supports compression using the `--compression` flag, which reduces the amount of storage used. +[`--schema-filter`]({% link molt/molt-fetch-commands-and-flags.md %}#schema-filter) specifies a range of schema objects to move to CockroachDB, formatted as a POSIX regex string. For example, to move every table in the source database's `migration_schema` schema: -`--use-copy` configures MOLT Fetch to use `COPY FROM`: +{% include_cached copy-clipboard.html %} +~~~ +--schema-filter 'migration_schema' +~~~ -- `COPY FROM` enables your tables to remain online and accessible. However, it is slower than using [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}). -- `COPY FROM` does not support compression. +{{site.data.alerts.callout_info}} +[`--schema-filter`]({% link molt/molt-fetch-commands-and-flags.md %}#schema-filter) does not apply to MySQL sources because MySQL tables belong directly to the database specified in the connection string, not to a separate schema. +{{site.data.alerts.end}} + +[`--table-filter`]({% link molt/molt-fetch-commands-and-flags.md %}#table-filter) and [`--table-exclusion-filter`]({% link molt/molt-fetch-commands-and-flags.md %}#table-exclusion-filter) specify tables to include and exclude from the migration, respectively, formatted as POSIX regex strings. For example, to move every source table that has "user" in the table name and exclude every source table that has "temp" in the table name: + +{% include_cached copy-clipboard.html %} +~~~ +--table-filter '.*user.*' --table-exclusion-filter '.*temp.*' +~~~ + +#### Row-level filtering + +Use [`--filter-path`]({% link molt/molt-fetch-commands-and-flags.md %}#filter-path) to specify the path to a JSON file that defines row-level filtering for data load. This enables you to move a subset of data in a table, rather than all data in the table. To apply row-level filters during replication, use [MOLT Replicator]({% link molt/molt-replicator.md %}) with userscripts. + +{% include_cached copy-clipboard.html %} +~~~ +--filter-path 'data-filter.json' +~~~ + +The JSON file should contain one or more entries in `filters`, each with a `resource_specifier` (`schema` and `table`) and a SQL expression `expr`. For example, the following example exports only rows from `migration_schema.t1` where `v > 100`: + +~~~ json +{ + "filters": [ + { + "resource_specifier": { + "schema": "migration_schema", + "table": "t1" + }, + "expr": "v > 100" + } + ] +} +~~~ + +`expr` is case-sensitive and must be valid in your source dialect. For example, when using Oracle as the source, quote all identifiers and escape embedded quotes: + +~~~ json +{ + "filters": [ + { + "resource_specifier": { + "schema": "C##FETCHORACLEFILTERTEST", + "table": "FILTERTBL" + }, + "expr": "ABS(\"X\") > 10 AND CEIL(\"X\") < 100 AND FLOOR(\"X\") > 0 AND ROUND(\"X\", 2) < 100.00 AND TRUNC(\"X\", 0) > 0 AND MOD(\"X\", 2) = 0 AND FLOOR(\"X\" / 3) > 1" + } + ] +} +~~~ {{site.data.alerts.callout_info}} -`COPY FROM` is also used for [direct copy](#direct-copy). +If the expression references columns that are not indexed, MOLT Fetch will emit a warning like: `filter expression 'v > 100' contains column 'v' which is not indexed. This may lead to performance issues.` {{site.data.alerts.end}} -### Table sharding +{% comment %} +#### `--filter-path` userscript for replication + +To use `--filter-path` with replication, create and save a TypeScript userscript (e.g., `filter-script.ts`). The following script ensures that only rows where `v > 100` are replicated to `defaultdb.migration_schema.t1`: + +{% include_cached copy-clipboard.html %} +~~~ ts +import * as api from "replicator@v1"; +function disp(doc, meta) { + if (Number(doc.v) > 100) { + return { "defaultdb.migration_schema.t1" : [ doc ] }; + } +} +// Always put target schema. +api.configureSource("defaultdb.migration_schema", { + deletesTo: disp, + dispatch: disp, +}); +~~~ + +Apply the userscript with the `--userscript` replication flag: + +{% include_cached copy-clipboard.html %} +~~~ +--userscript 'filter-script.ts' +~~~ +{% endcomment %} + +### Shard tables for concurrent export During the [data export phase](#data-export-phase), MOLT Fetch can divide large tables into multiple shards for concurrent export. -To control the number of shards created per table, use the `--export-concurrency` flag. For example: +To control the number of shards created per table, use the [`--export-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#export-concurrency) flag. For example: {% include_cached copy-clipboard.html %} ~~~ @@ -251,14 +227,14 @@ To control the number of shards created per table, use the `--export-concurrency ~~~ {{site.data.alerts.callout_success}} -For performance considerations with concurrency settings, refer to [Best practices](#best-practices). +For performance considerations with concurrency settings, refer to [Best practices]({% link molt/molt-fetch-best-practices.md %}). {{site.data.alerts.end}} Two sharding mechanisms are available: - **Range-based sharding (default):** Tables are divided based on numerical ranges found in primary key values. Only tables with [`INT`]({% link {{ site.current_cloud_version }}/int.md %}), [`FLOAT`]({% link {{ site.current_cloud_version }}/float.md %}), or [`UUID`]({% link {{ site.current_cloud_version }}/uuid.md %}) primary keys can use range-based sharding. Tables with other primary key data types export as a single shard. -- **Stats-based sharding (PostgreSQL only):** Enable with [`--use-stats-based-sharding`](#global-flags) for PostgreSQL 11+ sources. Tables are divided by analyzing the [`pg_stats`](https://www.postgresql.org/docs/current/view-pg-stats.htm) view to create more evenly distributed shards, up to a maximum of 200 shards. Primary keys of any data type are supported. +- **Stats-based sharding (PostgreSQL only):** Enable with [`--use-stats-based-sharding`]({% link molt/molt-fetch-commands-and-flags.md %}#use-stats-based-sharding) for PostgreSQL 11+ sources. Tables are divided by analyzing the [`pg_stats`](https://www.postgresql.org/docs/current/view-pg-stats.htm) view to create more evenly distributed shards, up to a maximum of 200 shards. Primary keys of any data type are supported. Stats-based sharding requires that the user has `SELECT` permissions on source tables and on each table's `pg_stats` view. The latter permission is automatically granted to users that can read the table. @@ -280,7 +256,7 @@ Large tables may take time to analyze, but `ANALYZE` can run in the background. Migration without running `ANALYZE` will still work, but shard distribution may be less even. {{site.data.alerts.end}} -When using `--use-stats-based-sharding`, monitor the log output for each table you want to migrate. +When using [`--use-stats-based-sharding`]({% link molt/molt-fetch-commands-and-flags.md %}#use-stats-based-sharding), monitor the log output for each table you want to migrate. If stats-based sharding is successful on a table, MOLT logs the following `INFO` message: @@ -294,25 +270,25 @@ If stats-based sharding fails on a table, MOLT logs the following `WARNING` mess Warning: failed to shard table {table_name} using stats based sharding: {reason_for_failure}, falling back to non stats based sharding ~~~ -The number of shards is dependent on the number of distinct values in the first primary key column of the table to be migrated. If this is different from the number of shards requested with `--export-concurrency`, MOLT logs the following `WARNING` and continues with the migration: +The number of shards is dependent on the number of distinct values in the first primary key column of the table to be migrated. If this is different from the number of shards requested with [`--export-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#export-concurrency), MOLT logs the following `WARNING` and continues with the migration: ~~~ number of shards formed: {num_shards_formed} is not equal to number of shards requested: {num_shards_requested} for table {table_name} ~~~ -Because stats-based sharding analyzes the entire table, running `--use-stats-based-sharding` with [`--filter-path`](#global-flags) (refer to [Selective data movement](#selective-data-movement)) will cause imbalanced shards to form. +Because stats-based sharding analyzes the entire table, running [`--use-stats-based-sharding`]({% link molt/molt-fetch-commands-and-flags.md %}#use-stats-based-sharding) with [`--filter-path`]({% link molt/molt-fetch-commands-and-flags.md %}#filter-path) (refer to [Selective data movement](#select-data-to-migrate)) will cause imbalanced shards to form. -### Data path +### Define intermediate storage MOLT Fetch can move the source data to CockroachDB via [cloud storage](#bucket-path), a [local file server](#local-path), or [directly](#direct-copy) without an intermediate store. #### Bucket path {{site.data.alerts.callout_success}} -Only the path specified in `--bucket-path` is used. Query parameters, such as credentials, are ignored. To authenticate cloud storage, follow the steps in [Secure cloud storage](#cloud-storage-security). +Only the path specified in [`--bucket-path`]({% link molt/molt-fetch-commands-and-flags.md %}#bucket-path) is used. Query parameters, such as credentials, are ignored. To authenticate cloud storage, follow the steps in [Secure cloud storage]({% link molt/molt-fetch-best-practices.md %}#cloud-storage-security). {{site.data.alerts.end}} -`--bucket-path` instructs MOLT Fetch to write intermediate files to a path within [Google Cloud Storage](https://cloud.google.com/storage/docs/buckets), [Amazon S3](https://aws.amazon.com/s3/), or [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) to which you have the necessary permissions. Use additional [flags](#global-flags), shown in the following examples, to specify authentication or region parameters as required for bucket access. +[`--bucket-path`]({% link molt/molt-fetch-commands-and-flags.md %}#bucket-path) instructs MOLT Fetch to write intermediate files to a path within [Google Cloud Storage](https://cloud.google.com/storage/docs/buckets), [Amazon S3](https://aws.amazon.com/s3/), or [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) to which you have the necessary permissions. Use additional [flags]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags), shown in the following examples, to specify authentication or region parameters as required for bucket access. Connect to a Google Cloud Storage bucket with [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}#google-cloud-storage-implicit) and [assume role]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}#set-up-google-cloud-storage-assume-role): @@ -332,7 +308,7 @@ Connect to an Amazon S3 bucket and explicitly specify the `ap_south-1` region: ~~~ {{site.data.alerts.callout_info}} -When `--import-region` is set, `IMPORT INTO` must be used for [data movement](#data-load-mode). +When [`--import-region`]({% link molt/molt-fetch-commands-and-flags.md %}#import-region) is set, `IMPORT INTO` must be used for [data movement](#import-into-vs-copy-from). {{site.data.alerts.end}} Connect to an Azure Blob Storage container with [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}?filters=azure#azure-blob-storage-implicit-authentication): @@ -345,7 +321,7 @@ Connect to an Azure Blob Storage container with [implicit authentication]({% lin #### Local path -`--local-path` instructs MOLT Fetch to write intermediate files to a path within a [local file server]({% link {{site.current_cloud_version}}/use-a-local-file-server.md %}). `local-path-listen-addr` specifies the address of the local file server. For example: +[`--local-path`]({% link molt/molt-fetch-commands-and-flags.md %}#local-path) instructs MOLT Fetch to write intermediate files to a path within a [local file server]({% link {{site.current_cloud_version}}/use-a-local-file-server.md %}). [`--local-path-listen-addr`]({% link molt/molt-fetch-commands-and-flags.md %}#local-path-listen-addr) specifies the address of the local file server. For example: {% include_cached copy-clipboard.html %} ~~~ @@ -353,9 +329,9 @@ Connect to an Azure Blob Storage container with [implicit authentication]({% lin --local-path-listen-addr 'localhost:3000' ~~~ -In some cases, CockroachDB will not be able to use the local address specified by `--local-path-listen-addr`. This will depend on where CockroachDB is deployed, the runtime OS, and the source dialect. +In some cases, CockroachDB will not be able to use the local address specified by [`--local-path-listen-addr`]({% link molt/molt-fetch-commands-and-flags.md %}#local-path-listen-addr). This will depend on where CockroachDB is deployed, the runtime OS, and the source dialect. -For example, if you are migrating to CockroachDB {{ site.data.products.cloud }}, such that the {{ site.data.products.cloud }} cluster is in a different physical location than the machine running `molt fetch`, then CockroachDB cannot reach an address such as `localhost:3000`. In these situations, use `--local-path-crdb-access-addr` to specify an address for the local file server that is **publicly accessible**. For example: +For example, if you are migrating to CockroachDB {{ site.data.products.cloud }}, such that the {{ site.data.products.cloud }} cluster is in a different physical location than the machine running `molt fetch`, then CockroachDB cannot reach an address such as `localhost:3000`. In these situations, use [`--local-path-crdb-access-addr`]({% link molt/molt-fetch-commands-and-flags.md %}#local-path-crdb-access-addr) to specify an address for the local file server that is **publicly accessible**. For example: {% include_cached copy-clipboard.html %} ~~~ @@ -370,115 +346,38 @@ For example, if you are migrating to CockroachDB {{ site.data.products.cloud }}, #### Direct copy -`--direct-copy` specifies that MOLT Fetch should use `COPY FROM` to move the source data directly to CockroachDB without an intermediate store: +[`--direct-copy`]({% link molt/molt-fetch-commands-and-flags.md %}#direct-copy) specifies that MOLT Fetch should use `COPY FROM` to move the source data directly to CockroachDB without an intermediate store: -- Because the data is held in memory, the machine must have sufficient RAM for the data currently in flight: +- Because the data is held in memory, the machine must have sufficient RAM for the data currently in flight: ~~~ average size of each row * --row-batch-size * --export-concurrency * --table-concurrency ~~~ -- Direct copy does not support compression or [continuation](#fetch-continuation). -- The [`--use-copy`](#data-load-mode) flag is redundant with `--direct-copy`. - -### Schema and table selection - -By default, MOLT Fetch moves all data from the [`--source`](#source-and-target-databases) database to CockroachDB. Use the following flags to move a subset of data. - -`--schema-filter` specifies a range of schema objects to move to CockroachDB, formatted as a POSIX regex string. For example, to move every table in the source database's `migration_schema` schema: - -{% include_cached copy-clipboard.html %} -~~~ ---schema-filter 'migration_schema' -~~~ - -{{site.data.alerts.callout_info}} -`--schema-filter` does not apply to MySQL sources because MySQL tables belong directly to the database specified in the connection string, not to a separate schema. -{{site.data.alerts.end}} - -`--table-filter` and `--table-exclusion-filter` specify tables to include and exclude from the migration, respectively, formatted as POSIX regex strings. For example, to move every source table that has "user" in the table name and exclude every source table that has "temp" in the table name: - -{% include_cached copy-clipboard.html %} -~~~ ---table-filter '.*user.*' --table-exclusion-filter '.*temp.*' -~~~ - -### Selective data movement +- Direct copy does not support compression or [continuation](#continue-molt-fetch-after-interruption). +- The [`--use-copy`](#import-into-vs-copy-from) flag is redundant with [`--direct-copy`]({% link molt/molt-fetch-commands-and-flags.md %}#direct-copy). -Use `--filter-path` to specify the path to a JSON file that defines row-level filtering for data load. This enables you to move a subset of data in a table, rather than all data in the table. To apply row-level filters during replication, use [MOLT Replicator]({% link molt/molt-replicator.md %}) with userscripts. +### `IMPORT INTO` vs. `COPY FROM` -{% include_cached copy-clipboard.html %} -~~~ ---filter-path 'data-filter.json' -~~~ +MOLT Fetch can use either [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}) or [`COPY FROM`]({% link {{site.current_cloud_version}}/copy.md %}) to load data into CockroachDB. -The JSON file should contain one or more entries in `filters`, each with a `resource_specifier` (`schema` and `table`) and a SQL expression `expr`. For example, the following example exports only rows from `migration_schema.t1` where `v > 100`: +By default, MOLT Fetch uses `IMPORT INTO`: -~~~ json -{ - "filters": [ - { - "resource_specifier": { - "schema": "migration_schema", - "table": "t1" - }, - "expr": "v > 100" - } - ] -} -~~~ +- `IMPORT INTO` achieves the highest throughput, but [requires taking the CockroachDB tables **offline**]({% link {{site.current_cloud_version}}/import-into.md %}#considerations) to achieve its import speed. Tables are taken back online once an [import job]({% link {{site.current_cloud_version}}/import-into.md %}#view-and-control-import-jobs) completes successfully. See [Best practices]({% link molt/molt-fetch-best-practices.md %}). +- `IMPORT INTO` supports compression using the [`--compression`]({% link molt/molt-fetch-commands-and-flags.md %}#compression) flag, which reduces the amount of storage used. -`expr` is case-sensitive and must be valid in your source dialect. For example, when using Oracle as the source, quote all identifiers and escape embedded quotes: +[`--use-copy`]({% link molt/molt-fetch-commands-and-flags.md %}#use-copy) configures MOLT Fetch to use `COPY FROM`: -~~~ json -{ - "filters": [ - { - "resource_specifier": { - "schema": "C##FETCHORACLEFILTERTEST", - "table": "FILTERTBL" - }, - "expr": "ABS(\"X\") > 10 AND CEIL(\"X\") < 100 AND FLOOR(\"X\") > 0 AND ROUND(\"X\", 2) < 100.00 AND TRUNC(\"X\", 0) > 0 AND MOD(\"X\", 2) = 0 AND FLOOR(\"X\" / 3) > 1" - } - ] -} -~~~ +- `COPY FROM` enables your tables to remain online and accessible. However, it is slower than using [`IMPORT INTO`]({% link {{site.current_cloud_version}}/import-into.md %}). +- `COPY FROM` does not support compression. {{site.data.alerts.callout_info}} -If the expression references columns that are not indexed, MOLT Fetch will emit a warning like: `filter expression ‘v > 100' contains column ‘v' which is not indexed. This may lead to performance issues.` +`COPY FROM` is also used for [direct copy](#direct-copy). {{site.data.alerts.end}} -{% comment %} -#### `--filter-path` userscript for replication - -To use `--filter-path` with replication, create and save a TypeScript userscript (e.g., `filter-script.ts`). The following script ensures that only rows where `v > 100` are replicated to `defaultdb.migration_schema.t1`: - -{% include_cached copy-clipboard.html %} -~~~ ts -import * as api from "replicator@v1"; -function disp(doc, meta) { - if (Number(doc.v) > 100) { - return { "defaultdb.migration_schema.t1" : [ doc ] }; - } -} -// Always put target schema. -api.configureSource("defaultdb.migration_schema", { - deletesTo: disp, - dispatch: disp, -}); -~~~ - -Apply the userscript with the `--userscript` replication flag: - -{% include_cached copy-clipboard.html %} -~~~ ---userscript 'filter-script.ts' -~~~ -{% endcomment %} +### Handle target tables -### Target table handling - -`--table-handling` defines how MOLT Fetch loads data on the CockroachDB tables that [match the selection](#schema-and-table-selection). +[`--table-handling`]({% link molt/molt-fetch-commands-and-flags.md %}#table-handling) defines how MOLT Fetch loads data on the CockroachDB tables that [match the selection](#schema-and-table-selection). To load the data without changing the existing data in the tables, use `none`: @@ -505,21 +404,21 @@ When using the `drop-on-target-and-recreate` option, MOLT Fetch creates a new Co #### Mismatch handling -If either [`none`](#target-table-handling) or [`truncate-if-exists`](#target-table-handling) is set, `molt fetch` loads data into the existing tables on the target CockroachDB database. If the target schema mismatches the source schema, `molt fetch` will exit early in certain cases, and will need to be re-run from the beginning. For details, refer to [Fetch exits early due to mismatches](#fetch-exits-early-due-to-mismatches). +If either [`none`](#handle-target-tables) or [`truncate-if-exists`](#handle-target-tables) is set, `molt fetch` loads data into the existing tables on the target CockroachDB database. If the target schema mismatches the source schema, `molt fetch` will exit early in certain cases, and will need to be re-run from the beginning. For details, refer to [Fetch exits early due to mismatches]({% link molt/molt-fetch-troubleshooting.md %}#fetch-exits-early-due-to-mismatches). {{site.data.alerts.callout_info}} -This does not apply when [`drop-on-target-and-recreate`](#target-table-handling) is specified, since this option automatically creates a compatible CockroachDB schema. +This does not apply when [`drop-on-target-and-recreate`](#handle-target-tables) is specified, since this option automatically creates a compatible CockroachDB schema. {{site.data.alerts.end}} #### Skip primary key matching -`--skip-pk-check` removes the [requirement that source and target tables share matching primary keys](#fetch-exits-early-due-to-mismatches) for data load. When this flag is set: +[`--skip-pk-check`]({% link molt/molt-fetch-commands-and-flags.md %}#skip-pk-check) removes the [requirement that source and target tables share matching primary keys]({% link molt/molt-fetch-troubleshooting.md %}#fetch-exits-early-due-to-mismatches) for data load. When this flag is set: - The data load proceeds even if the source or target table lacks a primary key, or if their primary key columns do not match. -- [Table sharding](#table-sharding) is disabled. Each table is exported in a single batch within one shard, bypassing `--export-concurrency` and `--row-batch-size`. As a result, memory usage and execution time may increase due to full table scans. +- [Table sharding](#shard-tables-for-concurrent-export) is disabled. Each table is exported in a single batch within one shard, bypassing [`--export-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#export-concurrency) and [`--row-batch-size`]({% link molt/molt-fetch-commands-and-flags.md %}#row-batch-size). As a result, memory usage and execution time may increase due to full table scans. - If the source table contains duplicate rows but the target has [`PRIMARY KEY`]({% link {{ site.current_cloud_version }}/primary-key.md %}) or [`UNIQUE`]({% link {{ site.current_cloud_version }}/unique.md %}) constraints, duplicate rows are deduplicated during import. -When `--skip-pk-check` is set, all tables are treated as if they lack a primary key, and are thus exported in a single unsharded batch. To avoid performance issues, use this flag with `--table-filter` to target only tables **without** a primary key. +When [`--skip-pk-check`]({% link molt/molt-fetch-commands-and-flags.md %}#skip-pk-check) is set, all tables are treated as if they lack a primary key, and are thus exported in a single unsharded batch. To avoid performance issues, use this flag with [`--table-filter`]({% link molt/molt-fetch-commands-and-flags.md %}#table-filter) to target only tables **without** a primary key. For example: @@ -531,7 +430,7 @@ molt fetch \ --skip-pk-check ~~~ -Example log output when `--skip-pk-check` is enabled: +Example log output when [`--skip-pk-check`]({% link molt/molt-fetch-commands-and-flags.md %}#skip-pk-check) is enabled: ~~~json {"level":"info","message":"sharding is skipped for table public.nopktbl - flag skip-pk-check is specified and thus no PK for source table is specified"} @@ -539,65 +438,9 @@ Example log output when `--skip-pk-check` is enabled: #### Type mapping -If [`drop-on-target-and-recreate`](#target-table-handling) is set, MOLT Fetch automatically creates a CockroachDB schema that is compatible with the source data. The column types are determined as follows: - -- PostgreSQL types are mapped to existing CockroachDB [types]({% link {{site.current_cloud_version}}/data-types.md %}) that have the same [`OID`]({% link {{site.current_cloud_version}}/oid.md %}). -- The following MySQL types are mapped to corresponding CockroachDB types: - - | MySQL type | CockroachDB type | Notes | - |-----------------------------------------------------|-------------------------------------------------------------------------------------------|--------------------------------------------------------------| - | `CHAR`, `CHARACTER`, `VARCHAR`, `NCHAR`, `NVARCHAR` | [`VARCHAR`]({% link {{site.current_cloud_version}}/string.md %}) | Varying-length string; raises warning if BYTE semantics used | - | `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT` | [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) | Unlimited-length string | - | `GEOMETRY` | [`GEOMETRY`]({% link {{site.current_cloud_version}}/architecture/glossary.md %}#geometry) | Spatial type (PostGIS-style) | - | `LINESTRING` | [`LINESTRING`]({% link {{site.current_cloud_version}}/linestring.md %}) | Spatial type (PostGIS-style) | - | `POINT` | [`POINT`]({% link {{site.current_cloud_version}}/point.md %}) | Spatial type (PostGIS-style) | - | `POLYGON` | [`POLYGON`]({% link {{site.current_cloud_version}}/polygon.md %}) | Spatial type (PostGIS-style) | - | `MULTIPOINT` | [`MULTIPOINT`]({% link {{site.current_cloud_version}}/multipoint.md %}) | Spatial type (PostGIS-style) | - | `MULTILINESTRING` | [`MULTILINESTRING`]({% link {{site.current_cloud_version}}/multilinestring.md %}) | Spatial type (PostGIS-style) | - | `MULTIPOLYGON` | [`MULTIPOLYGON`]({% link {{site.current_cloud_version}}/multipolygon.md %}) | Spatial type (PostGIS-style) | - | `GEOMETRYCOLLECTION`, `GEOMCOLLECTION` | [`GEOMETRYCOLLECTION`]({% link {{site.current_cloud_version}}/geometrycollection.md %}) | Spatial type (PostGIS-style) | - | `JSON` | [`JSONB`]({% link {{site.current_cloud_version}}/jsonb.md %}) | CRDB's native JSON format | - | `TINYINT`, `INT1` | [`INT2`]({% link {{site.current_cloud_version}}/int.md %}) | 2-byte integer | - | `BLOB` | [`BYTES`]({% link {{site.current_cloud_version}}/bytes.md %}) | Binary data | - | `SMALLINT`, `INT2` | [`INT2`]({% link {{site.current_cloud_version}}/int.md %}) | 2-byte integer | - | `MEDIUMINT`, `INT`, `INTEGER`, `INT4` | [`INT4`]({% link {{site.current_cloud_version}}/int.md %}) | 4-byte integer | - | `BIGINT`, `INT8` | [`INT`]({% link {{site.current_cloud_version}}/int.md %}) | 8-byte integer | - | `FLOAT` | [`FLOAT4`]({% link {{site.current_cloud_version}}/float.md %}) | 32-bit float | - | `DOUBLE` | [`FLOAT`]({% link {{site.current_cloud_version}}/float.md %}) | 64-bit float | - | `DECIMAL`, `NUMERIC`, `REAL` | [`DECIMAL`]({% link {{site.current_cloud_version}}/decimal.md %}) | Validates scale ≤ precision; warns if precision > 19 | - | `BINARY`, `VARBINARY` | [`BYTES`]({% link {{site.current_cloud_version}}/bytes.md %}) | Binary data | - | `DATETIME` | [`TIMESTAMP`]({% link {{site.current_cloud_version}}/timestamp.md %}) | Date and time (no time zone) | - | `TIMESTAMP` | [`TIMESTAMPTZ`]({% link {{site.current_cloud_version}}/timestamp.md %}) | Date and time with time zone | - | `TIME` | [`TIME`]({% link {{site.current_cloud_version}}/time.md %}) | Time of day (no date) | - | `BIT` | [`VARBIT`]({% link {{site.current_cloud_version}}/bit.md %}) | Variable-length bit array | - | `DATE` | [`DATE`]({% link {{site.current_cloud_version}}/date.md %}) | Date only (no time) | - | `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB` | [`BYTES`]({% link {{site.current_cloud_version}}/bytes.md %}) | Binary data | - | `BOOL`, `BOOLEAN` | [`BOOL`]({% link {{site.current_cloud_version}}/bool.md %}) | Boolean | - -- The following Oracle types are mapped to CockroachDB types: - - | Oracle type(s) | CockroachDB type | Notes | - |---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| - | `NCHAR`, `CHAR`, `CHARACTER` | [`CHAR`]({% link {{site.current_cloud_version}}/string.md %})(n) or [`CHAR`]({% link {{site.current_cloud_version}}/string.md %}) | Fixed-length character; falls back to unbounded if length not specified | - | `VARCHAR`, `VARCHAR2`, `NVARCHAR2` | [`VARCHAR`]({% link {{site.current_cloud_version}}/string.md %})(n) or [`VARCHAR`]({% link {{site.current_cloud_version}}/string.md %}) | Varying-length string; raises warning if BYTE semantics used | - | `STRING` | [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) | Unlimited-length string | - | `SMALLINT` | [`INT2`]({% link {{site.current_cloud_version}}/int.md %}) | 2-byte integer | - | `INTEGER`, `INT`, `SIMPLE_INTEGER` | [`INT4`]({% link {{site.current_cloud_version}}/int.md %}) | 4-byte integer | - | `LONG` | [`INT8`]({% link {{site.current_cloud_version}}/int.md %}) | 8-byte integer | - | `FLOAT`, `BINARY_FLOAT`, `REAL` | [`FLOAT4`]({% link {{site.current_cloud_version}}/float.md %}) | 32-bit float | - | `DOUBLE`, `BINARY_DOUBLE` | [`FLOAT8`]({% link {{site.current_cloud_version}}/float.md %}) | 64-bit float | - | `DEC`, `NUMBER`, `DECIMAL`, `NUMERIC` | [`DECIMAL`]({% link {{site.current_cloud_version}}/decimal.md %})(p, s) or [`DECIMAL`]({% link {{site.current_cloud_version}}/decimal.md %}) | Validates scale ≤ precision; warns if precision > 19 | - | `DATE` | [`DATE`]({% link {{site.current_cloud_version}}/date.md %}) | Date only (no time) | - | `BLOB`, `RAW`, `LONG RAW` | [`BYTES`]({% link {{site.current_cloud_version}}/bytes.md %}) | Binary data | - | `JSON` | [`JSONB`]({% link {{site.current_cloud_version}}/jsonb.md %}) | CRDB's native JSON format | - | `CLOB`, `NCLOB` | [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) | Treated as large text | - | `BOOLEAN` | [`BOOL`]({% link {{site.current_cloud_version}}/bool.md %}) | Boolean | - | `TIMESTAMP` | [`TIMESTAMP`]({% link {{site.current_cloud_version}}/timestamp.md %}) or [`TIMESTAMPTZ`]({% link {{site.current_cloud_version}}/timestamp.md %}) | If `WITH TIME ZONE` → `TIMESTAMPTZ`, else `TIMESTAMP` | - | `ROWID`, `UROWID` | [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) | Treated as opaque identifier | - | `SDO_GEOMETRY` | [`GEOMETRY`]({% link {{site.current_cloud_version}}/architecture/glossary.md %}#geometry) | Spatial type (PostGIS-style) | - | `XMLTYPE` | [`STRING`]({% link {{site.current_cloud_version}}/string.md %}) | Stored as text | - -- To override the default mappings for automatic schema creation, you can map source to target CockroachDB types explicitly. These are defined in the JSON file indicated by the `--type-map-file` flag. The allowable custom mappings are valid CockroachDB aliases, casts, and the following mappings specific to MOLT Fetch and [Verify]({% link molt/molt-verify.md %}): +If [`drop-on-target-and-recreate`](#handle-target-tables) is set, MOLT Fetch automatically creates a CockroachDB schema that is compatible with the source data. The column types are determined by [MOLT's default type mappings]({% link molt/molt-type-mapping.md %}). + +- To override the default mappings for automatic schema creation, you can map source to target CockroachDB types explicitly. These are defined in the JSON file indicated by the [`--type-map-file`]({% link molt/molt-fetch-commands-and-flags.md %}#type-map-file) flag. The allowable custom mappings are valid CockroachDB aliases, casts, and the following mappings specific to MOLT Fetch and [Verify]({% link molt/molt-verify.md %}): - [`TIMESTAMP`]({% link {{site.current_cloud_version}}/timestamp.md %}) <> [`TIMESTAMPTZ`]({% link {{site.current_cloud_version}}/timestamp.md %}) - [`VARCHAR`]({% link {{site.current_cloud_version}}/string.md %}) <> [`UUID`]({% link {{site.current_cloud_version}}/uuid.md %}) @@ -607,7 +450,7 @@ If [`drop-on-target-and-recreate`](#target-table-handling) is set, MOLT Fetch au - [`JSONB`]({% link {{site.current_cloud_version}}/jsonb.md %}) <> [`TEXT`]({% link {{site.current_cloud_version}}/string.md %}) - [`INET`]({% link {{site.current_cloud_version}}/inet.md %}) <> [`TEXT`]({% link {{site.current_cloud_version}}/string.md %}) -`--type-map-file` specifies the path to the JSON file containing the explicit type mappings. For example: +[`--type-map-file`]({% link molt/molt-fetch-commands-and-flags.md %}#type-map-file) specifies the path to the JSON file containing the explicit type mappings. For example: {% include_cached copy-clipboard.html %} ~~~ @@ -641,7 +484,7 @@ The following JSON example defines two type mappings: - `source_type` specifies the source type to be mapped. - `crdb_type` specifies the target CockroachDB [type]({% link {{ site.current_cloud_version }}/data-types.md %}) to be mapped. -### Transformations +### Define transformations You can define transformation rules to be performed on the target database during the fetch task. These can be used to: @@ -650,7 +493,7 @@ You can define transformation rules to be performed on the target database durin - Rename tables on the target database. - Rename database schemas. -Transformation rules are defined in the JSON file indicated by the `--transformations-file` flag. For example: +Transformation rules are defined in the JSON file indicated by the [`--transformations-file`]({% link molt/molt-fetch-commands-and-flags.md %}#transformations-file) flag. For example: {% include_cached copy-clipboard.html %} ~~~ @@ -736,8 +579,8 @@ The following JSON example defines three transformation rules: rule `1` [maps co For n-to-1 mappings: - - Use [`--use-copy`](#data-load-mode) or [`--direct-copy`](#direct-copy) for data movement. - - Manually create the target table. Do not use [`--table-handling drop-on-target-and-recreate`](#target-table-handling). + - Use [`--use-copy`](#import-into-vs-copy-from) or [`--direct-copy`](#direct-copy) for data movement. + - Manually create the target table. Do not use [`--table-handling drop-on-target-and-recreate`](#handle-target-tables). [Example rule `2`](#transformation-rules-example) maps all table names with prefix `charges_part` to a single `charges` table on CockroachDB (an n-to-1 mapping). This assumes that all matching `charges_part.*` tables have the same table definition: @@ -782,7 +625,7 @@ Each rule is applied in the order it is defined. If two rules overlap, the later To verify that the logging shows that the computed columns are being created: -When running `molt fetch`, set `--logging debug` and look for `ALTER TABLE ... ADD COLUMN` statements with the `STORED` or `VIRTUAL` keywords in the log output: +When running `molt fetch`, set [`--logging`]({% link molt/molt-fetch-commands-and-flags.md %}#logging) `debug` and look for `ALTER TABLE ... ADD COLUMN` statements with the `STORED` or `VIRTUAL` keywords in the log output: ~~~ json {"level":"debug","time":"2024-07-22T12:01:51-04:00","message":"running: ALTER TABLE IF EXISTS public.computed ADD COLUMN computed_col INT8 NOT NULL AS ((col1 + col2)) STORED"} @@ -804,7 +647,7 @@ SHOW CREATE TABLE computed; | ) ~~~ -### Fetch continuation +## Continue MOLT Fetch after interruption If MOLT Fetch fails while loading data into CockroachDB from intermediate files, it exits with an error message, fetch ID, and [continuation token](#list-active-continuation-tokens) for each table that failed to load on the target database. You can use this information to continue the task from the *continuation point* where it was interrupted. @@ -817,14 +660,14 @@ Continuation is only possible under the following conditions: Only one fetch ID and set of continuation tokens, each token corresponding to a table, are active at any time. See [List active continuation tokens](#list-active-continuation-tokens). {{site.data.alerts.end}} -To retry all data starting from the continuation point, reissue the `molt fetch` command and include the `--fetch-id`. +To retry all data starting from the continuation point, reissue the `molt fetch` command and include the [`--fetch-id`]({% link molt/molt-fetch-commands-and-flags.md %}#fetch-id). {% include_cached copy-clipboard.html %} ~~~ --fetch-id d44762e5-6f70-43f8-8e15-58b4de10a007 ~~~ -To retry a specific table that failed, include both `--fetch-id` and `--continuation-token`. The latter flag specifies a token string that corresponds to a specific table on the source database. A continuation token is written in the `molt fetch` output for each failed table. If the fetch task encounters a subsequent error, it generates a new token for each failed table. See [List active continuation tokens](#list-active-continuation-tokens). +To retry a specific table that failed, include both [`--fetch-id`]({% link molt/molt-fetch-commands-and-flags.md %}#fetch-id) and [`--continuation-token`]({% link molt/molt-fetch-commands-and-flags.md %}#continuation-token). The latter flag specifies a token string that corresponds to a specific table on the source database. A continuation token is written in the `molt fetch` output for each failed table. If the fetch task encounters a subsequent error, it generates a new token for each failed table. See [List active continuation tokens](#list-active-continuation-tokens). {{site.data.alerts.callout_info}} This will retry only the table that corresponds to the continuation token. If the fetch task succeeds, there may still be source data that is not yet loaded into CockroachDB. @@ -836,7 +679,7 @@ This will retry only the table that corresponds to the continuation token. If th --continuation-token 011762e5-6f70-43f8-8e15-58b4de10a007 ~~~ -To retry all data starting from a specific file, include both `--fetch-id` and `--continuation-file-name`. The latter flag specifies the filename of an intermediate file in [cloud or local storage](#data-path). All filenames are prepended with `part_` and have the `.csv.gz` or `.csv` extension, depending on compression type (gzip by default). For example: +To retry all data starting from a specific file, include both [`--fetch-id`]({% link molt/molt-fetch-commands-and-flags.md %}#fetch-id) and [`--continuation-file-name`]({% link molt/molt-fetch-commands-and-flags.md %}#continuation-file-name). The latter flag specifies the filename of an intermediate file in [cloud or local storage](#define-intermediate-storage). All filenames are prepended with `part_` and have the `.csv.gz` or `.csv` extension, depending on compression type (gzip by default). For example: {% include_cached copy-clipboard.html %} ~~~ @@ -848,9 +691,9 @@ To retry all data starting from a specific file, include both `--fetch-id` and ` Continuation is not possible when using [direct copy](#direct-copy). {{site.data.alerts.end}} -#### List active continuation tokens +### List active continuation tokens -To view all active continuation tokens, issue a `molt fetch tokens list` command along with `--conn-string`, which specifies the [connection string]({% link {{site.current_cloud_version}}/connection-parameters.md %}#connect-using-a-url) for the target CockroachDB database. For example: +To view all active continuation tokens, issue a `molt fetch tokens list` command along with [`--conn-string`]({% link molt/molt-fetch-commands-and-flags.md %}#conn-string), which specifies the [connection string]({% link {{site.current_cloud_version}}/connection-parameters.md %}#connect-using-a-url) for the target CockroachDB database. For example: {% include_cached copy-clipboard.html %} ~~~ shell @@ -867,9 +710,9 @@ molt fetch tokens list \ Continuation Tokens. ~~~ -### CDC cursor +## Enable replication -A change data capture (CDC) cursor is written to the output as `cdc_cursor` at the beginning and end of the fetch task. +A change data capture (CDC) cursor is written to the MOLT Fetch output as `cdc_cursor` at the beginning and end of the fetch task. For MySQL: @@ -887,35 +730,19 @@ Use the `cdc_cursor` value as the checkpoint for MySQL or Oracle replication wit You can also use the `cdc_cursor` value with an external change data capture (CDC) tool to continuously replicate subsequent changes from the source database to CockroachDB. -## Security - -Cockroach Labs strongly recommends the following security practices. - -### Connection security - -{% include molt/molt-secure-connection-strings.md %} - -{{site.data.alerts.callout_info}} -By default, insecure connections (i.e., `sslmode=disable` on PostgreSQL; `sslmode` not set on MySQL) are disallowed. When using an insecure connection, `molt fetch` returns an error. To override this check, you can enable the `--allow-tls-mode-disable` flag. Do this **only** when testing, or if a secure SSL/TLS connection to the source or target database is not possible. -{{site.data.alerts.end}} - -### Cloud storage security - -{% include molt/fetch-secure-cloud-storage.md %} - -## Common workflows +## Common uses ### Bulk data load +When migrating data to CockroachDB in a bulk load (without utilizing [continuous replication]({% link molt/migration-considerations-replication.md %}) to minimize system downtime), run the `molt fetch` command with the required flags, as shown below: +
@@ -1025,15 +859,13 @@ For detailed steps, refer to [Bulk load migration]({% link molt/migrate-bulk-loa
-To perform an initial data load before setting up ongoing replication with [MOLT Replicator]({% link molt/molt-replicator.md %}), run the `molt fetch` command without `--ignore-replication-check`. This captures replication checkpoints during the data load.
-
The workflow is the same as [Bulk data load](#bulk-data-load), except:
-- Exclude `--ignore-replication-check`. MOLT Fetch will query and record replication checkpoints.
+- Exclude [`--ignore-replication-check`]({% link molt/molt-fetch-commands-and-flags.md %}#ignore-replication-check). MOLT Fetch will query and record replication checkpoints.
`molt_fetch_rows_exported{table="public.users"}` | -| `molt_fetch_rows_imported` | Number of rows that have been imported from a table. For example:
`molt_fetch_rows_imported{table="public.users"}` | -| `molt_fetch_table_export_duration_ms` | Duration (in milliseconds) of a table's export. For example:
`molt_fetch_table_export_duration_ms{table="public.users"}` | -| `molt_fetch_table_import_duration_ms` | Duration (in milliseconds) of a table's import. For example:
`molt_fetch_table_import_duration_ms{table="public.users"}` | - -You can also use the [sample Grafana dashboard](https://molt.cockroachdb.com/molt/cli/grafana_dashboard.json) to view the preceding metrics. - -## Best practices - -### Test and validate - -To verify that your connections and configuration work properly, run MOLT Fetch in a staging environment before migrating any data in production. Use a test or development environment that closely resembles production. - -### Configure the source database and connection - -- To prevent connections from terminating prematurely during the [data export phase](#data-export-phase), set the following to high values on the source database: - - - **Maximum allowed number of connections.** MOLT Fetch can export data across multiple connections. The number of connections it will create is the number of shards ([`--export-concurrency`](#global-flags)) multiplied by the number of tables ([`--table-concurrency`](#global-flags)) being exported concurrently. - - {{site.data.alerts.callout_info}} - With the default numerical range sharding, only tables with [primary key]({% link {{ site.current_cloud_version }}/primary-key.md %}) types of [`INT`]({% link {{ site.current_cloud_version }}/int.md %}), [`FLOAT`]({% link {{ site.current_cloud_version }}/float.md %}), or [`UUID`]({% link {{ site.current_cloud_version }}/uuid.md %}) can be sharded. PostgreSQL users can enable [`--use-stats-based-sharding`](#global-flags) to use statistics-based sharding for tables with primary keys of any data type. For details, refer to [Table sharding](#table-sharding). - {{site.data.alerts.end}} - - - **Maximum lifetime of a connection.** - -- If a PostgreSQL database is set as a [source](#source-and-target-databases), ensure that [`idle_in_transaction_session_timeout`](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUT) on PostgreSQL is either disabled or set to a value longer than the duration of the [data export phase](#data-export-phase). Otherwise, the connection will be prematurely terminated. To estimate the time needed to export the PostgreSQL tables, you can perform a dry run and sum the value of [`molt_fetch_table_export_duration_ms`](#monitoring) for all exported tables. - -### Optimize performance - -- {% include molt/molt-drop-constraints-indexes.md %} - -- For PostgreSQL sources using [`--use-stats-based-sharding`](#global-flags), run [`ANALYZE`]({% link {{ site.current_cloud_version }}/create-statistics.md %}) on source tables before migration to ensure optimal shard distribution. This is especially important for large tables where even distribution can significantly improve export performance. - -- To prevent memory outages during `READ COMMITTED` [data export](#data-export-phase) of tables with large rows, estimate the amount of memory used to export a table: - - ~~~ - --row-batch-size * --export-concurrency * average size of the table rows - ~~~ - - If you are exporting more than one table at a time (i.e., [`--table-concurrency`](#global-flags) is set higher than `1`), add the estimated memory usage for the tables with the largest row sizes. Ensure that you have sufficient memory to run `molt fetch`, and adjust `--row-batch-size` accordingly. For details on how concurrency and sharding interact, refer to [Table sharding](#table-sharding). - -- If a table in the source database is much larger than the other tables, [filter and export the largest table](#schema-and-table-selection) in its own `molt fetch` task. Repeat this for each of the largest tables. Then export the remaining tables in another task. - -- Ensure that the machine running MOLT Fetch is large enough to handle the amount of data being migrated. Fetch performance can sometimes be limited by available resources, but should always be making progress. To identify possible resource constraints, observe the `molt_fetch_rows_exported` [metric](#monitoring) for decreases in the number of rows being processed. You can use the [sample Grafana dashboard](https://molt.cockroachdb.com/molt/cli/grafana_dashboard.json) to view metrics. For details on optimizing export performance through sharding, refer to [Table sharding](#table-sharding). - -### Import and continuation handling - -- When using [`IMPORT INTO`](#data-load-mode) during the [data import phase](#data-import-phase) to load tables into CockroachDB, if the fetch task terminates before the import job completes, the hanging import job on the target database will keep the table offline. To make this table accessible again, [manually resume or cancel the job]({% link {{site.current_cloud_version}}/import-into.md %}#view-and-control-import-jobs). Then resume `molt fetch` using [continuation](#fetch-continuation), or restart the task from the beginning. - -## Troubleshooting - -
-
-
-
-
+For detailed walkthroughs of migrations that use `molt fetch` in this way, refer to these common migration approaches:
-{% include molt/molt-troubleshooting-fetch.md %}
+- [Delta Migration]({% link molt/migration-approach-delta.md %})
+- [Streaming Migration]({% link molt/migration-approach-streaming.md %})
+- [Active-Active Migration]({% link molt/migration-approach-active-active.md %})
## See also
+- [MOLT Fetch Installation]({% link molt/molt-fetch-installation.md %})
+- [MOLT Fetch Commands and Flags]({% link molt/molt-fetch-commands-and-flags.md %})
+- [MOLT Fetch Metrics]({% link molt/molt-fetch-monitoring.md %})
+- [MOLT Fetch Best Practices]({% link molt/molt-fetch-best-practices.md %})
+- [MOLT Fetch Troubleshooting]({% link molt/molt-fetch-troubleshooting.md %})
- [Migration Overview]({% link molt/migration-overview.md %})
-- [Migration Strategy]({% link molt/migration-strategy.md %})
-- [MOLT Replicator]({% link molt/molt-replicator.md %})
-- [MOLT Verify]({% link molt/molt-verify.md %})
-- [Load and replicate]({% link molt/migrate-load-replicate.md %})
-- [Resume Replication]({% link molt/migrate-resume-replication.md %})
-- [Migration Failback]({% link molt/migrate-failback.md %})
\ No newline at end of file
+- [MOLT Replicator]({% link molt/molt-replicator.md %})
\ No newline at end of file
diff --git a/src/current/molt/molt-replicator-best-practices.md b/src/current/molt/molt-replicator-best-practices.md
new file mode 100644
index 00000000000..138ac3f4916
--- /dev/null
+++ b/src/current/molt/molt-replicator-best-practices.md
@@ -0,0 +1,147 @@
+---
+title: MOLT Replicator Best Practices
+summary: Learn best practices for using MOLT Replicator for continuous replication.
+toc: true
+docs_area: migrate
+---
+
+## Test and validate
+
+To verify that your connections and configuration work properly, run MOLT Replicator in a staging environment before replicating any data in production. Use a test or development environment that closely resembles production.
+
+## Optimize performance
+
+{% include molt/optimize-replicator-performance.md %}
+
+## Security
+
+Cockroach Labs **strongly** recommends the following:
+
+### Connection security and credentials
+
+{% include molt/molt-secure-connection-strings.md %}
+
+### CockroachDB changefeed security
+
+For failback scenarios, secure the connection from CockroachDB to MOLT Replicator using TLS certificates. Generate TLS certificates using self-signed certificates, certificate authorities like Let's Encrypt, or your organization's certificate management system.
+
+#### TLS from CockroachDB to Replicator
+
+Configure MOLT Replicator with server certificates using the [`--tlsCertificate`]({% link molt/replicator-flags.md %}#tls-certificate) and [`--tlsPrivateKey`]({% link molt/replicator-flags.md %}#tls-private-key) flags to specify the certificate and private key file paths. For example:
+
+{% include_cached copy-clipboard.html %}
+~~~ shell
+replicator start \
+--tlsCertificate ./certs/server.crt \
+--tlsPrivateKey ./certs/server.key \
+...
+~~~
+
+These server certificates must correspond to the client certificates specified in the changefeed webhook URL to ensure proper TLS handshake.
+
+Encode client certificates for changefeed webhook URLs:
+
+- Webhook URLs: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'`
+- Non-webhook contexts: Use base64 encoding only: `base64 -w 0 ca.cert`
+
+#### JWT authentication
+
+You can use JSON Web Tokens (JWT) to authorize incoming changefeed connections and restrict writes to a subset of SQL databases or user-defined schemas in the target cluster.
+
+Replicator supports JWT claims that allow writes to specific databases, schemas, or all of them. JWT tokens must be signed using RSA or EC keys. HMAC and `None` signatures are automatically rejected.
+
+To configure JWT authentication:
+
+1. Add PEM-formatted public signing keys to the `_replicator.jwt_public_keys` table in the staging database.
+
+1. To revoke a specific token, add its `jti` value to the `_replicator.jwt_revoked_ids` table in the staging database.
+
+The Replicator process re-reads these tables every minute to pick up changes.
+
+To pass the JWT token from the changefeed to the Replicator webhook sink, use the [`webhook_auth_header` option]({% link {{ site.current_cloud_version }}/create-changefeed.md %}#options):
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+CREATE CHANGEFEED ... WITH webhook_auth_header='Bearer - Enable logical replication by setting `wal_level = logical`.
- Enable [global transaction identifiers (GTID)](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) and configure binary logging. Set `binlog-row-metadata` or `binlog-row-image` to `full`.
- Configure sufficient binlog retention for migration duration.
- Install [Oracle Instant Client]({% link molt/migrate-load-replicate.md %}?filters=oracle#oracle-instant-client).
- [Enable `ARCHIVELOG` mode]({% link molt/migrate-load-replicate.md %}?filters=oracle#enable-archivelog-and-force-logging), supplemental logging for primary keys, and `FORCE LOGGING`.
- [Create sentinel table]({% link molt/migrate-load-replicate.md %}#create-source-sentinel-table) (`REPLICATOR_SENTINEL`) in source schema.
- Grant and verify [LogMiner privileges]({% link molt/migrate-load-replicate.md %}#grant-logminer-privileges).
- [Enable rangefeeds]({% link {{ site.current_cloud_version }}/create-and-configure-changefeeds.md %}#enable-rangefeeds) (`kv.rangefeed.enabled = true`) (CockroachDB {{ site.data.products.core }} clusters only).
- `SUPERUSER` role (recommended), or the following granular permissions:
- `CREATE` and `SELECT` on database and tables to replicate.
- Table ownership for adding tables to publications.
- `LOGIN` and `REPLICATION` privileges to create replication slots and access replication data.
- `SELECT` on tables to replicate.
- `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges for binlog access.
- For [`--fetchMetadata`]({% link molt/replicator-flags.md %}#fetch-metadata), either `SELECT` on the source database or `PROCESS` globally.
- `SELECT`, `INSERT`, `UPDATE` on `REPLICATOR_SENTINEL` table.
- `SELECT` on `V$` views (`V$LOG`, `V$LOGFILE`, `V$LOGMNR_CONTENTS`, `V$ARCHIVED_LOG`, `V$LOG_HISTORY`).
- `SELECT` on `SYS.V$LOGMNR_*` views (`SYS.V$LOGMNR_DICTIONARY`, `SYS.V$LOGMNR_LOGS`, `SYS.V$LOGMNR_PARAMETERS`, `SYS.V$LOGMNR_SESSION`).
- `LOGMINING` privilege.
- `EXECUTE` on `DBMS_LOGMNR`.
- For Oracle Multitenant, the user must be a common user (prefixed with `C##`) with privileges granted on both CDB and PDB.
[Create sentinel table]({% link molt/migrate-load-replicate.md %}?filters=oracle#create-source-sentinel-table)
[Grant LogMiner privileges]({% link molt/migrate-load-replicate.md %}?filters=oracle#grant-logminer-privileges) | +| CockroachDB target (forward replication) |
- `ALL` on target database.
- `CREATE` on schema.
- `SELECT`, `INSERT`, `UPDATE`, `DELETE` on target tables.
- `CREATEDB` privilege for creating staging schema.
- `SELECT`, `INSERT`, `UPDATE` on tables to fail back to.
- For Oracle, `FLASHBACK` is also required.
[Grant MySQL user permissions]({% link molt/migrate-failback.md %}?filter=mysql#grant-target-database-user-permissions)
[Grant Oracle user permissions]({% link molt/migrate-failback.md %}?filter=oracle#grant-target-database-user-permissions) | + +## Installation + +{% include molt/molt-install.md %} + +### Docker usage + +{% include molt/molt-docker.md %} + +## See also + +- [MOLT Replicator]({% link molt/molt-replicator.md %}) +- [MOLT Replicator Flags]({% link molt/replicator-flags.md %}) +- [Load and Replicate]({% link molt/migrate-load-replicate.md %}) +- [Migration Overview]({% link molt/migration-overview.md %}) diff --git a/src/current/molt/molt-replicator-troubleshooting.md b/src/current/molt/molt-replicator-troubleshooting.md new file mode 100644 index 00000000000..e96472fc4dd --- /dev/null +++ b/src/current/molt/molt-replicator-troubleshooting.md @@ -0,0 +1,23 @@ +--- +title: MOLT Replicator Troubleshooting +summary: Troubleshoot common issues with MOLT Replicator during continuous replication. +toc: true +docs_area: migrate +--- + +
+
+
+
+
+
+{% include molt/molt-troubleshooting-replication.md %}
+
+{% include molt/molt-troubleshooting-failback.md %}
+
+## See also
+
+- [MOLT Replicator]({% link molt/molt-replicator.md %})
+- [MOLT Replicator Installation]({% link molt/molt-replicator-installation.md %})
+- [MOLT Replicator Flags]({% link molt/replicator-flags.md %})
+- [Migration Overview]({% link molt/migration-overview.md %})
diff --git a/src/current/molt/molt-replicator.md b/src/current/molt/molt-replicator.md
index aa6d1dbe039..c375b6a74e5 100644
--- a/src/current/molt/molt-replicator.md
+++ b/src/current/molt/molt-replicator.md
@@ -13,101 +13,24 @@ MOLT Replicator consumes change data from PostgreSQL [logical replication](https
- *Checkpoint*: The position in the source database's transaction log from which replication begins or resumes: LSN (PostgreSQL), GTID (MySQL), or SCN (Oracle).
- *Staging database*: A CockroachDB database used by Replicator to store replication metadata, checkpoints, and buffered mutations. Specified with [`--stagingSchema`]({% link molt/replicator-flags.md %}#staging-schema) and automatically created with [`--stagingCreateSchema`]({% link molt/replicator-flags.md %}#staging-create-schema). For details, refer to [Staging database](#staging-database).
-- *Forward replication*: Replicate changes from a source database (PostgreSQL, MySQL, or Oracle) to CockroachDB during a migration. For usage details, refer to [Forward replication with initial load](#forward-replication-with-initial-load).
-- *Failback*: Replicate changes from CockroachDB back to the source database. Used for migration rollback or to maintain data consistency on the source during migration. For usage details, refer to [Failback to source database](#failback-to-source-database).
-
-## Prerequisites
-
-### Supported databases
-
-MOLT Replicator supports the following source and target databases:
-
-- PostgreSQL 11-16
-- MySQL 5.7, 8.0 and later
-- Oracle Database 19c (Enterprise Edition) and 21c (Express Edition)
-- CockroachDB (all currently [supported versions]({% link releases/release-support-policy.md %}#supported-versions))
-
-### Database configuration
-
-The source database must be configured for replication:
-
-| Database | Configuration Requirements | Details |
-|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
-| PostgreSQL source | - Enable logical replication by setting `wal_level = logical`.
- Enable [global transaction identifiers (GTID)](https://dev.mysql.com/doc/refman/8.0/en/replication-options-gtids.html) and configure binary logging. Set `binlog-row-metadata` or `binlog-row-image` to `full`.
- Configure sufficient binlog retention for migration duration.
- Install [Oracle Instant Client]({% link molt/migrate-load-replicate.md %}?filters=oracle#oracle-instant-client).
- [Enable `ARCHIVELOG` mode]({% link molt/migrate-load-replicate.md %}?filters=oracle#enable-archivelog-and-force-logging), supplemental logging for primary keys, and `FORCE LOGGING`.
- [Create sentinel table]({% link molt/migrate-load-replicate.md %}#create-source-sentinel-table) (`REPLICATOR_SENTINEL`) in source schema.
- Grant and verify [LogMiner privileges]({% link molt/migrate-load-replicate.md %}#grant-logminer-privileges).
- [Enable rangefeeds]({% link {{ site.current_cloud_version }}/create-and-configure-changefeeds.md %}#enable-rangefeeds) (`kv.rangefeed.enabled = true`) (CockroachDB {{ site.data.products.core }} clusters only).
- `SUPERUSER` role (recommended), or the following granular permissions:
- `CREATE` and `SELECT` on database and tables to replicate.
- Table ownership for adding tables to publications.
- `LOGIN` and `REPLICATION` privileges to create replication slots and access replication data.
- `SELECT` on tables to replicate.
- `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges for binlog access.
- For [`--fetchMetadata`]({% link molt/replicator-flags.md %}#fetch-metadata), either `SELECT` on the source database or `PROCESS` globally.
- `SELECT`, `INSERT`, `UPDATE` on `REPLICATOR_SENTINEL` table.
- `SELECT` on `V$` views (`V$LOG`, `V$LOGFILE`, `V$LOGMNR_CONTENTS`, `V$ARCHIVED_LOG`, `V$LOG_HISTORY`).
- `SELECT` on `SYS.V$LOGMNR_*` views (`SYS.V$LOGMNR_DICTIONARY`, `SYS.V$LOGMNR_LOGS`, `SYS.V$LOGMNR_PARAMETERS`, `SYS.V$LOGMNR_SESSION`).
- `LOGMINING` privilege.
- `EXECUTE` on `DBMS_LOGMNR`.
- For Oracle Multitenant, the user must be a common user (prefixed with `C##`) with privileges granted on both CDB and PDB.
[Create sentinel table]({% link molt/migrate-load-replicate.md %}?filters=oracle#create-source-sentinel-table)
[Grant LogMiner privileges]({% link molt/migrate-load-replicate.md %}?filters=oracle#grant-logminer-privileges) | -| CockroachDB target (forward replication) |
- `ALL` on target database.
- `CREATE` on schema.
- `SELECT`, `INSERT`, `UPDATE`, `DELETE` on target tables.
- `CREATEDB` privilege for creating staging schema.
- `SELECT`, `INSERT`, `UPDATE` on tables to fail back to.
- For Oracle, `FLASHBACK` is also required.
[Grant MySQL user permissions]({% link molt/migrate-failback.md %}?filter=mysql#grant-target-database-user-permissions)
[Grant Oracle user permissions]({% link molt/migrate-failback.md %}?filter=oracle#grant-target-database-user-permissions) | - -## Installation - -{% include molt/molt-install.md %} - -### Docker usage - -{% include molt/molt-docker.md %} +- *Forward replication*: Replicate changes from a source database (PostgreSQL, MySQL, or Oracle) to CockroachDB during a migration. For usage details, refer to [Forward replication (after initial load)](#forward-replication-after-initial-load). +- *Failback*: Replicate changes from CockroachDB back to the source database. Used for migration rollback or to maintain data consistency on the source during migration. For usage details, refer to [Failback replication](#failback-replication). ## How it works MOLT Replicator supports forward replication from PostgreSQL, MySQL, and Oracle, and failback from CockroachDB: -- PostgreSQL source ([`pglogical`](#commands)): MOLT Replicator uses [PostgreSQL logical replication](https://www.postgresql.org/docs/current/logical-replication.html), which is based on publications and replication slots. You create a publication for the target tables, and a slot marks consistent replication points. MOLT Replicator consumes this logical feed directly and applies the data in sorted batches to the target. - -- MySQL source ([`mylogical`](#commands)): MOLT Replicator relies on [MySQL GTID-based replication](https://dev.mysql.com/doc/refman/8.0/en/replication-gtids.html) to read change data from MySQL binlogs. It works with MySQL versions that support GTID-based replication and applies transactionally consistent feeds to the target. Binlog features that do not use GTIDs are not supported. - -- Oracle source ([`oraclelogminer`](#commands)): MOLT Replicator uses [Oracle LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/21/sutil/oracle-logminer-utility.html) to capture change data from Oracle redo logs. Both Oracle Multitenant (CDB/PDB) and single-tenant Oracle architectures are supported. Replicator periodically queries LogMiner-populated views and processes transactional data in ascending SCN windows for reliable throughput while maintaining consistency. - -- Failback from CockroachDB ([`start`](#commands)): MOLT Replicator acts as an HTTP webhook sink for a single CockroachDB changefeed. Replicator receives mutations from source cluster nodes, can optionally buffer them in a CockroachDB staging cluster, and then applies time-ordered transactional batches to the target database. Mutations are applied as [`UPSERT`]({% link {{ site.current_cloud_version }}/upsert.md %}) or [`DELETE`]({% link {{ site.current_cloud_version }}/delete.md %}) statements while respecting [foreign-key]({% link {{ site.current_cloud_version }}/foreign-key.md %}) and table dependencies. - -### Consistency modes - -MOLT Replicator supports three consistency modes for balancing throughput and transactional guarantees: - -1. *Consistent* (failback mode only, default for CockroachDB sources): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by [`--parallelism`]({% link molt/replicator-flags.md %}#parallelism). - -1. *BestEffort* (failback mode only): Relaxes atomicity across tables that do not have foreign key constraints between them (maintains coherence within FK-connected groups). Enable with [`--bestEffortOnly`]({% link molt/replicator-flags.md %}#best-effort-only) or allow auto-entry via [`--bestEffortWindow`]({% link molt/replicator-flags.md %}#best-effort-window) set to a positive duration (such as `1s`). - - {{site.data.alerts.callout_info}} - For independent tables (with no foreign key constraints), BestEffort mode applies changes immediately as they arrive, without waiting for the resolved timestamp. This provides higher throughput for tables that have no relationships with other tables. - {{site.data.alerts.end}} - -1. *Immediate* (default for PostgreSQL, MySQL, and Oracle sources): Applies updates as they arrive to Replicator with no buffering or waiting for resolved timestamps. For CockroachDB sources, provides highest throughput but requires no foreign keys on the target schema. - -## Commands - -MOLT Replicator provides the following commands: +- PostgreSQL source ([`pglogical`]({% link molt/replicator-flags.md %}#commands)): MOLT Replicator uses [PostgreSQL logical replication](https://www.postgresql.org/docs/current/logical-replication.html), which is based on publications and replication slots. You create a publication for the target tables, and a slot marks consistent replication points. MOLT Replicator consumes this logical feed directly and applies the data in sorted batches to the target. -| Command | Description | -|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `pglogical` | Replicate from PostgreSQL source to CockroachDB target using logical replication. | -| `mylogical` | Replicate from MySQL source to CockroachDB target using GTID-based replication. | -| `oraclelogminer` | Replicate from Oracle source to CockroachDB target using Oracle LogMiner. | -| `start` | Replicate from CockroachDB source to PostgreSQL, MySQL, or Oracle target ([failback mode](#failback-to-source-database)). Requires a CockroachDB changefeed with rangefeeds enabled. | -| `make-jwt` | Generate JWT tokens for authorizing changefeed connections in failback scenarios. Supports signing tokens with RSA or EC keys, or generating claims for external JWT providers. For details, refer to [JWT authentication](#jwt-authentication). | -| `version` | Display version information and Go module dependencies with checksums. For details, refer to [Supply chain security](#supply-chain-security). | +- MySQL source ([`mylogical`]({% link molt/replicator-flags.md %}#commands)): MOLT Replicator relies on [MySQL GTID-based replication](https://dev.mysql.com/doc/refman/8.0/en/replication-gtids.html) to read change data from MySQL binlogs. It works with MySQL versions that support GTID-based replication and applies transactionally consistent feeds to the target. Binlog features that do not use GTIDs are not supported. -For command-specific flags and examples, refer to [Usage](#usage) and [Common workflows](#common-workflows). +- Oracle source ([`oraclelogminer`]({% link molt/replicator-flags.md %}#commands)): MOLT Replicator uses [Oracle LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/21/sutil/oracle-logminer-utility.html) to capture change data from Oracle redo logs. Both Oracle Multitenant (CDB/PDB) and single-tenant Oracle architectures are supported. Replicator periodically queries LogMiner-populated views and processes transactional data in ascending SCN windows for reliable throughput while maintaining consistency. -## Flags - -Refer to [Replicator Flags]({% link molt/replicator-flags.md %}). - -## Usage +- Failback from CockroachDB ([`start`]({% link molt/replicator-flags.md %}#commands)): MOLT Replicator acts as an HTTP webhook sink for a single CockroachDB changefeed. Replicator receives mutations from source cluster nodes, can optionally buffer them in a CockroachDB staging cluster, and then applies time-ordered transactional batches to the target database. Mutations are applied as [`UPSERT`]({% link {{ site.current_cloud_version }}/upsert.md %}) or [`DELETE`]({% link {{ site.current_cloud_version }}/delete.md %}) statements while respecting [foreign-key]({% link {{ site.current_cloud_version }}/foreign-key.md %}) and table dependencies. ### Replicator commands -MOLT Replicator provides four commands for different replication scenarios. For detailed workflows, refer to [Common workflows](#common-workflows). +MOLT Replicator provides four commands for different replication scenarios. For example commands, refer to [Common uses](#common-uses). Use `pglogical` to replicate from PostgreSQL to CockroachDB: @@ -140,7 +63,7 @@ replicator start ### Source connection strings {{site.data.alerts.callout_success}} -Follow the security recommendations in [Connection security and credentials](#connection-security-and-credentials). +Follow the security recommendations in [Connection security and credentials]({% link molt/molt-replicator-best-practices.md %}#connection-security-and-credentials). {{site.data.alerts.end}} [`--sourceConn`]({% link molt/replicator-flags.md %}#source-conn) specifies the connection string of the source database for forward replication. @@ -195,7 +118,7 @@ For failback, [`--stagingConn`]({% link molt/replicator-flags.md %}#staging-conn ~~~ {{site.data.alerts.callout_info}} -For failback, [`--targetConn`]({% link molt/replicator-flags.md %}#target-conn) specifies the original source database (PostgreSQL, MySQL, or Oracle). For details, refer to [Failback to source database](#failback-to-source-database). +For failback, [`--targetConn`]({% link molt/replicator-flags.md %}#target-conn) specifies the original source database (PostgreSQL, MySQL, or Oracle). For details, refer to [Failback replication](#failback-replication). {{site.data.alerts.end}} ### Replication checkpoints @@ -209,14 +132,14 @@ For PostgreSQL, use [`--slotName`]({% link molt/replicator-flags.md %}#slot-name --slotName molt_slot ~~~ -For MySQL, set [`--defaultGTIDSet`]({% link molt/replicator-flags.md %}#default-gtid-set) to the [`cdc_cursor` value]({% link molt/molt-fetch.md %}#cdc-cursor) from the MOLT Fetch output: +For MySQL, set [`--defaultGTIDSet`]({% link molt/replicator-flags.md %}#default-gtid-set) to the [`cdc_cursor` value]({% link molt/molt-fetch.md %}#enable-replication) from the MOLT Fetch output: {% include_cached copy-clipboard.html %} ~~~ --defaultGTIDSet '4c658ae6-e8ad-11ef-8449-0242ac140006:1-29' ~~~ -For Oracle, set [`--scn`]({% link molt/replicator-flags.md %}#scn) and [`--backfillFromSCN`]({% link molt/replicator-flags.md %}#backfill-from-scn) to the [`cdc_cursor` values]({% link molt/molt-fetch.md %}#cdc-cursor) from the MOLT Fetch output: +For Oracle, set [`--scn`]({% link molt/replicator-flags.md %}#scn) and [`--backfillFromSCN`]({% link molt/replicator-flags.md %}#backfill-from-scn) to the [`cdc_cursor` values]({% link molt/molt-fetch.md %}#enable-replication) from the MOLT Fetch output: {% include_cached copy-clipboard.html %} ~~~ @@ -241,135 +164,60 @@ The staging database is used to: - Maintain consistency for time-ordered transactional batches while respecting table dependencies. - Provide restart capabilities after failures. -## Security - -Cockroach Labs **strongly** recommends the following: - -### Connection security and credentials - -{% include molt/molt-secure-connection-strings.md %} - -### CockroachDB changefeed security - -For failback scenarios, secure the connection from CockroachDB to MOLT Replicator using TLS certificates. Generate TLS certificates using self-signed certificates, certificate authorities like Let's Encrypt, or your organization's certificate management system. - -#### TLS from CockroachDB to Replicator - -Configure MOLT Replicator with server certificates using the [`--tlsCertificate`]({% link molt/replicator-flags.md %}#tls-certificate) and [`--tlsPrivateKey`]({% link molt/replicator-flags.md %}#tls-private-key) flags to specify the certificate and private key file paths. For example: - -{% include_cached copy-clipboard.html %} -~~~ shell -replicator start \ ---tlsCertificate ./certs/server.crt \ ---tlsPrivateKey ./certs/server.key \ -... -~~~ - -These server certificates must correspond to the client certificates specified in the changefeed webhook URL to ensure proper TLS handshake. - -Encode client certificates for changefeed webhook URLs: - -- Webhook URLs: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'` -- Non-webhook contexts: Use base64 encoding only: `base64 -w 0 ca.cert` +### Consistency modes -#### JWT authentication +MOLT Replicator supports three consistency modes for balancing throughput and transactional guarantees: -You can use JSON Web Tokens (JWT) to authorize incoming changefeed connections and restrict writes to a subset of SQL databases or user-defined schemas in the target cluster. +1. *Consistent* (failback mode only, default for CockroachDB sources): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by [`--parallelism`]({% link molt/replicator-flags.md %}#parallelism). -Replicator supports JWT claims that allow writes to specific databases, schemas, or all of them. JWT tokens must be signed using RSA or EC keys. HMAC and `None` signatures are automatically rejected. +1. *BestEffort* (failback mode only): Relaxes atomicity across tables that do not have foreign key constraints between them (maintains coherence within FK-connected groups). Enable with [`--bestEffortOnly`]({% link molt/replicator-flags.md %}#best-effort-only) or allow auto-entry via [`--bestEffortWindow`]({% link molt/replicator-flags.md %}#best-effort-window) set to a positive duration (such as `1s`). -To configure JWT authentication: + {{site.data.alerts.callout_info}} + For independent tables (with no foreign key constraints), BestEffort mode applies changes immediately as they arrive, without waiting for the resolved timestamp. This provides higher throughput for tables that have no relationships with other tables. + {{site.data.alerts.end}} -1. Add PEM-formatted public signing keys to the `_replicator.jwt_public_keys` table in the staging database. +1. *Immediate* (default for PostgreSQL, MySQL, and Oracle sources): Applies updates as they arrive to Replicator with no buffering or waiting for resolved timestamps. For CockroachDB sources, provides highest throughput but requires no foreign keys on the target schema. -1. To revoke a specific token, add its `jti` value to the `_replicator.jwt_revoked_ids` table in the staging database. +### Monitoring -The Replicator process re-reads these tables every minute to pick up changes. +#### Metrics -To pass the JWT token from the changefeed to the Replicator webhook sink, use the [`webhook_auth_header` option]({% link {{ site.current_cloud_version }}/create-changefeed.md %}#options): +MOLT Replicator metrics are not enabled by default. Enable Replicator metrics by specifying the [`--metricsAddr`]({% link molt/replicator-flags.md %}#metrics-addr) flag with a port (or `host:port`) when you start Replicator. This exposes Replicator metrics at `http://{host}:{port}/_/varz`. For example, the following flag exposes metrics on port `30005`: -{% include_cached copy-clipboard.html %} -~~~ sql -CREATE CHANGEFEED ... WITH webhook_auth_header='Bearer
@@ -378,7 +226,7 @@ Use this information to determine if your build may be subject to vulnerabilitie
-
-
-
-
-
-When resuming replication after an interruption, MOLT Replicator automatically uses the stored checkpoint to resume from the correct position.
-
-Rerun the same `replicator` command used during [forward replication](#forward-replication-with-initial-load), specifying the same fully-qualified [`--stagingSchema`]({% link molt/replicator-flags.md %}#staging-schema) value as before. Omit [`--stagingCreateSchema`]({% link molt/replicator-flags.md %}#staging-create-schema) and any checkpoint flags. For example:
-
-
@@ -695,20 +456,52 @@ To verify that your connections and configuration work properly, run MOLT Replic
-{% include molt/molt-troubleshooting-replication.md %}
+When resuming replication after an interruption, MOLT Replicator automatically uses the stored checkpoint to resume from the correct position.
+
+Rerun the same `replicator` command used during forward replication, specifying the same fully-qualified [`--stagingSchema`]({% link molt/replicator-flags.md %}#staging-schema) value as before. Omit [`--stagingCreateSchema`]({% link molt/replicator-flags.md %}#staging-create-schema) and any checkpoint flags. For example:
+
+**Default:** `1m0s` | | `--scn` | `oraclelogminer` | `INT` | **Required** the first time `replicator` is run. The snapshot System Change Number (SCN) from the initial data load, which provides a replication marker for streaming changes. | | `--scnWindowSize` | `oraclelogminer` | `INT` | The maximum size of SCN bounds per pull iteration from LogMiner. This helps prevent timeout errors when processing large SCN ranges. Set to `0` or a negative value to disable the cap.
**Default:** `3250` | -| `--slotName` | `pglogical` | `STRING` | **Required.** PostgreSQL replication slot name. Must match the slot name specified with `--pglogical-replication-slot-name` in the [MOLT Fetch command]({% link molt/molt-fetch.md %}#load-before-replication).
**Default:** `"replicator"` | +| `--slotName` | `pglogical` | `STRING` | **Required.** PostgreSQL replication slot name. Must match the slot name specified with `--pglogical-replication-slot-name` in the [MOLT Fetch command]({% link molt/molt-fetch.md %}#initial-bulk-load-before-replication).
**Default:** `"replicator"` | | `--sourceConn` | `pglogical`, `mylogical`, `oraclelogminer` | `STRING` | The source database's connection string. When replicating from Oracle, this is the connection string of the Oracle container database (CDB). | | `--sourcePDBConn` | `oraclelogminer` | `STRING` | Connection string for the Oracle pluggable database (PDB). Only required when using an [Oracle multitenant configuration](https://docs.oracle.com/en/database/oracle/oracle-database/21/cncpt/CDBs-and-PDBs.html). [`--sourceConn`](#source-conn) **must** be included. | | `--sourceSchema` | `oraclelogminer` | `STRING` | **Required.** Source schema name on Oracle where tables will be replicated from. | diff --git a/src/current/releases/molt.md b/src/current/releases/molt.md index accc7cdaa33..fda58ef22ca 100644 --- a/src/current/releases/molt.md +++ b/src/current/releases/molt.md @@ -99,9 +99,9 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.3.1 is [available](#installation). -- MOLT Fetch now supports [sharding]({% link molt/molt-fetch.md %}#table-sharding) of primary keys of any data type on PostgreSQL 11+ sources. This can be enabled with the [`--use-stats-based-sharding`]({% link molt/molt-fetch.md %}#global-flags) flag. -- Added the [`--ignore-replication-check`]({% link molt/molt-fetch.md %}#global-flags) flag to allow data loads with planned downtime and no replication setup. The `--pglogical-ignore-wal-check` flag has been removed. -- Added the `--enableParallelApplies` [replication flag]({% link molt/molt-replicator.md %}#flags) to enable parallel application of independent table groups during replication. By default, applies are synchronous. When enabled, this increases throughput at the cost of increased target pool and memory usage. +- MOLT Fetch now supports [sharding]({% link molt/molt-fetch.md %}#shard-tables-for-concurrent-export) of primary keys of any data type on PostgreSQL 11+ sources. This can be enabled with the [`--use-stats-based-sharding`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag. +- Added the [`--ignore-replication-check`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag to allow data loads with planned downtime and no replication setup. The `--pglogical-ignore-wal-check` flag has been removed. +- Added the `--enableParallelApplies` [replication flag]({% link molt/replicator-flags.md %}) to enable parallel application of independent table groups during replication. By default, applies are synchronous. When enabled, this increases throughput at the cost of increased target pool and memory usage. - Improved cleanup logic for scheduled tasks to ensure progress reporting and prevent indefinite hangs. - Added parallelism gating to ensure the parallelism setting is smaller than the `targetMaxPoolSize`. This helps prevent a potential indefinite hang. - Added new metrics that track start and end times for progress reports (`core_progress_reports_started_count` and `core_progress_reports_ended_count`) and error reports (`core_error_reports_started_count` and `core_error_reports_ended_count`). These provide visibility into the core sequencer progress and help identify hangs in the applier and progress tracking pipeline. @@ -151,7 +151,7 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.2.6 is [available](#installation). -- Fixed a bug in [`--direct-copy` mode]({% link molt/molt-fetch.md %}#direct-copy) that occurred when [`--case-sensitive`]({% link molt/molt-fetch.md %}#global-flags) was set to `false` (default). Previously, the `COPY` query could use incorrect column names in some cases during data transfer, causing errors. The query now uses the correct column names. +- Fixed a bug in [`--direct-copy` mode]({% link molt/molt-fetch.md %}#direct-copy) that occurred when [`--case-sensitive`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) was set to `false` (default). Previously, the `COPY` query could use incorrect column names in some cases during data transfer, causing errors. The query now uses the correct column names. - Fixed a bug in how origin messages were handled during replication from PostgreSQL sources. This allows replication to successfully continue. - `ENUM` types can now be replicated from MySQL 8.0 sources. @@ -169,7 +169,7 @@ Cockroach Labs recommends using the latest available version of each tool. Refer - MOLT Fetch failback to CockroachDB is now disallowed. - MOLT Verify can now compare tables that are named differently on the source and target schemas. - The `molt` logging date format is now period-delimited for Windows compatibility. -- During replication, an index is now created on all tables by default, improving replication performance. Because index creation can cause the replication process to initialize more slowly, this behavior can be disabled using the `--stageDisableCreateTableReaderIndex` [replication flag]({% link molt/molt-replicator.md %}#flags). +- During replication, an index is now created on all tables by default, improving replication performance. Because index creation can cause the replication process to initialize more slowly, this behavior can be disabled using the `--stageDisableCreateTableReaderIndex` [replication flag]({% link molt/replicator-flags.md %}#stage-disable-create-table-reader-index). - Added a failback metric that tracks the time to write a source commit to the staging schema for a given mutation. - Added a failback metric that tracks the time to write a source commit to the target database for a given mutation. @@ -177,41 +177,41 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.2.3 is [available](#installation). -- MOLT Fetch users can now set [`--table-concurrency`]({% link molt/molt-fetch.md %}#global-flags) and [`--export-concurrency`]({% link molt/molt-fetch.md %}#global-flags) to values greater than `1` for MySQL sources. -- MOLT Fetch now supports case-insensitive comparison of table and column names by default. Previously, case-sensitive comparisons could result in `no matching table on target` errors. To disable case-sensitive comparisons explicitly, set [`--case-sensitive=false`]({% link molt/molt-fetch.md %}#global-flags). If `=` is **not** included (e.g., `--case-sensitive false`), this is interpreted as `--case-sensitive` (i.e., `--case-sensitive=true`). +- MOLT Fetch users can now set [`--table-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) and [`--export-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) to values greater than `1` for MySQL sources. +- MOLT Fetch now supports case-insensitive comparison of table and column names by default. Previously, case-sensitive comparisons could result in `no matching table on target` errors. To disable case-sensitive comparisons explicitly, set [`--case-sensitive=false`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags). If `=` is **not** included (e.g., `--case-sensitive false`), this is interpreted as `--case-sensitive` (i.e., `--case-sensitive=true`). ### February 5, 2025 `molt` 1.2.2 is [available](#installation). -- Added an [`--import-region`]({% link molt/molt-fetch.md %}#global-flags) flag that is used to set the `AWS_REGION` query parameter explicitly in the [`s3` URL]({% link molt/molt-fetch.md %}#bucket-path). -- Fixed the [`truncate-if-exists`]({% link molt/molt-fetch.md %}#target-table-handling) schema mode for cases where there are uppercase table or schema names. +- Added an [`--import-region`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag that is used to set the `AWS_REGION` query parameter explicitly in the [`s3` URL]({% link molt/molt-fetch.md %}#bucket-path). +- Fixed the [`truncate-if-exists`]({% link molt/molt-fetch.md %}#handle-target-tables) schema mode for cases where there are uppercase table or schema names. - Fixed an issue with unsigned `BIGINT` values overflowing in replication. -- Added a `--schemaRefresh` [replication flag]({% link molt/molt-replicator.md %}#flags) that is used to configure the schema watcher refresh delay in the replication phase. Previously, the refresh delay was set to a constant value of 1 minute. Set the flag as follows: `--replicator-flags "--schemaRefresh {value}"`. +- Added a `--schemaRefresh` [replication flag]({% link molt/replicator-flags.md %}#schema-refresh) that is used to configure the schema watcher refresh delay in the replication phase. Previously, the refresh delay was set to a constant value of 1 minute. Set the flag as follows: `--replicator-flags "--schemaRefresh {value}"`. ### December 13, 2024 `molt` 1.2.1 is [available](#installation). -- MOLT Fetch users now can use [`--assume-role`]({% link molt/molt-fetch.md %}#global-flags) to specify a service account for assume role authentication to cloud storage. `--assume-role` must be used with `--use-implicit-auth`, or the flag will be ignored. +- MOLT Fetch users now can use [`--assume-role`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) to specify a service account for assume role authentication to cloud storage. `--assume-role` must be used with `--use-implicit-auth`, or the flag will be ignored. - MySQL 5.7 and later are now supported with MOLT Fetch replication modes. - Fetch replication mode now defaults to a less verbose `INFO` logging level. To specify `DEBUG` logging, pass in the `--replicator-flags '-v'` setting, or `--replicator-flags '-vv'` for trace logging. - MySQL columns of type `BIGINT UNSIGNED` or `SERIAL` are now auto-mapped to [`DECIMAL`]({% link {{ site.current_cloud_version }}/decimal.md %}) type in CockroachDB. MySQL regular `BIGINT` types are mapped to [`INT`]({% link {{ site.current_cloud_version }}/int.md %}) type in CockroachDB. -- The `pglogical` replication workflow was modified in order to enforce safer and simpler defaults for the [`data-load`]({% link molt/molt-fetch.md %}#fetch-mode), `data-load-and-replication`, and `replication-only` workflows for PostgreSQL sources. Fetch now ensures that the publication is created before the slot, and that `replication-only` defaults to using publications and slots created either in previous Fetch runs or manually. +- The `pglogical` replication workflow was modified in order to enforce safer and simpler defaults for the [`data-load`]({% link molt/molt-fetch.md %}#define-fetch-mode), `data-load-and-replication`, and `replication-only` workflows for PostgreSQL sources. Fetch now ensures that the publication is created before the slot, and that `replication-only` defaults to using publications and slots created either in previous Fetch runs or manually. - Fixed scan iterator query ordering for `BINARY` and `TEXT` (of same collation) PKs so that they lead to the correct queries and ordering. -- For a MySQL source in `replication-only` mode, the [`--stagingSchema` replicator flag]({% link molt/replicator-flags.md %}#staging-schema) can now be used to resume streaming replication after being interrupted. Otherwise, the [`--defaultGTIDSet` replicator flag]({% link molt/replicator-flags.md %}#default-gtid-set) is used to start initial replication after a previous Fetch run in [`data-load`]({% link molt/molt-fetch.md %}#fetch-mode) mode, or as an override to the current replication stream. +- For a MySQL source in `replication-only` mode, the [`--stagingSchema` replicator flag]({% link molt/replicator-flags.md %}#staging-schema) can now be used to resume streaming replication after being interrupted. Otherwise, the [`--defaultGTIDSet` replicator flag]({% link molt/replicator-flags.md %}#default-gtid-set) is used to start initial replication after a previous Fetch run in [`data-load`]({% link molt/molt-fetch.md %}#define-fetch-mode) mode, or as an override to the current replication stream. ### October 29, 2024 `molt` 1.2.0 is [available](#installation). - Added `failback` mode to MOLT Fetch, which allows the user to replicate changes on CockroachDB back to the initial source database. Failback is supported for MySQL and PostgreSQL databases. -- The [`--pprof-list-addr` flag]({% link molt/molt-fetch.md %}#global-flags), which specifies the address of the `pprof` endpoint, is now configurable. The default value is `'127.0.0.1:3031'`. -- [Fetch modes]({% link molt/molt-fetch.md %}#fetch-mode) involving replication now state that MySQL 8.0 and later are supported for replication between MySQL and CockroachDB. -- [Partitioned tables]({% link molt/molt-fetch.md %}#transformations) can now be moved to CockroachDB using [`IMPORT INTO`]({% link {{ site.current_cloud_version }}/import-into.md %}). -- Improved logging for the [Fetch]({% link molt/molt-fetch.md %}) schema check phases under the `trace` logging level, which is set with [`--logging trace`]({% link molt/molt-fetch.md %}#global-flags). +- The [`--pprof-list-addr` flag]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags), which specifies the address of the `pprof` endpoint, is now configurable. The default value is `'127.0.0.1:3031'`. +- [Fetch modes]({% link molt/molt-fetch.md %}#define-fetch-mode) involving replication now state that MySQL 8.0 and later are supported for replication between MySQL and CockroachDB. +- [Partitioned tables]({% link molt/molt-fetch.md %}#define-transformations) can now be moved to CockroachDB using [`IMPORT INTO`]({% link {{ site.current_cloud_version }}/import-into.md %}). +- Improved logging for the [Fetch]({% link molt/molt-fetch.md %}) schema check phases under the `trace` logging level, which is set with [`--logging trace`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags). - Added a [sample Grafana dashboard](https://molt.cockroachdb.com/molt/cli/grafana_dashboard.json) for monitoring MOLT tasks. -- Fetch now logs the name of the staging database in the target CockroachDB cluster used to store metadata for [replication modes]({% link molt/molt-fetch.md %}#fetch-mode). +- Fetch now logs the name of the staging database in the target CockroachDB cluster used to store metadata for [replication modes]({% link molt/molt-fetch.md %}#define-fetch-mode). - String [primary keys]({% link {{ site.current_cloud_version }}/primary-key.md %}) that use `C` [collations]({% link {{ site.current_cloud_version }}/collate.md %}) on PostgreSQL can now be compared to the default `en_US.utf8` on CockroachDB. - MOLT is now distributed under the [Cockroach Labs Product License Agreement](https://www.cockroachlabs.com/cockroach-labs-product-license-agreement/), which is bundled with the binary. @@ -219,7 +219,7 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.1.7 is [available](#installation). -- When a [Fetch transformation rule]({% link molt/molt-fetch.md %}#transformations) is used to rename a table or map partitioned tables, a script in the format `partitionTableScript.{timestamp}.ts` is now automatically generated to ensure that [replication]({% link molt/molt-fetch.md %}#fetch-mode) works properly if enabled. +- When a [Fetch transformation rule]({% link molt/molt-fetch.md %}#define-transformations) is used to rename a table or map partitioned tables, a script in the format `partitionTableScript.{timestamp}.ts` is now automatically generated to ensure that [replication]({% link molt/molt-fetch.md %}#define-fetch-mode) works properly if enabled. ### August 19, 2024 @@ -232,8 +232,8 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.1.5 is [available](#installation). - **Deprecated** the `--ongoing-replication` flag in favor of `--mode data-load-and-replication`, using the new `--mode` flag. Users should replace all instances of `--ongoing-replication` with `--mode data-load-and-replication`. -- Fetch can now be run in an export-only mode by specifying [`--mode export-only`]({% link molt/molt-fetch.md %}#fetch-mode). This will export all the data in `csv` or `csv.gz` format to the specified cloud or local store. -- Fetch can now be run in an import-only mode by specifying [`--mode import-only`]({% link molt/molt-fetch.md %}#fetch-mode). This will load all data in the specified cloud or local store into the target CockroachDB database, effectively skipping the export data phase. +- Fetch can now be run in an export-only mode by specifying [`--mode export-only`]({% link molt/molt-fetch.md %}#define-fetch-mode). This will export all the data in `csv` or `csv.gz` format to the specified cloud or local store. +- Fetch can now be run in an import-only mode by specifying [`--mode import-only`]({% link molt/molt-fetch.md %}#define-fetch-mode). This will load all data in the specified cloud or local store into the target CockroachDB database, effectively skipping the export data phase. - Strings for the `--mode` flag are now word-separated by hyphens instead of underscores. For example, `replication-only` instead of `replication_only`. ### August 8, 2024 @@ -241,7 +241,7 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.1.4 is [available](#installation). - Added a replication-only mode for Fetch that allows the user to run ongoing replication without schema creation or initial data load. This requires users to set `--mode replication_only` and `--replicator-flags` to specify the `defaultGTIDSet` ([MySQL](https://github.com/cockroachdb/replicator/wiki/MYLogical)) or `slotName` ([PostgreSQL](https://github.com/cockroachdb/replicator/wiki/PGLogical)). -- Partitioned tables can now be mapped to renamed tables on the target database, using the Fetch [transformations framework]({% link molt/molt-fetch.md %}#transformations). +- Partitioned tables can now be mapped to renamed tables on the target database, using the Fetch [transformations framework]({% link molt/molt-fetch.md %}#define-transformations). - Added a new `--metrics-scrape-interval` flag to allow users to specify their Prometheus scrape interval and apply a sleep at the end to allow for the final metrics to be scraped. - Previously, there was a mismatch between the errors logged in log lines and those recorded in the exceptions table when an `IMPORT INTO` or `COPY FROM` operation failed due to a non-PostgreSQL error. Now, all errors will lead to an exceptions table entry that allows the user to continue progress from a certain table's file. - Fixed a bug that will allow Fetch to properly determine a GTID if there are multiple `source_uuids`. @@ -251,8 +251,8 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.1.3 is [available](#installation). - `'infinity'::timestamp` values can now be moved with Fetch. -- Fixed an issue where connections were not being closed immediately after sharding was completed. This could lead to errors if the [maximum number of connections]({% link molt/molt-fetch.md %}#best-practices) was set to a low value. -- Fetch users can now exclude specific tables from migration using the [`--table-exclusion-filter` flag]({% link molt/molt-fetch.md %}#global-flags). +- Fixed an issue where connections were not being closed immediately after sharding was completed. This could lead to errors if the [maximum number of connections]({% link molt/molt-fetch-best-practices.md %}#configure-the-source-database-and-connection) was set to a low value. +- Fetch users can now exclude specific tables from migration using the [`--table-exclusion-filter` flag]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags). ### July 18, 2024 @@ -260,16 +260,16 @@ Cockroach Labs recommends using the latest available version of each tool. Refer - Fetch users can now specify columns to exclude from table migrations in order to migrate a subset of their data. This is supported in the schema creation, export, import, and direct copy phases. - Fetch now automatically maps a partitioned table from a PostgreSQL source to the target CockroachDB schema. -- Fetch now supports column exclusions and computed column mappings via a new [transformations framework]({% link molt/molt-fetch.md %}#transformations). -- The new Fetch [`--transformations-file`]({% link molt/molt-fetch.md %}#global-flags) flag specifies a JSON file for schema/table/column transformations, which has validation utilities built in. +- Fetch now supports column exclusions and computed column mappings via a new [transformations framework]({% link molt/molt-fetch.md %}#define-transformations). +- The new Fetch [`--transformations-file`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag specifies a JSON file for schema/table/column transformations, which has validation utilities built in. ### July 10, 2024 `molt` 1.1.1 is [available](#installation). -- Fixed a bug that led to incorrect list continuation file behavior if a trailing slash was provided in [`--bucket-path`]({% link molt/molt-fetch.md %}#global-flags). +- Fixed a bug that led to incorrect list continuation file behavior if a trailing slash was provided in [`--bucket-path`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags). - Fixed a bug with extracting the filename from a failed import URL. Previously, an older filename was being used, which could result in duplicated data. Now, the filename that is used in import matches what is stored in the exceptions log table. -- Added a [`--use-implicit-auth`]({% link molt/molt-fetch.md %}#global-flags) flag that determines whether [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}) is used for cloud storage import URIs. +- Added a [`--use-implicit-auth`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag that determines whether [implicit authentication]({% link {{ site.current_cloud_version }}/cloud-storage-authentication.md %}) is used for cloud storage import URIs. ### July 8, 2024 @@ -284,14 +284,14 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 1.0.0 is [available](#installation). -- Renamed the `--table-splits` flag to [`--concurrency-per-table`]({% link molt/molt-fetch.md %}#global-flags), which is more descriptive. -- Increased the default value of [`--import-batch-size`]({% link molt/molt-fetch.md %}#global-flags) to `1000`. This leads to better performance on the target post-migration. Each individual import job will take longer, since more data is now imported in each batch, but the sum total of all jobs should take the same (or less) time. +- Renamed the `--table-splits` flag to [`--concurrency-per-table`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags), which is more descriptive. +- Increased the default value of [`--import-batch-size`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) to `1000`. This leads to better performance on the target post-migration. Each individual import job will take longer, since more data is now imported in each batch, but the sum total of all jobs should take the same (or less) time. ### May 29, 2024 `molt` 0.3.0 is [available](#installation). -- Added an [`--import-batch-size`]({% link molt/molt-fetch.md %}#global-flags) flag, which configures the number of files to be imported in each `IMPORT` job. +- Added an [`--import-batch-size`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag, which configures the number of files to be imported in each `IMPORT` job. - In some cases on the previous version, binaries would not work due to how `molt` was being built. Updated the build method to use static linking, which creates binaries that should be more portable. - [`VARBIT`]({% link {{ site.current_cloud_version }}/bit.md %}) <> [`BOOL`]({% link {{ site.current_cloud_version }}/bool.md %}) conversion is now allowed for Fetch and Verify. The bit array is first converted to `UINT64`. A resulting `1` or `0` is converted to `true` or `false` accordingly. If the `UINT64` is another value, an error is emitted. @@ -300,7 +300,7 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 0.2.1 is [available](#installation). - MOLT tools now enforce secure connections to databases as a default. The `--allow-tls-mode-disable` flag allows users to override that behavior if secure access is not possible. -- When using MySQL as a source, [`--table-concurrency`]({% link molt/molt-fetch.md %}#global-flags) and [`--export-concurrency`]({% link molt/molt-fetch.md %}#global-flags) are strictly set to `1`. +- When using MySQL as a source, [`--table-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) and [`--export-concurrency`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) are strictly set to `1`. - Fixed a bug involving history retention for [`DECIMAL`]({% link {{ site.current_cloud_version }}/decimal.md %}) values. ### May 3, 2024 @@ -308,9 +308,9 @@ Cockroach Labs recommends using the latest available version of each tool. Refer `molt` 0.2.0 is [available](#installation). - Fetch now supports CockroachDB [multi-region tables]({% link {{ site.current_cloud_version }}/multiregion-overview.md %}). -- Fetch now supports continuous replication for PostgreSQL and MySQL source databases via the [`--ongoing-replication`]({% link molt/molt-fetch.md %}#global-flags) flag. When Fetch finishes the initial data load phase, it will start the replicator process as a subprocess, which runs indefinitely until the user ends the process with a `SIGTERM` (`ctrl-c`). -- Replicator flags for ([PostgreSQL](https://github.com/cockroachdb/replicator/wiki/PGLogical#postgresql-logical-replication) and [MySQL](https://github.com/cockroachdb/replicator/wiki/MYLogical#mysqlmariadb-replication)) are now supported, allowing users to further configure the [`--ongoing-replication`]({% link molt/molt-fetch.md %}#global-flags) mode for their use case. -- Added the [`--type-map-file`]({% link molt/molt-fetch.md %}#global-flags) flag, which enables custom type mapping for schema creation. +- Fetch now supports continuous replication for PostgreSQL and MySQL source databases via the [`--ongoing-replication`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag. When Fetch finishes the initial data load phase, it will start the replicator process as a subprocess, which runs indefinitely until the user ends the process with a `SIGTERM` (`ctrl-c`). +- Replicator flags for ([PostgreSQL](https://github.com/cockroachdb/replicator/wiki/PGLogical#postgresql-logical-replication) and [MySQL](https://github.com/cockroachdb/replicator/wiki/MYLogical#mysqlmariadb-replication)) are now supported, allowing users to further configure the [`--ongoing-replication`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) mode for their use case. +- Added the [`--type-map-file`]({% link molt/molt-fetch-commands-and-flags.md %}#global-flags) flag, which enables custom type mapping for schema creation. - Fixed a bug where primary key positions could be missed when creating a schema with multiple primary keys. - Added a default mode for MySQL sources that ensures consistency and does not leverage parallelism. New text is displayed that alerts the user and links to documentation in cases where fetching from MySQL might not be consistent. - Logging for continuation tokens is now omitted when data export does not successfully complete.