Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
2ef5919
Enhance deferred fields for production ETL reliability
bosd Dec 21, 2025
e08aa84
Fix CLI deferred-fields parsing and Pass 1 ignore filtering
bosd Dec 21, 2025
d10b5f3
feat: add --auto-defer option for progressive import mode
bosd Dec 21, 2025
7ec5c72
fix: create ir.model.data entries when using create() method
bosd Dec 21, 2025
583ed63
feat: add advanced import options for handling missing refs and fallb…
bosd Dec 22, 2025
bd461fa
perf: remove connection cap, add caching and pre-calculation
bosd Dec 22, 2025
a277b51
feat: add --protocol option for RPC protocol selection
bosd Dec 22, 2025
d1599d1
fix: convert --ignore from comma-separated string to list
bosd Dec 22, 2025
026aa75
docs: add protocol selection and worker tuning documentation
bosd Dec 22, 2025
8314992
test: add Unicode and multiline CSV handling test
bosd Dec 22, 2025
5a3be70
feat: add --delay option for rate limiting between batches
bosd Dec 22, 2025
b9e4c18
feat: add adaptive throttling for 503/502 server overload errors
bosd Dec 22, 2025
47d6278
fix: prevent Rich progress bar shifting by suppressing log handler
bosd Dec 22, 2025
07ef183
chore: update mypyc configuration for better compatibility
bosd Dec 22, 2025
6fa9cc5
test: improve test coverage to 85%
bosd Dec 22, 2025
797d68a
feat: add streaming CSV support for memory-efficient large file imports
bosd Dec 23, 2025
fedcd43
Add checkpoint/resume support and --all-companies flag
bosd Dec 23, 2025
0a5e7b2
feat: add dry-run validation mode
bosd Dec 23, 2025
061e6eb
feat: add pre-import reference check
bosd Dec 23, 2025
f57cc02
feat: add smart retry logic module
bosd Dec 23, 2025
442f61c
feat: add idempotent import module
bosd Dec 23, 2025
9896813
feat: add health-aware throttling module
bosd Dec 23, 2025
d3acf0c
feat: integrate retry, idempotent, and throttle modules
bosd Dec 24, 2025
86bce87
Add VIES/VAT validation management workflow
bosd Dec 25, 2025
00341ea
Add documentation for VAT validation management
bosd Dec 25, 2025
2f8b036
Fix linting and type annotations for pre-commit/mypy compliance
bosd Dec 26, 2025
e5942b0
Fix mypy type errors in validation, idempotent, and preflight modules
bosd Dec 26, 2025
a228570
Add date_formats and datetime_formats parameters to Processor
bosd Dec 26, 2025
a478535
Add expr module with Polars expression-based mappers
bosd Dec 26, 2025
1f741c6
Add type annotations to test files for mypy compliance
bosd Dec 26, 2025
d0913e4
Integrate dynamic batch size scaling into import process
bosd Dec 27, 2025
10d8032
Add adaptive throttling documentation and fix test type annotations
bosd Dec 27, 2025
a83db51
Add data cleaning module for transformation pipelines
bosd Dec 28, 2025
59d001c
Add data cleaning documentation to transformation guide
bosd Dec 29, 2025
60e6b29
Improve phone normalization to detect country codes and 00 prefix
bosd Dec 29, 2025
bcf0d9f
Add email cleaner support for colons (mailto: prefix, separators, tra…
bosd Dec 29, 2025
486a624
feat(clean): add city/postal separator and country detection
bosd Dec 29, 2025
ed5a969
feat(clean): add company_suffix() for business entity normalization
bosd Dec 29, 2025
3303e14
refactor(clean): remove hardcoded MAJOR_CITIES constant
bosd Dec 29, 2025
b3fce15
docs: add company_suffix() examples to data transformations guide
bosd Dec 29, 2025
0326375
feat(geonames): add GeoNames data utilities for geographic lookups
bosd Dec 29, 2025
ea439b5
fix(clean): improve zip_code() to filter e- prefix and remove commas
bosd Dec 29, 2025
59ae2e5
docs: update zip_code() documentation with new features
bosd Dec 29, 2025
84a1afe
feat(clean): add city() cleaner for city name normalization
bosd Dec 29, 2025
223acf2
feat(clean): add street() cleaner for street address cleaning
bosd Dec 29, 2025
fcb4bc5
fix(preflight): exclude 'id' field from readonly warning
bosd Dec 31, 2025
9cfb151
feat: environment-based fail file placement
bosd Dec 31, 2025
a274b31
docs: add multi-environment imports section
bosd Dec 31, 2025
10594f2
feat: improve access error messages in fail files
bosd Dec 31, 2025
94cc664
fix: avoid using model.browse() for models where it's not allowed rem…
bosd Jan 2, 2026
47ec5e4
fix: remove remaining model.env.ref() usage that triggered browse errors
bosd Jan 2, 2026
70458cf
fix: avoid triggering browse when accessing record ID from create()
bosd Jan 2, 2026
24ed6be
fix: sanitize id_map lookups in pass 2 for parent_id resolution
bosd Jan 4, 2026
4bf04a3
fix: add logging around thread pool shutdown to diagnose hanging
bosd Jan 4, 2026
0b6e393
fix: use console.print for diagnostic messages during progress display
bosd Jan 4, 2026
de1587e
fix: add diagnostic logging throughout Pass 2
bosd Jan 4, 2026
1471f1a
fix: add granular diagnostic logging inside _prepare_pass_2_data
bosd Jan 4, 2026
994f892
fix: add counters to track id_map hits vs RPC lookups in Pass 2
bosd Jan 4, 2026
220b62d
perf: add caching and better progress for Pass 2 external ID lookups
bosd Jan 4, 2026
9e8efd9
fix: prevent automatic deferral of m2m fields without user consent
bosd Jan 5, 2026
26591af
fix: suppress confusing deferrable fields message when not auto-defer…
bosd Jan 5, 2026
b638e49
refactor: use load() instead of create() for fallback imports
bosd Jan 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,7 @@ node_modules
_build
ODF_ Strategic Blueprint.md
.coverage

# mypyc compiled extensions
*.so
build/
59 changes: 59 additions & 0 deletions docs/guides/advanced_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,65 @@ product_mapping = {

---

## Multi-Environment Imports

When working with multiple Odoo environments (e.g., test, UAT, production), the importer automatically organizes fail files into environment-specific subfolders based on your connection file name.

### How It Works

The environment name is extracted from your connection file:

| Connection File | Environment | Fail File Location |
|----------------|-------------|-------------------|
| `test_connection.conf` | `test` | `data/test/res_partner_fail.csv` |
| `uat_connection.conf` | `uat` | `data/uat/res_partner_fail.csv` |
| `prod_connection.conf` | `prod` | `data/prod/res_partner_fail.csv` |
| `uat.conf` | `uat` | `data/uat/res_partner_fail.csv` |

The `_connection` suffix is automatically stripped to determine the environment name.

### Example: Importing to Multiple Environments

**Directory Structure:**
```
project/
├── data/
│ └── res_partner.csv
├── test_connection.conf
├── uat_connection.conf
└── prod_connection.conf
```

**Import to UAT:**
```bash
odoo-data-flow import \
--connection-file uat_connection.conf \
--file data/res_partner.csv \
--model res.partner
```

If any records fail, they are written to `data/uat/res_partner_fail.csv`.

**Retry Failed Records:**
```bash
odoo-data-flow import \
--connection-file uat_connection.conf \
--file data/res_partner.csv \
--model res.partner \
--fail
```

The `--fail` flag automatically looks for the fail file in the correct environment folder (`data/uat/res_partner_fail.csv`).

### Benefits

- **Isolated environments**: Fail files from different environments don't mix
- **Easy retry**: The `--fail` flag finds the correct fail file automatically
- **Clean organization**: Each environment has its own subfolder for tracking failures
- **Automatic folder creation**: Environment folders are created automatically when needed

---

## Importing Translations

The most efficient way to import translations is to perform a standard import with a special `lang` key in the context. This lets Odoo's ORM handle the translation creation process correctly.
Expand Down
30 changes: 27 additions & 3 deletions docs/guides/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ database = my_odoo_db
login = admin
password = my_admin_password
uid = 2
protocol = xmlrpc
protocol = jsonrpc
```

### Configuration Keys
Expand Down Expand Up @@ -62,9 +62,33 @@ protocol = xmlrpc

#### `protocol`
* **Required**: No
* **Description**: The connection protocol to use for XML-RPC calls. `xmlrpc` uses HTTP, while `xmlrpcs` uses HTTPS for a secure connection. While modern Odoo uses JSON-RPC for its web interface, the external API for this type of integration typically uses XML-RPC.
* **Description**: The RPC protocol to use for communication with Odoo. The choice of protocol can significantly impact performance.
* **Default**: `xmlrpc`
* **Example**: `protocol = xmlrpcs`
* **Available Options**:
* `xmlrpc` - XML-RPC over HTTP (default, compatible with all Odoo versions)
* `xmlrpcs` - XML-RPC over HTTPS (secure)
* `jsonrpc` - JSON-RPC over HTTP (**recommended for Odoo 10+**, ~30% faster)
* `jsonrpcs` - JSON-RPC over HTTPS (secure, recommended for production)
* `json2` - JSON-2 API over HTTP (Odoo 19+ only, requires API key)
* `json2s` - JSON-2 API over HTTPS (Odoo 19+ only, requires API key)

* **Performance Note**: JSON-RPC is approximately 30% faster than XML-RPC due to more efficient parsing and smaller payload sizes. For Odoo 10 and newer, using `jsonrpc` or `jsonrpcs` is recommended.

* **Odoo 19+ Note**: Odoo 19 introduces the new JSON-2 API which will replace XML-RPC and JSON-RPC in Odoo 20. JSON-2 requires an API key instead of a password. Generate an API key from your Odoo user preferences (Account Security section) and use it in the `password` field.

* **Example**: `protocol = jsonrpcs`

#### Overriding Protocol via CLI

You can override the protocol setting from your config file using the `--protocol` CLI option:

```bash
# Use JSON-RPC for better performance
odoo-data-flow import --protocol jsonrpc --connection-file conf/connection.conf ...

# Use JSON-2 for Odoo 19+
odoo-data-flow import --protocol json2 --connection-file conf/connection.conf ...
```

---

Expand Down
Loading
Loading