Skip to content

Conversation

@JC1738
Copy link

@JC1738 JC1738 commented Jan 12, 2026

… and Grafana to connect and discover tables in DuckLake mode.

Key changes:

Schema remapping for DuckLake:
- Remap public.* table references to main.* (DuckLake uses main schema)
- Remap public.* column references in SELECT clauses
- Add ORDER BY 1 workaround for LIMIT queries without ORDER BY (prevents DuckLake from reading stale data files)

New pg_catalog views:
- pg_attribute: Column metadata for table discovery
- pg_type: Comprehensive type OID mapping (~30 PostgreSQL types)
- pg_index: Primary key discovery with synthetic PKs for DuckLake
- pg_constraint: Constraint metadata stub
- pg_description: Object descriptions stub
- pg_attrdef: Column defaults stub
- pg_proc: Function metadata stub
- pg_tables/pg_views: Schema mapping from main to public

New functions/macros:
- current_schemas(bool): Schema search path
- _pg_expandarray(arr): Array expansion for JDBC PK discovery
- has_schema_privilege, has_table_privilege, has_any_column_privilege
- format_type, obj_description, col_description enhancements

Regex operator fixes:
- Convert PostgreSQL regex operators (~, ~*, ) to DuckDB regexp_matches() function calls
- Handle case-insensitive variants with (?i) flag
- Fix JDBC escaped operator syntax (\~,\!~)

Other improvements:
- DUCKGRES_TABLE_LIMIT env var for faster testing cycles
- SQL injection fix in R2 secret creation (escape single quotes)
- Recreate pg_tables after DuckLake attachment for proper metadata
- Add duckgres.yaml.example config template
- Update .gitignore to exclude credential filesUpdate for Spencer

… and Grafana to connect and discover tables in DuckLake mode.

    Key changes:

    Schema remapping for DuckLake:
    - Remap public.* table references to main.* (DuckLake uses main schema)
    - Remap public.* column references in SELECT clauses
    - Add ORDER BY 1 workaround for LIMIT queries without ORDER BY
      (prevents DuckLake from reading stale data files)

    New pg_catalog views:
    - pg_attribute: Column metadata for table discovery
    - pg_type: Comprehensive type OID mapping (~30 PostgreSQL types)
    - pg_index: Primary key discovery with synthetic PKs for DuckLake
    - pg_constraint: Constraint metadata stub
    - pg_description: Object descriptions stub
    - pg_attrdef: Column defaults stub
    - pg_proc: Function metadata stub
    - pg_tables/pg_views: Schema mapping from main to public

    New functions/macros:
    - current_schemas(bool): Schema search path
    - _pg_expandarray(arr): Array expansion for JDBC PK discovery
    - has_schema_privilege, has_table_privilege, has_any_column_privilege
    - format_type, obj_description, col_description enhancements

    Regex operator fixes:
    - Convert PostgreSQL regex operators (~, ~*, ) to DuckDB
      regexp_matches() function calls
    - Handle case-insensitive variants with (?i) flag
    - Fix JDBC escaped operator syntax (\~,\!~)

    Other improvements:
    - DUCKGRES_TABLE_LIMIT env var for faster testing cycles
    - SQL injection fix in R2 secret creation (escape single quotes)
    - Recreate pg_tables after DuckLake attachment for proper metadata
    - Add duckgres.yaml.example config template
    - Update .gitignore to exclude credential filesUpdate for Spencer
@bill-ph
Copy link
Collaborator

bill-ph commented Jan 20, 2026

Thanks for the PR @JC1738 , and sorry for the delay in review, we did not expect external contributions at this stage of the repo.

Your changes look good in the general direction, would you mind breaking it up into smaller PRs so it's less conflict prone and easier to review?

@JC1738 JC1738 marked this pull request as draft January 21, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants