Antalya does not see tables on iceberg using nessie catalog + GCS

[volokluev]

Describe the bug
When using the nessie REST catalog for iceberg, and using SHOW TABLES, antalya does not see any tables despite successfully traversing the catalog

To Reproduce

In brief this is the reproduction steps

1. Spin up nessie and clickhouse
2. Create and write to an iceberg table
3. Create an iceberg database
4. USE said iceberg database
5. SHOW TABLES -> see no tables show up

For Detailed steps: see the attached zip file README.md for complete steps on reproduction. I think the issue has something to do with the parsing of the iceberg REST catalog response.

Expected behavior
The database shows the tables existing in the catalog

Key information
Provide relevant runtime details.

• Project Antalya Build Version: 25.8.9.20496.altinityantalya
• Cloud provider: GCP
• Kubernetes provider: None, used docker compose
• Object storage: GCS
• Iceberg catalog: Nessie

Additional context
See zip file for an easy repro step

iceberg-clickhouse-repro.zip

[volokluev]

Update:

I did more testing on it and I don't think the problem is nessie. The problem is GCS. I decided to try a different catalog (lakekeeper), spun that up locally and everything worked with minio. once I used a gcs bucket, I got the same problem once again with tables not showing up

[mkmkme]

Hey @volokluev,

Thanks for the report and the repro! At the moment we are working on reproducing it on our side. We will keep you updated.

[redmode]

Seems we're facing very the same issue. @mkmkme hopefully the report below helps. or at least do not harm :)

ClickHouse Antalya + Iceberg + GCS Integration Issue

Related to: Altinity/ClickHouse#1199
Date: December 18, 2025
Status: Confirmed - No workaround found

---

Summary

We encountered the exact same issue reported in #1199 while integrating ClickHouse Antalya with Iceberg tables on GCS. After testing 5 different approaches, we identified the root cause and can confirm this is an architectural limitation in ClickHouse's DataLakeCatalog engine.

Root Cause: ClickHouse's DataLakeCatalog does not support the gs:// protocol. When Iceberg metadata contains GCS URLs, ClickHouse fails with Unsupported storage type: gs.

---

Environment

Component	Version/Details
ClickHouse	25.8.9.20496.altinityantalya
Platform	Google Kubernetes Engine (GKE)
Object Storage	Google Cloud Storage (GCS)
Iceberg Catalog	Lakekeeper REST Catalog v0.6.0
Database Backend	Cloud SQL PostgreSQL 15
Deployment	Kubernetes 1.28

---

Observed Behavior

What Works ✅

1. ClickHouse successfully connects to Lakekeeper REST Catalog
2. ClickHouse discovers tables from the catalog
3. Database creation succeeds:

   CREATE DATABASE iceberg_db
   ENGINE = DataLakeCatalog('http://catalog-service:8181/catalog')
   SETTINGS catalog_type = 'rest',
            storage_endpoint = 'http://s3-proxy:9000',
            warehouse = 'my-warehouse'

What Fails ❌

1. SHOW TABLES FROM iceberg_db returns empty
2. Direct table queries fail:

   SELECT * FROM iceberg_db.`namespace.table` LIMIT 1;
   -- Error: Unknown table expression identifier

Critical Error in Logs

<Error> DatabaseDataLake(iceberg_db): ignoring table namespace.table:
Code: 48. DB::Exception: Unsupported storage type: gs. (NOT_IMPLEMENTED)

Stack trace:
4. DataLake::parseStorageTypeFromString(String const&) @ 0x0000000016810376
5. DB::DatabaseDataLake::tryGetTableImpl(...) @ 0x000000001727269f

---

Root Cause Analysis

The Problem

1. Iceberg metadata contains GCS URLs:

   {
     "location": "gs://bucket-name/warehouse/namespace/table/",
     "manifest-list": "gs://bucket-name/.../snap-123.avro"
   }

2. ClickHouse parses metadata and extracts storage URLs

3. ClickHouse attempts to access gs:// URLs directly

4. ClickHouse's storage layer only supports:

   • ✅ s3:// (S3-compatible storage)
   • ✅ hdfs:// (Hadoop Distributed File System)
   • ✅ file:// (Local filesystem)
   • ❌ gs:// (Google Cloud Storage) - NOT SUPPORTED

5. Result: Unsupported storage type: gs

Why Proxies Don't Help

We tested using rclone as an S3-compatible proxy in front of GCS:

ClickHouse → Lakekeeper → Iceberg Metadata (contains gs:// URLs) → ClickHouse tries gs:// → FAIL

The proxy cannot rewrite URLs inside Iceberg metadata files - they are immutable and contain the original gs:// paths.

---

Approaches Tested (All Failed)

1. Native GCS with Workload Identity

• Config: GCP Workload Identity for service account authentication
• Result: Lakekeeper initialization failed (separate issue)

2. Native GCS with Service Account Key

• Config: GCS service account key embedded in warehouse config
• Result: Works for Spark/BigQuery, but ClickHouse still sees gs:// URLs
• Error: Unsupported storage type: gs

3. S3-Compatible Storage with HMAC Keys

• Config: GCS HMAC keys for S3-compatible access
• Result: Spark remote signing failures
• Error: Forbidden: action can_get_metadata forbidden for Anonymous

4. rclone S3 Proxy (Database-Level Credentials)

• Config: rclone as S3 proxy, credentials in CREATE DATABASE statement
• Result: ClickHouse doesn't support S3 credentials in database settings
• Error: Unknown setting 's3_access_key_id'

5. rclone S3 Proxy (Server-Level Credentials)

• Config:

  env:
    - name: AWS_ACCESS_KEY_ID
      value: "..."
    - name: AWS_SECRET_ACCESS_KEY
      value: "..."
    - name: AWS_ENDPOINT_URL
      value: "http://s3-proxy:9000"

• Result: Database created, credentials work, but tables still invisible
• Error: Unsupported storage type: gs (metadata still contains gs://)

---

Supported Storage Protocols

Tool	s3://	gs://	hdfs://	file://
Spark	✅	✅	✅	✅
BigQuery	❌	✅	❌	❌
ClickHouse	✅	❌	✅	✅
Lakekeeper	✅	✅	❌	❌

Key Finding: ClickHouse is the only tool in our data stacks that doesn't support GCS.

---

Potential Solutions

Solution 1: Add Native GCS Support to ClickHouse ⭐ PREFERRED

Changes Required:

1. Add gs:// protocol support to ClickHouse's storage layer
2. Implement GCS client in DataLakeCatalog engine
3. Support GCS authentication (service account keys, Workload Identity)

Impact:

• ✅ Enables ClickHouse for all GCP users
• ✅ No migration required for existing Iceberg tables
• ✅ Works with any REST catalog (Nessie, Lakekeeper, Polaris, Unity)

Complexity: Medium-High (requires new storage backend)

---

Solution 2: Add URL Rewriting in DataLakeCatalog

Changes Required:

1. Add configuration for URL translation rules
2. Rewrite gs:// → s3:// when parsing Iceberg metadata
3. Route S3 requests to configured S3-compatible proxy

Example Config:

settings:
  iceberg/url_rewrite_enabled: "1"
  iceberg/url_rewrite_from: "gs://bucket-name"
  iceberg/url_rewrite_to: "s3://bucket-name"
  s3/endpoint_url: "http://gcs-proxy:9000"

Impact:

• ✅ Works with existing GCS tables via proxy
• ⚠️ Requires deploying S3-compatible proxy (rclone, MinIO gateway)
• ⚠️ Additional infrastructure complexity

Complexity: Medium (requires URL rewriting logic)

---

Additional Context

Related Issues

• Altinity/ClickHouse#1199 - Original report (Nessie + GCS)
• This report adds: Lakekeeper testing, 5 workaround attempts, detailed root cause

Tested Components

• REST Catalogs: Lakekeeper v0.6.0 (same issue expected with Nessie, Polaris, Unity)
• Iceberg Version: 1.10.0
• Spark Version: 3.5.1 (Dataproc Serverless)
• BigQuery: BigLake external tables (working)

Logs Available

We have detailed logs showing:

1. Successful catalog connection
2. Successful table discovery
3. Unsupported storage type: gs error
4. Stack traces from all 5 approaches

Happy to provide additional logs or testing if helpful.

[mkmkme]

Hey @redmode,

Thanks for the detailed report! I'll have a look at it as soon as possible.

@volokluev, so far I couldn't reproduce the issue exactly as you did. I'm getting "Connection refused" at the point of "SHOW TABLES", and Nessie log doesn't show any access to it during that query.

Would it be possible for you to give me the output of the following commands from your container:

docker exec -it clickhouse cat /var/log/clickhouse-server/clickhouse-server.log
docker exec -it clickhouse cat /var/log/clickhouse-server/clickhouse-server.err.log

Also, the output of SHOW TABLES would be helpful. Does it return any error or just an empty response?

In the meantime, I will try to fix the possible network issue in my container. Thanks!

[mkmkme]

Dear @volokluev,

I have managed to reproduce the issue and currently working on a possible solution for that. I'll let you know once I triage the issue