Skip to content

Add timeout to temp table lineage graph node processing #25409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mohittilala wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add timeout to temp table lineage graph node processing #25409

mohittilala wants to merge 3 commits into from
+576 −3

Conversation

@mohittilala
Copy link
Contributor

@mohittilala mohittilala commented Jan 20, 2026 •
edited by gitar-bot bot
Loading

Describe your changes:

This PR adds timeout of 30 seconds to each node processing for temporary lineage creation and related tests.

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Summary by Gitar

  • Timeout mechanism:
    • Added 30-second timeout to _get_paths_from_subtree using @timeout decorator to prevent indefinite hangs on complex lineage graphs
  • Path processing improvement:
    • Modified to collect paths per root node before extending main list, ensuring complete path sets even when timeout occurs
  • New test file:
    • test_temp_table_lineage.py (473 lines) with comprehensive graph processing tests covering various topologies, timeout behavior, and edge cases
  • E2E validation:
    • Added temp table lineage tests in test_query_lineage.py and SQL fixtures validating lineage traces through deleted temporary tables

This will update automatically on new commits.

@mohittilala mohittilala self-assigned this Jan 20, 2026
@mohittilala mohittilala requested a review from a team as a code owner January 20, 2026 13:06
@mohittilala mohittilala added lineage To release Will cherry-pick this PR into the release branch labels Jan 20, 2026
@mohittilala mohittilala changed the title Add timeout to temp table graph node processing Add timeout to temp table lineage graph node processing Jan 20, 2026
@github-actions github-actions bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Jan 20, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 20, 2026

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 20, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.13)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (7)

Package Vulnerability ID Severity Installed Version Fixed Version
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
jaraco.context GHSA-58pv-8j8x-9vj2 🚨 HIGH 5.3.0 6.1.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

@github-actions
Copy link
Contributor

github-actions bot commented Jan 20, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
libpam-modules CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-modules-bin CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam-runtime CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2
libpam0g CVE-2025-6020 🚨 HIGH 1.5.2-6+deb12u1 1.5.2-6+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (15)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
aiohttp CVE-2025-69223 🚨 HIGH 3.12.12 3.13.3
aiohttp CVE-2025-69223 🚨 HIGH 3.13.2 3.13.3
apache-airflow CVE-2025-68438 🚨 HIGH 3.1.5 3.1.6
apache-airflow CVE-2025-68675 🚨 HIGH 3.1.5 3.1.6
azure-core CVE-2026-21226 🚨 HIGH 1.37.0 1.38.0
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
jaraco.context GHSA-58pv-8j8x-9vj2 🚨 HIGH 5.3.0 6.1.0
jaraco.context GHSA-58pv-8j8x-9vj2 🚨 HIGH 5.3.0 6.1.0
pyasn1 CVE-2026-23490 🚨 HIGH 0.6.1 0.6.2
ray CVE-2025-62593 🔥 CRITICAL 2.47.1 2.52.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@mohittilala mohittilala moved this to In Review / QA 👀 in Jan - 2026 Jan 20, 2026
@gitar-bot
Copy link

gitar-bot bot commented Jan 20, 2026 •
edited
Loading

🔍 CI failure analysis for 4512f72: Python tests need `enableTempTableLineage: true` fix (PR-related). Playwright failures are pre-existing UI test flakes recurring across multiple runs (unrelated to PR).

CI Failures Summary - Multiple CI Runs Analysis

This PR has two separate types of CI failures:

1. Python Unit Test Failures (Related to PR) - STILL FAILING

Status: Configuration issue in new tests - Affects Python 3.10 and 3.11

Failing Tests:

  • test_interim_target_table_temp_lineage
  • test_interim_target_view_temp_lineage

Root Cause: Missing enableTempTableLineage: true in ingestion/tests/unit/lineage/e2e/conftest.py:67-73

The fix is to add one line at line 71:

enableTempTableLineage: True,

Impact: 2 tests fail, 3,504+ tests pass

Multiple CI Runs: Python 3.10 and 3.11 jobs - 8+ runs with identical failure

2. Playwright E2E Test Failures (UNRELATED to PR) - RECURRING

Latest Job: playwright-ci-postgresql (2, 6) - Job ID 60902085913

Test Results:

  • 2 consistently failed (same as previous run):
    • TestCaseImportExportE2eFlow.spec.ts:88 - Admin: Complete export-import-validate flow
    • TestCaseImportExportE2eFlow.spec.ts:153 - EditAll User: Complete export-import-validate flow
  • 5 flaky (passed on retry):
    • Advanced Search (Schema Field OR operator)
    • Bulk Edit Entity (Glossary)
    • Bulk Import Export (Keyboard Delete)
    • Glossary Asset Operations
    • Glossary Hierarchy
  • 507 passed (92.6% success rate)

Comparison with Previous Playwright Job (60892874239):

  • Same 2 tests consistently failing
  • Similar number of flaky tests (5 vs 7)
  • Same test areas affected (Data Quality, Bulk Edit, Glossary)
  • Confirms pattern: pre-existing UI test flakiness

Evidence PR is UNRELATED:

  1. No UI Changes in PR:

    • ingestion/src/metadata/ingestion/lineage/sql_lineage.py (Python backend)
    • ingestion/tests/unit/lineage/e2e/data/lineage.sql (SQL test data)
    • ingestion/tests/unit/lineage/e2e/test_query_lineage.py (Python tests)
    • ingestion/tests/unit/lineage/test_temp_table_lineage.py (Python tests)

  2. Zero Frontend Changes:

    • No TypeScript files modified
    • No React components changed
    • No UI/frontend code touched

  3. Test Failures in UI Layer Only:

    • All Playwright failures are in frontend E2E tests
    • Areas: Data Quality UI, Bulk Edit UI, Glossary UI, Advanced Search UI
    • Cannot be caused by Python ingestion code

  4. Pattern Consistency:

    • Same tests failing across multiple independent Playwright runs
    • Both runs show 2 consistent failures + multiple flaky tests
    • Indicates pre-existing test infrastructure/stability issues

Failure Characteristics (Infrastructure/Timing Issues):

  • Test timeouts (180s exceeded)
  • Browser context closures
  • Element timing/rendering race conditions
  • API response timeouts

Summary

PR-Related Failures (Actionable by PR Author):

Python Unit Tests:

  • Issue: Missing test configuration flag
  • Solution: Add enableTempTableLineage: True, at line 71 in conftest.py
  • Scope: 2 tests failing across Python 3.10 and 3.11
  • Fix: Single line in test configuration file

Unrelated Failures (NOT Actionable by PR Author):

Playwright E2E Tests:

  • Issue: Pre-existing flaky UI tests in Data Quality, Bulk Edit, Glossary areas
  • Cause: Test infrastructure/timing issues, NOT related to Python backend changes
  • Evidence: PR modifies zero frontend files; failures are in UI layer only
  • Pattern: Consistent across multiple independent Playwright CI runs
  • Success Rate: Still 92.6% (507/549 tests pass)

Recommendation: The Playwright failures should not block this PR as they are pre-existing UI test stability issues unrelated to the Python lineage changes.

Code Review 👍 Approved with suggestions 0 resolved / 1 findings

Good implementation of timeout protection for lineage graph processing with comprehensive test coverage, though one test has a minor assertion issue.

Suggestions 💡 1 suggestion
Bug: Test assertion contradicts patched timeout value

📄 ingestion/tests/unit/lineage/test_temp_table_lineage.py:483-488

In test_timeout_logs_warning, the test uses @patch("metadata.ingestion.lineage.sql_lineage.NODE_PROCESSING_TIMEOUT", 1) to patch the timeout to 1 second, but then asserts assert NODE_PROCESSING_TIMEOUT == 30.

The assertion checks the imported constant at the top of the test file (which remains 30), not the patched value in the module. This test doesn't actually verify timeout behavior - it just confirms the default constant value.

To properly test timeout behavior, consider:

  1. Remove the @patch decorator if only checking the default value
  2. Or, if testing actual timeout behavior, mock the timeout decorator or use a slow operation that exceeds 1 second and verify the warning is logged

What Works Well

Clean timeout implementation using a decorator pattern around root node processing. Comprehensive unit tests covering various graph topologies (linear, diamond, DAG), edge cases (unicode, self-loops, long names), and integration scenarios. The e2e tests properly validate the temp table lineage feature end-to-end.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off Gitar will not commit updates to this branch.
Display: compact Hiding non-applicable rules.

Comment with these commands to change:

Auto-apply Compact 
gitar auto-apply:on         

gitar display:verbose

Was this helpful? React with 👍 / 👎 | This comment will update automatically (Docs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@mohittilala mohittilala

Labels

Ingestion lineage safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

Jan - 2026
Status: In Review / QA 👀

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mohittilala