From af8a131b83bf55562ce538597c0b1738744c0114 Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Tue, 2 Dec 2025 16:36:48 -0800 Subject: [PATCH 1/7] ODBC Standalone MSVC Build CI Remove DataSet Fix cache key Use odbc-specific cache key Fix Lint and Add odbc registration step Link appropriate GitHub issues that blocks ODBC tests Disable test phase and add schedule Run everyday 7 am Vancouver time Enable ODBC build on MSVC CI Code Clean up and enable ODBC tests in CI Still need to modify ODBCUtilEnvironment if we decide to use it. Still need to add ODBC V2 support so a different env and conn is used for ODBC 2 tests. Draft enable ODBC global setup/teardown Run ODBC test once outside of test script If ODBC test is run using MinGW Shell, segfault occurs. I am not getting any seg fault on my local MSVC Windows when I run the tests without the bash script. But if Windows CI breaks from running the standalone exe then I will look into this Prepend vcpkg to search vcpkg before `/lib/cmake` Since we are installing dependencies on vcpkg, if `lib/cmake` is searched first, then cmake will look into that directory and use the wrong paths. Convert VCPKG Windows path to MSYS path Set `VCPKG_ROOT` in test phase Fix ODBC dll name Use `arrow_flight_sql_odbc.dll` instead of `libarrow_flight_sql_odbc.dll` which is the naming convention used on MinGW Windows Link ODBC library on all platforms On my local MSVC Windows machine, Visual Studio is used to build without needing to link ` ${ODBCINST}` explicitly, its behavior might be different from Ninja which is what CI uses. `${ODBCINST}` is likely needed by Linux as well, so adding it for all platforms. Attempt to resolve `arrow-compute-grouper-benchmark` build issue I think `if(ARROW_FLIGHT_TEST_LINKAGE STREQUAL "static")` might be more appropriate here, as I am seeing build issues due to both dynamic and static linking occurring. I see in https://github.com/apache/arrow/commit/59903d089ec5b1766e1622d94c1f618b539b205d, this check is used for `arrow-filesystem-s3fs-benchmark` Disable `UNITY_BUILD` for ODBC test due to conflict on `sqlite_sql_info.cc` On some workflows, unity build is set to ON to make the build faster. This is an attempt to resolve the build issue. Remove debug messages Fix lint Add `sql_info_undef.h` to sqlite_sql_info.cc Undefine duplicates in SQLGetInfo Add `#pragma once` to odbc_test_suite.h To avoid redefinition error Attempt to resolve conflict with sql/types.h Attempt to include Arrow headers before ODBC headers to avoid conflict with arrow/flight/sql/types.h [GH-48084] Replace boost::optional with std::optional Addresses comment https://github.com/apache/arrow/pull/40939#discussion_r2099118497 Replace boost::optional with std::optional in ODBC codebase https://github.com/apache/arrow/issues/48084 Fix lint Remove `BOOST_SOURCE=BUNDLED` to use vcpkg's dynamic link to boost Since we need to use link to boost dynamically, we need to remove the bundled boost library flag. Add debug messages. Remove `ARROW_BOOST_USE_SHARED = OFF` Since we have a release build, can enable boost as shared. With `ARROW_BOOST_USE_SHARED = OFF`, I am getting error ``` D:\a\arrow\arrow\build\cpp\vcpkg_installed\x64-windows\include\boost/filesystem/config.hpp(96): fatal error C1189: #error: Must not define both BOOST_FILESYSTEM_DYN_LINK and BOOST_FILESYSTEM_STATIC_LINK ``` Also increased time limit, since 2 hours don't seem to be enough to finish the vcpkg build Change to release build Getting ``` orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in unity_2_cxx.cxx.obj orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in unity_2_cxx.cxx.obj ``` when building with debug and presumably ARROW_ORC Set ARROW_BOOST_USE_SHARED to OFF `ARROW_BOOST_USE_SHARED` is restored to `OFF`, which according to Windows doc should help with the debug build now. Retore value of `ARROW_BUILD_BENCHMARKS`, though noting it is set to `OFF` in GLib MSVC workflow. Add VCPKG set up Borrowed from the Glib workflow Add `/EHsc` flag * Add back `CMAKE_CXX_STANDARD: "17"` Remove `CMAKE_CXX_STANDARD` 17 * reason: gtest can't be used with C++17. Arrow project doesn't support C++ 17 yet. Extend run-time to 2hr windows-mingw also has timeout of 2hr Empty commit to trigger CI Specify `VCPKG_BINARY_SOURCES` and `VCPKG_DEFAULT_TRIPLET` Specify VCPKG_TARGET_TRIPLET as x64 windows Set ARROW_DEPENDENCY_SOURCE to VCPKG To make it easier to manage dependencies Enable static build on MSVC `ARROW_BUILD_BENCHMARKS` prevents Flight from being built Remove `ARROW_BOOST_USE_SHARED` Having static vs. shared issue Enable Flight & Flight SQL for ODBC Enable ODBC build on MSVC CI Enable regular ctest tests Remove undef items Add concurrency and permissions to odbc yml Create cpp_odbc.yml Fix architecture in CI * Add ODBC installer MSVC CI --- .github/workflows/cpp_odbc.yml | 182 ++++++++++++++++++ ci/scripts/cpp_test.sh | 3 +- cpp/cmake_modules/SetupCxxFlags.cmake | 1 + .../flight/sql/odbc/odbc_impl/CMakeLists.txt | 10 +- .../sql/odbc/odbc_impl/get_info_cache.cc | 6 +- .../sql/odbc/odbc_impl/odbc_connection.cc | 1 + .../flight/sql/odbc/tests/CMakeLists.txt | 3 + .../flight/sql/odbc/tests/odbc_test_suite.h | 2 + 8 files changed, 198 insertions(+), 10 deletions(-) create mode 100644 .github/workflows/cpp_odbc.yml diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml new file mode 100644 index 00000000000..bc34121dd9f --- /dev/null +++ b/.github/workflows/cpp_odbc.yml @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: C++ ODBC + +on: + push: + branches: + - '**' + - '!dependabot/**' + tags: + - '**' + paths: + - '.github/workflows/cpp_odbc.yml' + - 'ci/scripts/cpp_*' + - 'cpp/src/arrow/flight/sql/odbc/*' + pull_request: + paths: + - '.github/workflows/cpp_odbc.yml' + - 'ci/scripts/cpp_*' + - 'cpp/src/arrow/flight/sql/odbc/*' + schedule: + - cron: '0 13 * * *' + +concurrency: + group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + windows: + runs-on: windows-2022 + timeout-minutes: 240 + env: + ARROW_BUILD_SHARED: ON + ARROW_BUILD_STATIC: ON + ARROW_BUILD_TESTS: ON + ARROW_BUILD_TYPE: release + ARROW_DEPENDENCY_SOURCE: VCPKG + ARROW_FLIGHT: ON + ARROW_FLIGHT_SQL: ON + ARROW_FLIGHT_SQL_ODBC: ON + # GH-47787 TODO Build ODBC installer + # ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON + ARROW_SIMD_LEVEL: AVX2 + CMAKE_CXX_STANDARD: "17" + CMAKE_GENERATOR: Ninja + CMAKE_INSTALL_PREFIX: /usr + VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite' + VCPKG_DEFAULT_TRIPLET: x64-windows + steps: + - name: Disable Crash Dialogs + run: | + reg add ` + "HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" ` + /v DontShowUI ` + /t REG_DWORD ` + /d 1 ` + /f + - name: Checkout Arrow + uses: actions/checkout@v5 + with: + fetch-depth: 0 + submodules: recursive + - name: Download Timezone Database + shell: bash + run: ci/scripts/download_tz_database.sh + - name: Install msys2 (for tzdata for ORC tests) + uses: msys2/setup-msys2@v2 + id: setup-msys2 + - name: Install cmake + shell: bash + run: | + ci/scripts/install_cmake.sh 4.1.2 /usr + - name: Install ccache + shell: bash + run: | + ci/scripts/install_ccache.sh 4.12.1 /usr + - name: Setup ccache + shell: bash + run: | + ci/scripts/ccache_setup.sh + - name: ccache info + id: ccache-info + shell: bash + run: | + echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT + - name: Cache ccache + uses: actions/cache@v4 + with: + path: ${{ steps.ccache-info.outputs.cache-dir }} + key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} + restore-keys: cpp-odbc-ccache-windows-x64- + - name: Checkout vcpkg + uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 + with: + fetch-depth: 0 + path: vcpkg + repository: microsoft/vcpkg + - name: Bootstrap vcpkg + run: | + vcpkg\bootstrap-vcpkg.bat + $VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString() + Write-Output ${VCPKG_ROOT} | ` + Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append + Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | ` + Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append + - name: Setup NuGet credentials for vcpkg caching + shell: bash + run: | + $(vcpkg fetch nuget | tail -n 1) \ + sources add \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \ + -storepasswordincleartext \ + -name "GitHub" \ + -username "$GITHUB_REPOSITORY_OWNER" \ + -password "${{ secrets.GITHUB_TOKEN }}" + $(vcpkg fetch nuget | tail -n 1) \ + setapikey "${{ secrets.GITHUB_TOKEN }}" \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" + - name: Build + shell: cmd + run: | + set VCPKG_ROOT_KEEP=%VCPKG_ROOT% + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + set VCPKG_ROOT=%VCPKG_ROOT_KEEP% + bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build" + - name: Register Flight SQL ODBC Driver + shell: cmd + run: | + call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll + # GH-48270 TODO: Resolve segementation fault during Arrow library unload + # GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI + # TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved. + # - name: Test + # shell: cmd + # run: | + # set VCPKG_ROOT_KEEP=%VCPKG_ROOT% + # call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + # # For ORC + # set TZDIR=${{ steps.setup-msys2.outputs.msys2-location }}\usr\share\zoneinfo + + # # Convert VCPKG Windows path to MSYS path + # for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I + + # bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build" + + # GH-47787 TODO Build ODBC installer + # - name: Install WiX Toolset + # shell: pwsh + # run: | + # Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi + # Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait + # echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + # - name: Build MSI ODBC installer + # shell: pwsh + # run: | + # # Verify WiX version + # wix --version + # cd "${{ github.workspace }}\build\cpp" + # cpack + # - name: Upload the artifacts to the job + # uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0 + # with: + # name: flight-sql-odbc-msi-installer + # path: ${{ github.workspace }}\build\cpp\Apache Arrow Flight SQL ODBC-1.0.0-win64.msi diff --git a/ci/scripts/cpp_test.sh b/ci/scripts/cpp_test.sh index a0b77b11be2..729d2cea787 100755 --- a/ci/scripts/cpp_test.sh +++ b/ci/scripts/cpp_test.sh @@ -144,8 +144,9 @@ if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \ CMAKE_PREFIX_PATH+="/lib/cmake/" ;; esac + # Search vcpkg before /lib/cmake. if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then - CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET}" + CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}" fi cmake \ -S "${source_dir}/examples/minimal_build" \ diff --git a/cpp/cmake_modules/SetupCxxFlags.cmake b/cpp/cmake_modules/SetupCxxFlags.cmake index 3c172aebdf8..e0d1fa0dcca 100644 --- a/cpp/cmake_modules/SetupCxxFlags.cmake +++ b/cpp/cmake_modules/SetupCxxFlags.cmake @@ -186,6 +186,7 @@ if(WIN32) # # ARROW-2986: Without /EHsc we get C4530 warning set(CXX_COMMON_FLAGS "/W3 /EHsc") + string(APPEND CMAKE_CXX_FLAGS " /EHsc") endif() # Disable C5105 (macro expansion producing 'defined' has undefined diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt b/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt index 71a315660bf..2fe9c41e3ce 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt @@ -129,13 +129,9 @@ if(WIN32) system_dsn.h) endif() -target_link_libraries(arrow_odbc_spi_impl PUBLIC arrow_flight_sql_shared - arrow_compute_shared Boost::locale) - -# Link libraries on MINGW64 and macOS -if(MINGW OR APPLE) - target_link_libraries(arrow_odbc_spi_impl PUBLIC ${ODBCINST}) -endif() +target_link_libraries(arrow_odbc_spi_impl + PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale + ${ODBCINST}) set_target_properties(arrow_odbc_spi_impl PROPERTIES ARCHIVE_OUTPUT_DIRECTORY diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc b/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc index cd637be165e..7d6239f24a7 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc @@ -19,8 +19,6 @@ #include "arrow/flight/sql/odbc/odbc_impl/platform.h" -#include -#include #include "arrow/array.h" #include "arrow/array/array_nested.h" #include "arrow/flight/sql/api.h" @@ -32,6 +30,10 @@ #include "arrow/flight/sql/odbc/odbc_impl/scalar_function_reporter.h" #include "arrow/flight/sql/odbc/odbc_impl/util.h" +// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h +#include +#include + // Aliases for entries in SqlInfoOptions::SqlInfo that are defined here // due to causing compilation errors conflicting with ODBC definitions. #define ARROW_SQL_IDENTIFIER_CASE 503 diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc b/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc index d6dfb471bd0..59cdc2e12d9 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc @@ -29,6 +29,7 @@ #include "arrow/flight/sql/odbc/odbc_impl/spi/statement.h" #include "arrow/flight/sql/odbc/odbc_impl/util.h" +// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h #include #include #include diff --git a/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt b/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt index 2f4ed9964da..39284c750b0 100644 --- a/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt +++ b/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt @@ -51,3 +51,6 @@ add_arrow_test(flight_sql_odbc_test ${ODBCINST} ${SQLite3_LIBRARIES} arrow_odbc_spi_impl) + +# Disable unity build due to sqlite_sql_info.cc conflict with sql.h and sqlext.h headers. +set_target_properties(arrow-flight-sql-odbc-test PROPERTIES UNITY_BUILD OFF) diff --git a/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h b/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h index c349704cc52..7dd77d8fa62 100644 --- a/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h +++ b/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +#pragma once + #include "arrow/testing/gtest_util.h" #include "arrow/util/io_util.h" #include "arrow/util/utf8.h" From 8dc692a9f8db664c9e505c7109baedb1392aa86a Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Thu, 4 Dec 2025 11:02:38 -0800 Subject: [PATCH 2/7] Address comments from Kou --- .github/workflows/cpp_odbc.yml | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml index bc34121dd9f..a6da557b74e 100644 --- a/.github/workflows/cpp_odbc.yml +++ b/.github/workflows/cpp_odbc.yml @@ -81,9 +81,6 @@ jobs: - name: Download Timezone Database shell: bash run: ci/scripts/download_tz_database.sh - - name: Install msys2 (for tzdata for ORC tests) - uses: msys2/setup-msys2@v2 - id: setup-msys2 - name: Install cmake shell: bash run: | @@ -108,7 +105,7 @@ jobs: key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} restore-keys: cpp-odbc-ccache-windows-x64- - name: Checkout vcpkg - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 + uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 with: fetch-depth: 0 path: vcpkg @@ -153,8 +150,6 @@ jobs: # run: | # set VCPKG_ROOT_KEEP=%VCPKG_ROOT% # call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - # # For ORC - # set TZDIR=${{ steps.setup-msys2.outputs.msys2-location }}\usr\share\zoneinfo # # Convert VCPKG Windows path to MSYS path # for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I From 7841861e2e3e39588fd3ad6cd98f1463979661ec Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Fri, 5 Dec 2025 15:39:05 -0800 Subject: [PATCH 3/7] Attempt to move ODBC CI to `cpp_extra.yml` - Use newest `actions/checkout` Co-Authored-By: alinalibq --- .github/workflows/cpp_extra.yml | 12 ++++++++++++ .github/workflows/cpp_odbc.yml | 29 +++-------------------------- 2 files changed, 15 insertions(+), 26 deletions(-) diff --git a/.github/workflows/cpp_extra.yml b/.github/workflows/cpp_extra.yml index 5645a8d0d3b..ae81fe7ff7d 100644 --- a/.github/workflows/cpp_extra.yml +++ b/.github/workflows/cpp_extra.yml @@ -26,6 +26,7 @@ on: - '.dockerignore' - '.github/workflows/check_labels.yml' - '.github/workflows/cpp_extra.yml' + - '.github/workflows/cpp_odbc.yml' - '.github/workflows/report_ci.yml' - 'ci/conda_env_*' - 'ci/docker/**' @@ -47,6 +48,7 @@ on: - '.dockerignore' - '.github/workflows/check_labels.yml' - '.github/workflows/cpp_extra.yml' + - '.github/workflows/cpp_odbc.yml' - '.github/workflows/report_ci.yml' - 'ci/conda_env_*' - 'ci/docker/**' @@ -330,6 +332,15 @@ jobs: cd cpp/examples/minimal_build ../minimal_build.build/arrow-example + odbc: + needs: check-labels + if: >- + needs.check-labels.outputs.force == 'true' || + contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') || + contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++') + name: C++ ODBC + uses: ./.github/workflows/cpp_odbc.yml + report-extra-cpp: if: github.event_name == 'schedule' && always() needs: @@ -337,5 +348,6 @@ jobs: - jni-linux - jni-macos - msvc-arm64 + - odbc uses: ./.github/workflows/report_ci.yml secrets: inherit diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml index a6da557b74e..bd3bc54e355 100644 --- a/.github/workflows/cpp_odbc.yml +++ b/.github/workflows/cpp_odbc.yml @@ -18,30 +18,7 @@ name: C++ ODBC on: - push: - branches: - - '**' - - '!dependabot/**' - tags: - - '**' - paths: - - '.github/workflows/cpp_odbc.yml' - - 'ci/scripts/cpp_*' - - 'cpp/src/arrow/flight/sql/odbc/*' - pull_request: - paths: - - '.github/workflows/cpp_odbc.yml' - - 'ci/scripts/cpp_*' - - 'cpp/src/arrow/flight/sql/odbc/*' - schedule: - - cron: '0 13 * * *' - -concurrency: - group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} - cancel-in-progress: true - -permissions: - contents: read + workflow_call: jobs: windows: @@ -74,7 +51,7 @@ jobs: /d 1 ` /f - name: Checkout Arrow - uses: actions/checkout@v5 + uses: actions/checkout@v6 with: fetch-depth: 0 submodules: recursive @@ -105,7 +82,7 @@ jobs: key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} restore-keys: cpp-odbc-ccache-windows-x64- - name: Checkout vcpkg - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 + uses: actions/checkout@v6 with: fetch-depth: 0 path: vcpkg From a25015a043aaeb9903377f745f83d1a5d7d4f6e1 Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Mon, 8 Dec 2025 10:38:01 -0800 Subject: [PATCH 4/7] Keep only `ARROW_FLIGHT_SQL_ODBC` flag on `cpp_odbc.yml` --- .github/workflows/cpp_odbc.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml index bd3bc54e355..f554ffa2121 100644 --- a/.github/workflows/cpp_odbc.yml +++ b/.github/workflows/cpp_odbc.yml @@ -30,8 +30,6 @@ jobs: ARROW_BUILD_TESTS: ON ARROW_BUILD_TYPE: release ARROW_DEPENDENCY_SOURCE: VCPKG - ARROW_FLIGHT: ON - ARROW_FLIGHT_SQL: ON ARROW_FLIGHT_SQL_ODBC: ON # GH-47787 TODO Build ODBC installer # ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON From fed94f0501537139058bad8f6717e60e3432cc3f Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Mon, 15 Dec 2025 11:05:38 -0800 Subject: [PATCH 5/7] Address Kou's comments --- .github/workflows/cpp_odbc.yml | 36 +++------------------------------- ci/scripts/cpp_test.sh | 2 +- 2 files changed, 4 insertions(+), 34 deletions(-) diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml index f554ffa2121..c92f214be80 100644 --- a/.github/workflows/cpp_odbc.yml +++ b/.github/workflows/cpp_odbc.yml @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -name: C++ ODBC +name: ODBC on: workflow_call: @@ -26,13 +26,11 @@ jobs: timeout-minutes: 240 env: ARROW_BUILD_SHARED: ON - ARROW_BUILD_STATIC: ON + ARROW_BUILD_STATIC: OFF ARROW_BUILD_TESTS: ON ARROW_BUILD_TYPE: release ARROW_DEPENDENCY_SOURCE: VCPKG ARROW_FLIGHT_SQL_ODBC: ON - # GH-47787 TODO Build ODBC installer - # ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON ARROW_SIMD_LEVEL: AVX2 CMAKE_CXX_STANDARD: "17" CMAKE_GENERATOR: Ninja @@ -120,33 +118,5 @@ jobs: # GH-48270 TODO: Resolve segementation fault during Arrow library unload # GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI # TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved. - # - name: Test - # shell: cmd - # run: | - # set VCPKG_ROOT_KEEP=%VCPKG_ROOT% - # call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - # # Convert VCPKG Windows path to MSYS path - # for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I - - # bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build" - - # GH-47787 TODO Build ODBC installer - # - name: Install WiX Toolset - # shell: pwsh - # run: | - # Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi - # Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait - # echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append - # - name: Build MSI ODBC installer - # shell: pwsh - # run: | - # # Verify WiX version - # wix --version - # cd "${{ github.workspace }}\build\cpp" - # cpack - # - name: Upload the artifacts to the job - # uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0 - # with: - # name: flight-sql-odbc-msi-installer - # path: ${{ github.workspace }}\build\cpp\Apache Arrow Flight SQL ODBC-1.0.0-win64.msi + # GH-47787 TODO Build ODBC installer \ No newline at end of file diff --git a/ci/scripts/cpp_test.sh b/ci/scripts/cpp_test.sh index 729d2cea787..0ad59bc308f 100755 --- a/ci/scripts/cpp_test.sh +++ b/ci/scripts/cpp_test.sh @@ -144,8 +144,8 @@ if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \ CMAKE_PREFIX_PATH+="/lib/cmake/" ;; esac - # Search vcpkg before /lib/cmake. if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then + # Search vcpkg before /lib/cmake. CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}" fi cmake \ From 20d2e85d02630aaa3770f11f697a140efe5e2d4d Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Mon, 15 Dec 2025 11:26:27 -0800 Subject: [PATCH 6/7] Move cpp_odbc to cpp_extra --- .github/workflows/cpp_extra.yml | 103 +++++++++++++++++++++++++-- .github/workflows/cpp_odbc.yml | 122 -------------------------------- 2 files changed, 99 insertions(+), 126 deletions(-) delete mode 100644 .github/workflows/cpp_odbc.yml diff --git a/.github/workflows/cpp_extra.yml b/.github/workflows/cpp_extra.yml index ae81fe7ff7d..06a6b754858 100644 --- a/.github/workflows/cpp_extra.yml +++ b/.github/workflows/cpp_extra.yml @@ -26,7 +26,6 @@ on: - '.dockerignore' - '.github/workflows/check_labels.yml' - '.github/workflows/cpp_extra.yml' - - '.github/workflows/cpp_odbc.yml' - '.github/workflows/report_ci.yml' - 'ci/conda_env_*' - 'ci/docker/**' @@ -48,7 +47,6 @@ on: - '.dockerignore' - '.github/workflows/check_labels.yml' - '.github/workflows/cpp_extra.yml' - - '.github/workflows/cpp_odbc.yml' - '.github/workflows/report_ci.yml' - 'ci/conda_env_*' - 'ci/docker/**' @@ -334,12 +332,109 @@ jobs: odbc: needs: check-labels + name: ODBC + runs-on: windows-2022 if: >- needs.check-labels.outputs.force == 'true' || contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') || contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: C++') - name: C++ ODBC - uses: ./.github/workflows/cpp_odbc.yml + timeout-minutes: 240 + env: + ARROW_BUILD_SHARED: ON + ARROW_BUILD_STATIC: OFF + ARROW_BUILD_TESTS: ON + ARROW_BUILD_TYPE: release + ARROW_DEPENDENCY_SOURCE: VCPKG + ARROW_FLIGHT_SQL_ODBC: ON + ARROW_SIMD_LEVEL: AVX2 + CMAKE_CXX_STANDARD: "17" + CMAKE_GENERATOR: Ninja + CMAKE_INSTALL_PREFIX: /usr + VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite' + VCPKG_DEFAULT_TRIPLET: x64-windows + steps: + - name: Disable Crash Dialogs + run: | + reg add ` + "HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" ` + /v DontShowUI ` + /t REG_DWORD ` + /d 1 ` + /f + - name: Checkout Arrow + uses: actions/checkout@v6 + with: + fetch-depth: 0 + submodules: recursive + - name: Download Timezone Database + shell: bash + run: ci/scripts/download_tz_database.sh + - name: Install cmake + shell: bash + run: | + ci/scripts/install_cmake.sh 4.1.2 /usr + - name: Install ccache + shell: bash + run: | + ci/scripts/install_ccache.sh 4.12.1 /usr + - name: Setup ccache + shell: bash + run: | + ci/scripts/ccache_setup.sh + - name: ccache info + id: ccache-info + shell: bash + run: | + echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT + - name: Cache ccache + uses: actions/cache@v4 + with: + path: ${{ steps.ccache-info.outputs.cache-dir }} + key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} + restore-keys: cpp-odbc-ccache-windows-x64- + - name: Checkout vcpkg + uses: actions/checkout@v6 + with: + fetch-depth: 0 + path: vcpkg + repository: microsoft/vcpkg + - name: Bootstrap vcpkg + run: | + vcpkg\bootstrap-vcpkg.bat + $VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString() + Write-Output ${VCPKG_ROOT} | ` + Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append + Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | ` + Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append + - name: Setup NuGet credentials for vcpkg caching + shell: bash + run: | + $(vcpkg fetch nuget | tail -n 1) \ + sources add \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \ + -storepasswordincleartext \ + -name "GitHub" \ + -username "$GITHUB_REPOSITORY_OWNER" \ + -password "${{ secrets.GITHUB_TOKEN }}" + $(vcpkg fetch nuget | tail -n 1) \ + setapikey "${{ secrets.GITHUB_TOKEN }}" \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" + - name: Build + shell: cmd + run: | + set VCPKG_ROOT_KEEP=%VCPKG_ROOT% + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + set VCPKG_ROOT=%VCPKG_ROOT_KEEP% + bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build" + - name: Register Flight SQL ODBC Driver + shell: cmd + run: | + call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll + # GH-48270 TODO: Resolve segementation fault during Arrow library unload + # GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI + # GH-48547 TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved. + + # GH-47787 TODO: Build ODBC installer report-extra-cpp: if: github.event_name == 'schedule' && always() diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml deleted file mode 100644 index c92f214be80..00000000000 --- a/.github/workflows/cpp_odbc.yml +++ /dev/null @@ -1,122 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -name: ODBC - -on: - workflow_call: - -jobs: - windows: - runs-on: windows-2022 - timeout-minutes: 240 - env: - ARROW_BUILD_SHARED: ON - ARROW_BUILD_STATIC: OFF - ARROW_BUILD_TESTS: ON - ARROW_BUILD_TYPE: release - ARROW_DEPENDENCY_SOURCE: VCPKG - ARROW_FLIGHT_SQL_ODBC: ON - ARROW_SIMD_LEVEL: AVX2 - CMAKE_CXX_STANDARD: "17" - CMAKE_GENERATOR: Ninja - CMAKE_INSTALL_PREFIX: /usr - VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite' - VCPKG_DEFAULT_TRIPLET: x64-windows - steps: - - name: Disable Crash Dialogs - run: | - reg add ` - "HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" ` - /v DontShowUI ` - /t REG_DWORD ` - /d 1 ` - /f - - name: Checkout Arrow - uses: actions/checkout@v6 - with: - fetch-depth: 0 - submodules: recursive - - name: Download Timezone Database - shell: bash - run: ci/scripts/download_tz_database.sh - - name: Install cmake - shell: bash - run: | - ci/scripts/install_cmake.sh 4.1.2 /usr - - name: Install ccache - shell: bash - run: | - ci/scripts/install_ccache.sh 4.12.1 /usr - - name: Setup ccache - shell: bash - run: | - ci/scripts/ccache_setup.sh - - name: ccache info - id: ccache-info - shell: bash - run: | - echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT - - name: Cache ccache - uses: actions/cache@v4 - with: - path: ${{ steps.ccache-info.outputs.cache-dir }} - key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} - restore-keys: cpp-odbc-ccache-windows-x64- - - name: Checkout vcpkg - uses: actions/checkout@v6 - with: - fetch-depth: 0 - path: vcpkg - repository: microsoft/vcpkg - - name: Bootstrap vcpkg - run: | - vcpkg\bootstrap-vcpkg.bat - $VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString() - Write-Output ${VCPKG_ROOT} | ` - Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append - Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | ` - Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append - - name: Setup NuGet credentials for vcpkg caching - shell: bash - run: | - $(vcpkg fetch nuget | tail -n 1) \ - sources add \ - -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \ - -storepasswordincleartext \ - -name "GitHub" \ - -username "$GITHUB_REPOSITORY_OWNER" \ - -password "${{ secrets.GITHUB_TOKEN }}" - $(vcpkg fetch nuget | tail -n 1) \ - setapikey "${{ secrets.GITHUB_TOKEN }}" \ - -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" - - name: Build - shell: cmd - run: | - set VCPKG_ROOT_KEEP=%VCPKG_ROOT% - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - set VCPKG_ROOT=%VCPKG_ROOT_KEEP% - bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build" - - name: Register Flight SQL ODBC Driver - shell: cmd - run: | - call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll - # GH-48270 TODO: Resolve segementation fault during Arrow library unload - # GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI - # TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved. - - # GH-47787 TODO Build ODBC installer \ No newline at end of file From 4ad7af19b74cd5231adbf03890b6d748d941179a Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Mon, 15 Dec 2025 14:05:45 -0800 Subject: [PATCH 7/7] Address Kou's comments Dec-16 Move `include(ThirdpartyToolchain)` --- cpp/CMakeLists.txt | 4 ++-- cpp/cmake_modules/SetupCxxFlags.cmake | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 7286616c4fb..c9f026f926a 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -520,8 +520,6 @@ endif() set(PARQUET_PC_REQUIRES "") set(PARQUET_PC_REQUIRES_PRIVATE "") -include(ThirdpartyToolchain) - # Add common flags set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX_COMMON_FLAGS}") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${ARROW_CXXFLAGS}") @@ -536,6 +534,8 @@ string(REPLACE "-std=c++17" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS}) # Add C++-only flags, like -std=c++17 set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}") +include(ThirdpartyToolchain) + # ASAN / TSAN / UBSAN if(ARROW_FUZZING) set(ARROW_USE_COVERAGE ON) diff --git a/cpp/cmake_modules/SetupCxxFlags.cmake b/cpp/cmake_modules/SetupCxxFlags.cmake index e0d1fa0dcca..3c172aebdf8 100644 --- a/cpp/cmake_modules/SetupCxxFlags.cmake +++ b/cpp/cmake_modules/SetupCxxFlags.cmake @@ -186,7 +186,6 @@ if(WIN32) # # ARROW-2986: Without /EHsc we get C4530 warning set(CXX_COMMON_FLAGS "/W3 /EHsc") - string(APPEND CMAKE_CXX_FLAGS " /EHsc") endif() # Disable C5105 (macro expansion producing 'defined' has undefined