Update a ClickHouse Contrib (Third-Party Library)

Bump a git submodule under contrib/ to a new version, adapting CMake build files and ClickHouse source code as needed.

Arguments

• $0 (required): Library name as it appears under contrib/ (e.g., curl, openssl, arrow)
• $1 (optional): Target version — a git tag, branch, or commit SHA. If omitted, the latest upstream release tag is used.

Two modes

Interactive (a human runs /update-contrib): follow every step below, including the confirmations.

Non-interactive / pipeline mode: the invoking prompt states that the run is automated and passes the target, the branch name, the build directory and often a facts block (submodule URL, current pin, target commit, fork branch, CVE list). In that mode:

• a given target is confirmed — skip step 2's discovery and never call AskUserQuestion;
• never push, never create branches in forks, never open PRs, never use gh; the pipeline pushes and opens the PR;
• do not re-derive facts the prompt already states (URL, pin, tags, build flags) — the pipeline fetched full history and tags into the submodule before the run;
• if the update needs a change inside a ClickHouse fork that does not exist yet, stop and write the reason to the BLOCKED.md path named in the prompt instead of improvising;
• finish by writing the report the prompt asks for (REPORT.md) with an explicit "Open concerns" section ("none" when there are none).

Background

ClickHouse vendors all third-party libraries as git submodules under contrib/. Each library has:

• contrib/<lib>/ — the submodule (upstream repo or a ClickHouse fork)
• contrib/<lib>-cmake/ — ClickHouse's own CMake build files

contrib/update-submodules.sh deletes the upstream CMakeLists.txt files after checkout; ClickHouse relies exclusively on its own *-cmake/ files. Consequences: git -C contrib/<lib> status normally shows deleted upstream CMake files — that is expected, do not restore them and do not re-run the deletion by hand; ninja does not need it either.

Submodule URLs point to either upstream directly (unpatched) or a fork at github.com/ClickHouse/<repo> (when patches are needed; the repo name may differ from the contrib name, e.g. contrib/cassandra → ClickHouse/cpp-driver). Fork branches are named ClickHouse/<version> — usually the bare upstream tag (ClickHouse/v1.52.1, ClickHouse/2.17.1), sometimes with the upstream's own prefix (ClickHouse/openssl-3.5.8, ClickHouse/release-2.14.1) — never ClickHouse/master or ClickHouse/main. A pin can also sit on a fork branch while .gitmodules still names the upstream URL (GitHub serves fork-network objects through the parent repo); check the fork's branches before assuming a pin is a plain upstream commit.

Submodules are cloned shallow in most checkouts. git describe and git log OLD..NEW then report nonsense or fail; run git -C contrib/<lib> fetch --unshallow --tags origin first (the pipeline does this for you).

A few libraries are built through cargo instead of a plain CMake file list — either from rust/workspace/<lib>/ (wasmtime) or from a contrib/<lib>-cmake/ file that drives cargo (chdig, delta-kernel-rs); they export ch_rust::* targets. See rust.md for those.

Process

1. Gather information about the library

Throughout this skill, $LIB refers to the library name passed as $0 and $VERSION to the target version once resolved (step 2). Both must match [A-Za-z0-9._-]+ so they are safe to use unquoted in paths and branch names; reject anything else.

Skip whatever the prompt already states. Otherwise:

git config --file .gitmodules --get "submodule.contrib/$LIB.url"   # upstream or ClickHouse fork?
git -C "contrib/$LIB" rev-parse HEAD                                 # current pin
git -C "contrib/$LIB" fetch --unshallow --tags origin 2>/dev/null || git -C "contrib/$LIB" fetch --tags origin
git -C "contrib/$LIB" describe --tags --abbrev=0 2>/dev/null || echo "no tags"

Then locate the integration file(s) and keep them in $INTEGRATION; every later step works from that variable. Three shapes exist:

INTEGRATION=$(ls contrib/${LIB}-cmake/CMakeLists.txt rust/workspace/${LIB}/CMakeLists.txt 2>/dev/null)
# Shared wrapper: no directory of its own, built by another contrib's CMake file
# (aws-c-auth → contrib/aws-cmake/, the aws-c-* family; individual boost libs → contrib/boost-cmake/).
[ -n "$INTEGRATION" ] || INTEGRATION=$(grep -rl "contrib/${LIB}\b" contrib/*-cmake/CMakeLists.txt)
echo "$INTEGRATION"; [ -n "$INTEGRATION" ] || echo "no integration file found — stop and report"
grep -n "\b${LIB}\b" contrib/CMakeLists.txt rust/workspace/CMakeLists.txt   # where it is added

• plain CMake: contrib/<lib>-cmake/CMakeLists.txt;
• Rust: either rust/workspace/<lib>/CMakeLists.txt (wasmtime) or a contrib/<lib>-cmake/ file that drives cargo (chdig, delta-kernel-rs) — see rust.md;
• shared wrapper: the fallback above; the wrapper's file list and aliases (ch_contrib::aws_s3 for every aws-c-* library) are what steps 8, 9 and 11 must use.

Do not assume the exported target is ch_contrib::${LIB}. Take the actual alias names from $INTEGRATION and grep for those — Rust-backed libraries export ch_rust::* (ch_rust::wasmtime, ch_rust::chdig, ch_rust::delta_kernel_rs), a few keep upstream-style names (OpenSSL::SSL, boost::filesystem), and one library can export several:

ALIASES=$(grep -ho 'add_library *( *[A-Za-z0-9_:.-]*::[A-Za-z0-9_.-]* *ALIAS' $INTEGRATION \
          | sed -E 's/add_library *\( *//; s/ *ALIAS$//' | sort -u)
echo "$ALIASES"
for a in $ALIASES; do grep -rl --include=CMakeLists.txt -F "$a" src programs base rust; done | sort -u   # consumers

Read $INTEGRATION once; it tells you which sources are compiled, which defines and generated headers exist, and whether the library is header-only (add_library(... INTERFACE) — then nothing of it is compiled and only its consumers can break). The consumer list is what steps 9 and 11 work from.

2. Determine the target version

If $1 was provided, use it. Otherwise, find the latest release:

git -C "contrib/$LIB" tag -l --sort=-v:refname | head -20
# For ClickHouse forks, also check the upstream repo: git ls-remote --tags <upstream-url>

Interactive mode only: present the current and target versions to the user with AskUserQuestion for confirmation before proceeding. VERSION is whatever was confirmed (or $1). Verify it matches the charset above before using it in branch names or commits.

3. Check for ClickHouse patches

First decide whether the pin is a plain upstream commit. Do not rely on the .gitmodules URL alone: origin is the upstream repo when the URL names upstream, and ls-remote --heads origin 'ClickHouse/*' then returns nothing even if the pin sits on a fork branch. Check that HEAD is reachable from an upstream ref, and probe the fork separately (pipeline mode: the facts block names the fork URL and branch — use them):

git -C "contrib/$LIB" fetch --tags origin '+refs/heads/*:refs/remotes/origin/*'    # full upstream refs (step 1)
git -C "contrib/$LIB" branch -r --contains HEAD | head -3                          # empty: not an upstream commit
git -C "contrib/$LIB" tag --contains HEAD | head -3

UPSTREAM_URL=$(git config --file .gitmodules --get "submodule.contrib/$LIB.url")
FORK_URL="https://github.com/ClickHouse/$(basename "$UPSTREAM_URL" .git)"        # guess; may differ (cassandra → cpp-driver)
git -C "contrib/$LIB" fetch "$FORK_URL" '+refs/heads/ClickHouse/*:refs/remotes/ch-fork/ClickHouse/*' 2>/dev/null
git -C "contrib/$LIB" branch -r --contains HEAD 'ch-fork/*'                       # non-empty: pin lives on a fork branch

Classify:

• HEAD is on an upstream branch or tag and on no ch-fork/ClickHouse/* branch → unpatched upstream pin; skip the rest of this step.
• HEAD is on a ch-fork/ClickHouse/* branch → forked, patched pin (even when .gitmodules names upstream; step 5 fixes the URL). Continue below.
• HEAD is on neither and the fork guess failed (fetch errored or the repo name differs) → do not conclude "unpatched". Interactive mode: ask the user for the fork URL. Pipeline mode: stop and record the reason in BLOCKED.md.

For a forked pin, identify the patches that must survive the bump:

UPSTREAM_TAG=$(git -C "contrib/$LIB" describe --tags --abbrev=0)     # needs full history (step 1)
git -C "contrib/$LIB" log --oneline "$UPSTREAM_TAG"..HEAD              # ClickHouse-specific commits

Report the patches found. A patch is obsolete only when upstream now contains an equivalent fix — verify in the code, not by commit subject.

4. Create a branch for the update

git checkout -b "bump-${LIB}-${VERSION}"     # pipeline mode: use the branch name given in the prompt

5. Update the fork (if applicable)

If the library uses a ClickHouse fork and has patches, the new branch ClickHouse/<new-version> must exist in the fork with the patches rebased onto the upstream tag before the ClickHouse-side bump:

1. Interactive mode: prepare it in a clone of the fork — branch from the upstream tag, cherry-pick -x the patches in order (authors preserved), resolve conflicts from the new upstream code, drop only patches that upstream has absorbed. Do not push until the build is validated and the user confirms.
2. Pipeline mode: the branch is prepared by a separate job and named in the prompt; point the submodule at exactly that commit and never touch the fork.

If the library has no ClickHouse patches but uses a fork, switch the submodule to the upstream URL instead of creating a new fork branch (update .gitmodules, see step 6). Conversely, if step 3 found the pin on a fork branch while .gitmodules names upstream, fix the URL to the fork in the same commit. Never rewrite a URL to upstream unless step 3 positively classified the pin as an unpatched upstream commit.

6. Update the submodule pointer

git -C "contrib/$LIB" fetch origin              # plus the fork URL / branch when the target lives there
git -C "contrib/$LIB" checkout <target-commit>
git add "contrib/$LIB"

If the submodule URL changes (fork → upstream, or a fork whose URL was wrong), also update .gitmodules and run git submodule sync -- "contrib/$LIB":

git config --file .gitmodules "submodule.contrib/$LIB.url" "<new-url>"
git add .gitmodules

git submodule status must show no + line for the library when you are done (a + means the working tree is not at the recorded commit).

7. Diff the source tree between old and new versions

This is the step that determines what CMake and source changes are needed:

OLD_COMMIT=$(git rev-parse "HEAD:contrib/$LIB"); NEW_COMMIT=$(git -C "contrib/$LIB" rev-parse HEAD)
git -C "contrib/$LIB" diff --stat "$OLD_COMMIT..$NEW_COMMIT"
git -C "contrib/$LIB" diff --name-status "$OLD_COMMIT..$NEW_COMMIT" -- '*.c' '*.cpp' '*.cc' '*.h' '*.hpp' '*.in' 'CMakeLists.txt'

Pay attention to:

• Added/removed/renamed source files — the CMake file list must follow
• New build options or defines — may need new CMake variables or hardcoded config headers
• Changed public headers — may require source adaptation in src/
• Changed *.in templates / generated headers — regenerate the delta, never overwrite a checked-in generated header wholesale (diff generator(old) vs generator(new) and apply only that)

Read the changelog/release notes for API and behaviour changes (stored formats, output changes, dependency version floors that force co-bumps).

8. Update build integration files

Update the file(s) in $INTEGRATION from step 1 (contrib/<lib>-cmake/CMakeLists.txt, a shared wrapper such as contrib/aws-cmake/CMakeLists.txt, or the Rust files — see rust.md) to reflect the new version:

1. Compare source file lists against the actual files in the updated submodule; add new files after the first existing sibling from the same directory, remove deleted ones.
2. Check for new required defines, include paths, or compile options.
3. Verify the result follows ClickHouse CMake conventions — no find_package/find_path/find_library, check_c_compiler_flag, check_cxx_compiler_flag, check_c_source_compiles, check_include_file, check_symbol_exists, check_type_size, cmake_push_check_state, CMAKE_REQUIRED_FLAGS, or any Check* CMake module — these are forbidden by CI style checks (ci/jobs/scripts/check_style/check_cpp.sh) for hermetic, cross-compiled builds. Use hardcoded feature flags instead of runtime detection.
4. If contrib/<lib>-cmake/ carries config headers (config.h.in, hardcoded *_config.h), update them for new version strings, defines or feature flags.
5. Anything executed at build time (a code generator such as protoc or flatc) must be a native host tool: register it with add_native_target, use the IMPORTED_LOCATION under native/ for cross builds, and add its directory from the block in contrib/CMakeLists.txt where disable_dummy_launchers_if_needed restores the real toolchain (clang-tidy builds otherwise produce an empty, non-executable binary).
6. If the version is read from a file at configure time (e.g. liburing's .spec), re-run cmake after the bump; otherwise incremental builds keep a stale generated version header.

9. Fix ClickHouse source code

Find the consumers (step 1's alias grep — ch_contrib::*, ch_rust::* or whatever the integration file exports) and, if the public API changed, the includes:

grep -rl "#include.*[<\"]$LIB" src/ --include='*.h' --include='*.cpp'

Common adaptation patterns: renamed functions or classes, changed signatures, removed deprecated API, new required initialisation calls, changed header paths, missing feature guards in copied public headers.

Build to find compilation errors, using the existing build directory ($BUILD_DIR, the one named in the prompt or an existing build*/ directory; configure one with cmake -S . -B <dir> only if none exists — see docs/resources/develop-contribute/build/build.mdx):

ninja -C "$BUILD_DIR" <lib-target> > "$BUILD_DIR/build_bump_${LIB}_lib.log" 2>&1; echo "exit=$?"
ninja -C "$BUILD_DIR" clickhouse  > "$BUILD_DIR/build_bump_${LIB}.log" 2>&1;     echo "exit=$?"
grep ' error:' "$BUILD_DIR/build_bump_${LIB}.log" | head -20 || true   # grep exits 1 on a clean log: not a failure

Run builds in the foreground and wait for them; never start them in the background and poll, and never use pgrep -f/kill -0 loops on a pattern that matches your own shell. The ninja exit code decides whether the build passed (exit=0); the grep only lists the errors to fix, and its own exit status is 1 when there are none, so do not chain it with && or run it under set -e. Do not spawn a sub-agent to read the log. For a header-only library the clickhouse target is the only thing that can break. Fix errors iteratively, committing each logical fix separately.

Do not run unit-test or gtest binaries unless the task asks for it, and never run any binary with the repository root as the working directory (it litters the checkout with store/, access/, preprocessed_configs/, test_*); use the build directory or /tmp.

10. Verify the build

After all fixes, rebuild the library target and clickhouse once more and check the exit codes as above. Common issues: missing source files in CMake, changed include paths, API changes requiring source adaptation, platform-specific issues (macOS, FreeBSD, musl, cross-compilation) that only CI can show, stale generated headers.

Interactive mode: only after the build is validated ask the user whether it is ok to push. Until then, keep both the ClickHouse branch and any fork branch local-only.

11. Check behaviour, not just compilation

If the library's output is user-visible (formatting, hashing, compression, geometry, text processing), compare a few representative queries between the previous binary and the new one. Byte-identical output is required for text formatting and hashes; for floating-point, geometry and compressor libraries a difference is acceptable only when the decoded data round-trips identically, every difference is explained by a named upstream change, and the affected stateless test references are updated. A stored/on-disk/wire format change (codec frame version, serialisation) is a compatibility break: state exactly what becomes unreadable and which setting gates the feature.

Then find the stateless tests that exercise the library and run them if a server is available to you (pipeline mode: the pipeline runs them, just list them in the report):

grep -rl "$LIB" tests/queries/0_stateless/ --include='*.sql' --include='*.sh' | head -20

12. Commit

Use the title format Bump \` from <old_version> to <new_version> for straightforward bumps, or a more descriptive title if the update has a specific motivation. Stage every affected integration artifact, not just the submodule pointer (.gitmodules, contrib/-cmake/, src/` fixes, test references). Do not amend or rebase; iterative "Fix build" / "Fix darwin" / "Fix MSan" commits are normal.

Examples of good commit messages from past PRs:

• Bump \curl` to 8.12.1`
• Update Boost from 1.83 to 1.90
• Update librdkafka to fix lock-order-inversion in queue refcount

13. Open a draft PR (interactive mode only)

Before creating the PR, make sure every required branch has been pushed with the user's confirmation: the ClickHouse branch, any ClickHouse/<version> branch in a contrib fork (opened as a PR against that fork's base branch first), and any rust_vendor branch.

Use the PR template at .github/PULL_REQUEST_TEMPLATE.md. Keep the description short and structured: one sentence of what and why; bullets for CVEs with their applicability; bullets for the ClickHouse-side changes; bullets for user-visible behaviour changes (a stored-format change is a compatibility warning); for forks, which patches are carried. Never enumerate what did not change. Changelog category for contrib bumps is Build/Testing/Packaging Improvement with the entry Update `<lib>` to <version>.; use Bug Fix only when the PR exists to fix a specific user-visible bug.

gh pr create --draft --title "<title>" --body-file <body.md>

Libraries with known dependency chains

Some libraries require co-bumping dependencies. contrib/CMakeLists.txt documents them in # requires: comments (the list below reflects it as of 2026-09; check the file, it is the source of truth):

• arrow requires: snappy, thrift, double-conversion, xsimd (and regenerates its flatbuffers bindings)
• avro requires: snappy
• AMQP-CPP requires: libuv
• cassandra requires: libuv
• librdkafka requires: libgsasl
• libhdfs3 requires: google-protobuf, krb5, isa-l
• hive-metastore requires: thrift, avro, arrow, libhdfs3
• rocksdb requires: jemalloc, snappy, zlib, lz4, zstd, liburing
• mongo-c-driver requires: zlib; mongo-cxx-driver requires mongo-c-driver (libmongoc, libbson) at a minimum version stated in its CMakeLists.txt — bump both in one PR when the floor moves
• usearch requires: FP16, SimSIMD
• substrait requires: google-protobuf
• google-protobuf and grpc are coupled (gRPC bundles its own upb; both copies must not end up in one binary)
• AWS SDK: aws, aws-c-auth, aws-c-cal, aws-c-common, aws-c-compression, aws-c-event-stream, aws-c-http, aws-c-io, aws-c-mqtt, aws-c-s3, aws-c-sdkutils, aws-checksums, aws-crt-cpp

If the target library appears in a chain, check whether its dependencies also need updating.

CI Validation

After pushing, CI will automatically:

• Apply the submodule changed label
• Run check_submodules.sh: every .gitmodules entry has a directory, URLs start with https://github.com/, submodule name equals its path, no recursive submodules (no [submodule entries inside submodules)
• Run check_cpp.sh: contrib/*-cmake/ must not use the forbidden CMake patterns listed in step 8
• Build on multiple platforms (x86, ARM, macOS, FreeBSD, loongarch64, s390x, riscv64) and sanitizer configurations (ASan, MSan, TSan, UBSan), plus clang-tidy with dummy launchers

Notes

• Do not use rebase or amend — add new commits
• Do not commit to the master branch — always use a feature branch
• Use Allman-style braces in any C++ code changes
• The .gitmodules file must not use branch = ... tags
• Deleted upstream CMake files in git -C contrib/<lib> status are expected (see Background)