[IGNORE] Squashed commit of cudf exchange presto setup

[karthikeyann]

Description

The changes in this PR allow cudf-exchange to work with PR facebookincubator/velox#16214

Motivation and Context

These changes are present in ibm-research-preview branch and it's hard to update the branch, This PR will serve as placeholder and easy to update presto with a single button "Update branch" in the PR whenever Velox is advanced in Presto.

Impact

Easy to replicate working setup for cudf-exchange.

Authors:
Squashed commit of cudf exchange presto setup
Co-authored-by: Zoltan Arnold Nagy
Co-authored-by: Luis Garcés-Erice
Co-authored-by: Daniel Bauer

Summary by Sourcery

Integrate cuDF exchange support into Presto native execution, including communicator lifecycle, configuration, and container/runtime dependencies.

New Features:

• Start and manage a cuDF exchange communicator server alongside Presto when cuDF exchange is enabled.
• Add system configuration for a dedicated cuDF exchange server port.
• Enable optional Nsight Systems profiling for the Presto server via container entrypoint configuration.

Enhancements:

• Extend native connectors build to link against UCX (ucxx) for cuDF-based execution and exchange.
• Update CentOS dependency and runtime Docker images to install CUDA, UCX, and related libraries and expose them at runtime.
• Adjust query plan construction for table write nodes to propagate an additional flag required by the updated Velox integration.

Build:

• Change CentOS dependency setup to parameterize CUDA and UCX versions and install UCX as part of the build image.

Deployment:

• Update prestissimo runtime image to bundle CUDA and UCX shared libraries, configure dynamic linker paths, and add RDMA/NVIDIA tooling for GPU execution and profiling.

[sourcery-ai]

Reviewer's Guide

Enables cuDF exchange support and profiling in Presto native execution by wiring up a cudf-exchange communicator server lifecycle, extending system configuration with a cuDF exchange port, updating docker images and entrypoint for CUDA/UCX libraries and Nsight profiling, and linking UCX into the cuDF connector path.

Sequence diagram for cuDF exchange Communicator server lifecycle

sequenceDiagram
    participant PrestoServer
    participant SystemConfig
    participant CudfConfig
    participant CudfRegistry as CudfRegistration
    participant Communicator
    participant ServerThread

    PrestoServer->>PrestoServer: run()
    PrestoServer->>PrestoServer: registerVeloxCudf()

    PrestoServer->>SystemConfig: instance()
    SystemConfig-->>PrestoServer: systemConfig
    PrestoServer->>SystemConfig: values()
    SystemConfig-->>PrestoServer: configValues
    PrestoServer->>CudfConfig: initialize(configValues)

    CudfConfig-->>PrestoServer: enabled, exchange flags
    alt cuDF enabled
        PrestoServer->>CudfRegistry: registerCudf()
        CudfRegistry-->>PrestoServer: registration complete

        alt cuDF exchange enabled
            PrestoServer->>SystemConfig: cudfServerPort()
            SystemConfig-->>PrestoServer: port
            PrestoServer->>SystemConfig: discoveryUri()
            SystemConfig-->>PrestoServer: discoveryUri
            PrestoServer->>Communicator: initAndGet(port, discoveryUri)
            Communicator-->>PrestoServer: communicatorInstance
            PrestoServer->>ServerThread: create thread(run, communicatorInstance)
            ServerThread-->>PrestoServer: communicatorThread handle
        else cuDF exchange disabled
            PrestoServer-->>PrestoServer: no Communicator thread
        end
    else cuDF disabled
        PrestoServer-->>PrestoServer: cuDF and exchange not used
    end

    PrestoServer->>PrestoServer: registerConnectors(), run queries

    PrestoServer-->>PrestoServer: shutdown sequence
    PrestoServer->>PrestoServer: unregisterVeloxCudf(communicatorThread)
    PrestoServer->>SystemConfig: values()
    SystemConfig-->>PrestoServer: configValues
    PrestoServer->>CudfConfig: check enabled
    alt cuDF enabled
        PrestoServer->>CudfRegistry: unregisterCudf()
        CudfRegistry-->>PrestoServer: unregistration complete
        alt communicatorThread exists
            PrestoServer->>Communicator: getInstance()
            Communicator-->>PrestoServer: communicatorInstance
            PrestoServer->>Communicator: stop()
            Communicator-->>PrestoServer: stopped
            PrestoServer->>ServerThread: join()
            ServerThread-->>PrestoServer: thread joined
        else no communicatorThread
            PrestoServer-->>PrestoServer: nothing to stop
        end
    else cuDF disabled
        PrestoServer-->>PrestoServer: no cuDF teardown
    end

Loading

Updated class diagram for SystemConfig and cuDF exchange configuration

classDiagram
    class ConfigBase {
    }

    class SystemConfig {
        +static constexpr string_view kCudfServerPort
        +int httpServerHttpPort()
        +int cudfServerPort()
        +bool httpServerReusePort()
        +static SystemConfig instance()
    }

    class CudfConfig {
        +static CudfConfig getInstance()
        +bool enabled
        +bool exchange
    }

    class Communicator {
        +static shared_ptr~Communicator~ initAndGet(int port, string discoveryUri)
        +static shared_ptr~Communicator~ getInstance()
        +void run()
        +void stop()
    }

    class PrestoServer {
        +void run()
    }

    ConfigBase <|-- SystemConfig
    PrestoServer ..> SystemConfig : uses
    PrestoServer ..> CudfConfig : configures cuDF
    PrestoServer ..> Communicator : controls lifecycle
    SystemConfig ..> Communicator : provides cudfServerPort
    CudfConfig ..> Communicator : enables exchange

Loading

File-Level Changes

Change	Details	Files
Manage lifecycle of cuDF exchange Communicator server within PrestoServer run sequence.	Include cudf-exchange Communicator header when cuDF is enabled Change cuDF registration helper to return an optional server thread handle Initialize cudf-exchange Communicator based on system config, start its run loop in a background thread, and log startup Update cuDF unregistration helper to accept the server thread, stop and reset the Communicator instance, log, and join the thread Wire the returned server thread through PrestoServer::run to ensure clean shutdown	presto-native-execution/presto_cpp/main/PrestoServer.cpp
Extend system configuration to support a dedicated cuDF exchange server port.	Add cudf.exchange.server.port config key definition to SystemConfig Initialize the new property in SystemConfig constructor Expose a cudfServerPort() accessor that reads the required property	presto-native-execution/presto_cpp/main/common/Configs.cpp presto-native-execution/presto_cpp/main/common/Configs.h
Adjust connector build to link UCX for cuDF connector/exchange support.	Update presto_connectors target link line under PRESTO_ENABLE_CUDF to also link against ucxx::ucxx	presto-native-execution/presto_cpp/main/connectors/CMakeLists.txt
Update docker runtime image to include CUDA/UCX libraries and tools required by cudf-exchange and profiling.	Expand LD_LIBRARY_PATH used when scanning presto_server dependencies to include CUDA compat path Copy CUDA target libs and nvidia PTX JIT compiler libs into runtime image under a cuda subdirectory Copy UCX libraries into a dedicated ucx runtime directory Expose UCX and CUDA runtime directories to the final image and configure ldconfig to pick them up Install RDMA-related libraries (librdmacm, libibverbs) and NVIDIA Nsight Systems CLI from NVIDIA repo	presto-native-execution/scripts/dockerfiles/prestissimo-runtime.dockerfile
Enable optional Nsight Systems profiling of presto_server via entrypoint configuration.	Add PROFILE and PROFILE_ARGS handling in entrypoint to optionally construct an nsys launch command Provide default profiling arguments targeting NVTX, CUDA, and OS runtime metrics when PROFILE_ARGS is not set Prefix presto_server invocation with the computed profiling command when profiling is enabled	presto-native-execution/entrypoint.sh
Ensure CentOS dependency image installs matching CUDA and UCX versions for cudf-exchange.	Introduce CUDA_VERSION and UCX_VERSION environment variables Switch CUDA installation to use install_cuda wrapper without hardcoded version Invoke install_ucx during dependency setup	presto-native-execution/scripts/dockerfiles/centos-dependency.dockerfile
Adjust Presto-to-Velox query plan translation for table writer to enable a new behavior flag.	Extend table writer plan node translation to pass an extra boolean argument (true) into the Velox plan node constructor	presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.