Materialize tables in the experimental Parquet reader #19308

mhaseeb123 · 2025-07-08T03:41:02Z

Description

Contributes to #17896. Completes #18011

This PR implements table materialization functions in the experimental parquet reader. The experimental reader now derives from the base parquet reader and only overloads the necessary functions reusing the base functions wherever possible.

Most of the functions reimplemented by the experimental reader are also mostly identical with the differences from the base reader mentioned in the comments

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

https://github.com/mhaseeb123/cudf into fea/materialize-hybrid-scan-columns

copy-pr-bot · 2025-07-08T03:41:06Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

…:detail::reader` (#19305) Contributes to #19308 This PR moves out `parquet::reader::impl` class out of `parquet::reader` to allow subclassing (the experimental reader) from it and reusing its functions without having to duplicate code Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Paul Mattione (https://github.com/pmattione-nvidia) - Tianyu Liu (https://github.com/kingcrimsontianyu) - Nghia Truong (https://github.com/ttnghia) URL: #19305

mhaseeb123 · 2025-07-11T20:47:15Z

cpp/src/io/parquet/experimental/hybrid_scan_helpers.cpp

@@ -262,44 +262,6 @@ aggregate_reader_metadata::select_payload_columns(
    valid_payload_columns, {}, include_index, strings_to_categorical, timestamp_type_id);
 }

-std::tuple<cudf::size_type, std::vector<row_group_info>>


No longer needed. Using the base class function instead

mhaseeb123 · 2025-07-11T20:49:09Z

cpp/src/io/parquet/experimental/hybrid_scan_chunking.cu

+  set_page_mask(data_page_mask);
+
+  // setup the next sub pass
+  setup_next_subpass(read_mode::READ_ALL);


The only change from the base reader's impl:

Need to set_page_mask at L86

Use hardcoded READ_ALL mode until we enable chunking

mhaseeb123 · 2025-07-11T20:51:20Z

cpp/src/io/parquet/experimental/hybrid_scan_chunking.cu

+    pass.num_rows  = _file_itm_data.global_num_rows;
+
+    // Setup page information for the chunk (which we can access without decompressing)
+    setup_compressed_data(std::move(column_chunk_buffers));


The only change from the base reader's impl:
Here we are just setting up internal data pointers to the externally passed in column chunk data buffers instead of using read_compressed_data in the base reader's impl

mhaseeb123 · 2025-07-11T20:51:44Z

cpp/src/io/parquet/experimental/hybrid_scan_preprocess.cu

@@ -41,21 +44,6 @@ using parquet::detail::chunk_page_info;
 using parquet::detail::ColumnChunkDesc;
 using parquet::detail::PageInfo;

-/**


No longer needed

mhaseeb123 · 2025-07-11T20:53:06Z

cpp/src/io/parquet/experimental/hybrid_scan_preprocess.cu

@@ -135,7 +214,8 @@ hybrid_scan_reader_impl::prepare_dictionaries(
  rmm::cuda_stream_view stream)
 {
  // Create row group information for the input row group indices
-  auto const row_groups_info = std::get<1>(_metadata->select_row_groups(row_group_indices));
+  auto const row_groups_info = std::get<2>(


Call the base class's function instead

mhaseeb123 · 2025-07-11T21:16:46Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

-                                        use_pandas_metadata,
-                                        strings_to_categorical,
-                                        timestamp_type_id);
+      _extended_metadata->select_payload_columns(options.get_columns(),


Use _extended_metadata, the ptr to the child version of aggregate_reader_metadata which implements these extended functions

mhaseeb123 · 2025-07-11T21:20:20Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

+      cudf::detail::apply_boolean_mask(read_table->view(), *final_row_mask, _stream, _mr);
+
+    // Update the input row mask to reflect the final row mask.
+    update_row_mask(final_row_mask->view(), row_mask, _stream);


Difference: Update the (input) mutable row_mask to delete rows that are deleted by final_row_mask

mhaseeb123 · 2025-07-11T21:20:53Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

+  else {
+    CUDF_EXPECTS(read_columns_mode == read_columns_mode::PAYLOAD_COLUMNS, "Invalid read mode");
+    CUDF_EXPECTS(row_mask.type().id() == type_id::BOOL8,
+                 "Predicate filter should return a boolean");
+    CUDF_EXPECTS(row_mask.size() == read_table->num_rows(),
+                 "Encountered invalid sized row mask to apply");
+    auto output_table =
+      cudf::detail::apply_boolean_mask(read_table->view(), row_mask, _stream, _mr);
+    return {std::move(output_table), std::move(out_metadata)};
+  }


Difference: In case of payload columns, only apply the input row_mask to the read table

mhaseeb123 · 2025-07-11T21:21:41Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

+  // If reading filter columns, compute the predicate, apply it to the table, and update the input
+  // row mask to reflect the final surviving rows.
+  if constexpr (std::is_same_v<RowMaskView, cudf::mutable_column_view>) {


Difference: Comment at L1089 😄 vs we always apply the _expr_conv filter if available to the read_table in the mainline reader

mhaseeb123 · 2025-07-11T21:22:01Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

+template <typename RowMaskView>
+table_with_metadata hybrid_scan_reader_impl::finalize_output(
+  read_columns_mode read_columns_mode,
+  table_metadata& out_metadata,
+  std::vector<std::unique_ptr<column>>& out_columns,
+  RowMaskView row_mask)


Difference in function signature. Accept a tparam and a row_mask

mhaseeb123 · 2025-07-11T21:26:19Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.hpp

@@ -304,25 +327,6 @@ class hybrid_scan_reader_impl {
  void setup_next_pass(std::vector<rmm::device_buffer> column_chunk_buffers,
                       parquet_reader_options const& options);

-  /**


All these member functions now no longer needed as we can reuse their base versions

mhaseeb123 · 2025-07-14T21:48:09Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

@@ -370,7 +427,30 @@ table_with_metadata hybrid_scan_reader_impl::materialize_filter_columns(
  parquet_reader_options const& options,
  rmm::cuda_stream_view stream)
 {
-  return {};
+  CUDF_EXPECTS(not row_group_indices.empty(), "Empty input row group indices encountered");


This is a new function. Please review 🙂

mhaseeb123 · 2025-07-14T21:48:19Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

@@ -380,35 +460,826 @@ table_with_metadata hybrid_scan_reader_impl::materialize_payload_columns(
  parquet_reader_options const& options,
  rmm::cuda_stream_view stream)
 {
-  return {};
+  CUDF_EXPECTS(not row_group_indices.empty(), "Empty input row group indices encountered");


This is a new function. Please review 🙂

mhaseeb123 · 2025-07-14T21:48:24Z

cpp/src/io/parquet/experimental/hybrid_scan_impl.cpp

@@ -343,23 +349,74 @@ std::pair<std::vector<byte_range_info>, std::vector<cudf::size_type>>
 hybrid_scan_reader_impl::get_input_column_chunk_byte_ranges(
  cudf::host_span<std::vector<size_type> const> row_group_indices) const
 {
-  return {};
+  // Descriptors for all the chunks that make up the selected columns


This is a new function. Please review 🙂

…put-nullmasks-for-pruned-pages' into fea/materialize-hybrid-scan-columns

…asing in bulk null mask set (#19349) Related to #18489. Contributes to #19308 This PR implements a new `cudf::set_null_masks_safe` API which safely handles intra-word aliasing when setting null masks with a small performance hit compared to `cudf::set_null_masks_unsafe`. Additionally, the `cudf::set_null_masks` API has been deprecated in favor of `cudf::set_null_masks_unsafe` Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Yunsong Wang (https://github.com/PointKernel) - David Wendt (https://github.com/davidwendt) URL: #19349

Contributes to #19308 This PR refactors gtests related to the hybrid scan reader (experimental parquet reader) to a separate executable. No new code is added by this PR except trivial style and CMake changes. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Devavret Makkar (https://github.com/devavret) - Bradley Dice (https://github.com/bdice) URL: #19359

…19359) Contributes to rapidsai#19308 This PR refactors gtests related to the hybrid scan reader (experimental parquet reader) to a separate executable. No new code is added by this PR except trivial style and CMake changes. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Devavret Makkar (https://github.com/devavret) - Bradley Dice (https://github.com/bdice) URL: rapidsai#19359

Contributes to PR #19308 This PR adds a parquet reader utility to update the null masks of output buffers in order to nullify rows corresponding to the pruned out pages. No independent tests for this PR here. Instead this is tested in #19308 Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Bradley Dice (https://github.com/bdice) - Vukasin Milovanovic (https://github.com/vuule) URL: #19370

mhaseeb123 and others added 13 commits July 7, 2025 22:36

Initial materialize columns

b4bcaf0

Move out parquet reader impl class out of the reader

d2daede

Add default constructor for subclassing

f344294

style check plz

89abc6c

Rename read_mode to read_column_mode

beeb60d

Merge branch 'fea/refactor-parquet-chunking-preprocessing-functions' of

edc5bbb

https://github.com/mhaseeb123/cudf into fea/materialize-hybrid-scan-columns

Add _page_mask member to parquet reader impl

6c0ebc3

Merge branch 'fea/refactor-parquet-chunking-preprocessing-functions' of

0364968

https://github.com/mhaseeb123/cudf into fea/materialize-hybrid-scan-columns

Revert _metadata

865aacd

Merge the readers

7174cf2

Make _options protected again

a0427ea

Merge branch 'fea/refactor-parquet-chunking-preprocessing-functions' of

583d974

https://github.com/mhaseeb123/cudf into fea/materialize-hybrid-scan-columns

Compiles but doesn't work yet

5385879

mhaseeb123 self-assigned this Jul 8, 2025

mhaseeb123 added the DO NOT MERGE Hold off on merging; see PR for details label Jul 8, 2025

github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Jul 8, 2025

mhaseeb123 and others added 2 commits July 8, 2025 04:35

Push debug changes

bb3b18d

Still need debugging

b93554c

mhaseeb123 changed the title ~~Fea/materialize hybrid scan columns~~ 🚧 Materialize tables in the experimental Parquet reader Jul 8, 2025

mhaseeb123 added 2 - In Progress Currently a work in progress cuIO cuIO issue labels Jul 8, 2025

mhaseeb123 added 2 commits July 9, 2025 18:29

Works now

79a3dbf

Working solution

f0fd718

mhaseeb123 added feature request New feature or request non-breaking Non-breaking change and removed DO NOT MERGE Hold off on merging; see PR for details labels Jul 10, 2025

mhaseeb123 mentioned this pull request Jul 10, 2025

Move the Parquet reader_impl class declaration out of the parquet::detail::reader #19305

Merged

3 tasks

Merge branch 'branch-25.08' into fea/materialize-hybrid-scan-columns

7b4659c

Merge branch 'branch-25.08' into fea/materialize-hybrid-scan-columns

9b40396

mhaseeb123 mentioned this pull request Jul 11, 2025

Refactor hybrid scan reader tests to a separate executable #19359

Merged

3 tasks

mhaseeb123 added 2 commits July 11, 2025 20:46

Sync with rapidsai#19359

661bc81

Minor improvement

b28befe

mhaseeb123 commented Jul 11, 2025

View reviewed changes

mhaseeb123 requested review from vuule and nvdbaranec July 11, 2025 21:27

mhaseeb123 mentioned this pull request Jul 11, 2025

🚧 Implement an experimental Parquet reader optimized for highly-selective hybrid scan reads #18011

Closed

3 tasks

mhaseeb123 and others added 2 commits July 14, 2025 21:46

Use cudf::fill_in_place instead of a separate wrapper

18335da

Parquet reader utility to update output null masks for pruned pages

897c55f

mhaseeb123 mentioned this pull request Jul 14, 2025

Add a parquet reader utility to update output null masks #19370

Merged

3 tasks

mhaseeb123 added 2 commits July 14, 2025 22:32

Revert cursor's overefficiencies to std::identity

5affd1f

Minor improvements

c00042a

mhaseeb123 commented Jul 14, 2025

View reviewed changes

mhaseeb123 added 2 commits July 15, 2025 21:43

Merge remote-tracking branch 'origin/fea/pq-reader-util-to-update-out…

15bddeb

…put-nullmasks-for-pruned-pages' into fea/materialize-hybrid-scan-columns

Sync with PR rapidsai#19370

ffc813c

Merge branch 'branch-25.08' into fea/materialize-hybrid-scan-columns

10940ef

mhaseeb123 added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Jul 17, 2025

mhaseeb123 changed the title ~~🚧 Materialize tables in the experimental Parquet reader~~ Materialize tables in the experimental Parquet reader Jul 17, 2025

mhaseeb123 marked this pull request as ready for review July 17, 2025 20:23

mhaseeb123 requested review from a team as code owners July 17, 2025 20:23

mhaseeb123 requested a review from ttnghia July 17, 2025 20:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Materialize tables in the experimental Parquet reader #19308

Materialize tables in the experimental Parquet reader #19308

Uh oh!

mhaseeb123 commented Jul 8, 2025 •

edited

Loading

Uh oh!

copy-pr-bot bot commented Jul 8, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025 •

edited

Loading

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025 •

edited

Loading

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 11, 2025

Uh oh!

mhaseeb123 Jul 14, 2025

Uh oh!

mhaseeb123 Jul 14, 2025

Uh oh!

mhaseeb123 Jul 14, 2025

Uh oh!

Uh oh!

Materialize tables in the experimental Parquet reader #19308

Are you sure you want to change the base?

Materialize tables in the experimental Parquet reader #19308

Uh oh!

Conversation

mhaseeb123 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot bot commented Jul 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhaseeb123 Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhaseeb123 Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mhaseeb123 commented Jul 8, 2025 •

edited

Loading

mhaseeb123 Jul 11, 2025 •

edited

Loading

mhaseeb123 Jul 11, 2025 •

edited

Loading