[BUG]: Serializing a cuDF DataFrame with struct columns results in loosing keys

[dagardner-nv]

Version

25.06

Which installation method(s) does this occur on?

Source

Describe the bug.

This appears to only happen with our C++ code

Minimum reproducible example

repro:

from morpheus.io.deserializers import read_file_to_df
from morpheus.io.serializers import write_df_to_file

df = read_file_to_df('.tmp/output/data_loss_prevention_out.jsonlines', df_type='cudf')
write_df_to_file(df, '.tmp/output/st.jsonlines')

Relevant log output

Click here to see error details

[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

Bug occurs if cudf.io.read_json is used to read, so the bug is in our serialization code not the deserialization code.

[dagardner-nv]

Issue is our code which builds cudf table metadata based upon a Morpheus table info. When we read the DF we end up with metadata that looks like:

Column: source_text, Nullable: 0, Binary: 0, Length: -1 Num children: 1
	Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: dlp_findings, Nullable: 0, Binary: 0, Length: -1 Num children: 2
	Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
	Column: element, Nullable: 0, Binary: 0, Length: -1 Num children: 5
		Column: end, Nullable: 0, Binary: 0, Length: -1 Num children: 0
		Column: label, Nullable: 0, Binary: 0, Length: -1 Num children: 1
			Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
		Column: score, Nullable: 0, Binary: 0, Length: -1 Num children: 0
		Column: start, Nullable: 0, Binary: 0, Length: -1 Num children: 0
		Column: text, Nullable: 0, Binary: 0, Length: -1 Num children: 1
			Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: risk_level, Nullable: 0, Binary: 0, Length: -1 Num children: 1
	Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: risk_score, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: highest_confidence, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_low, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_medium, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_high, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: data_types_found, Nullable: 0, Binary: 0, Length: -1 Num children: 2
	Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0
	Column: element, Nullable: 0, Binary: 0, Length: -1 Num children: 1
		Column: offsets, Nullable: 0, Binary: 0, Length: -1 Num children: 0

However we lose the children columns on the way out:

Column: source_text, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: dlp_findings, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: risk_level, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: risk_score, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: highest_confidence, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_low, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_medium, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: num_high, Nullable: 0, Binary: 0, Length: -1 Num children: 0
Column: data_types_found, Nullable: 0, Binary: 0, Length: -1 Num children: 0

[dagardner-nv]

Related cudf issue: rapidsai/cudf#19215