feat: re-export name mapping #1116

jdockerty · 2025-03-19T13:32:25Z

Which issue does this PR close?

Likely helps towards #919 and this was also discussed in Slack.

What changes are included in this PR?

This publicly re-exports the name_mapping module to iceberg::spec. Prior to this, it is private and inaccessible outside of this crate.

Are these changes tested?

The main changes here are not functional changes except to visibility.

The additional fields to the NameMapping structure have test coverage and are covered by prior tests, it is expected that these internal fields do not show up in the serialised NameMapping output, hence they are marked with the serde(skip) attribute - this is upheld with prior tests.

liurenjie1024

Thanks @jdockerty for this pr, generally LGTM! Left some comments to improve.

crates/iceberg/src/spec/name_mapping.rs

liurenjie1024

Thanks @jdockerty for this pr, generally LGTM! Left some comments to improve.

liurenjie1024 · 2025-03-21T02:50:01Z

crates/iceberg/src/spec/mapped_fields.rs

+/// Utility mapping which contains field names to IDs and
+/// field IDs to the underlying [`MappedField`].
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+pub struct MappedFields {


Given we are going to add a lot of visitors for NameMapping, how about we create a NameMapping module, and puts everything related there.

I'm not quite sure what you mean, there is already a name_mapping.rs as a separate module. Or do you mean include everything in this file instead and use 👇 ?

mod name_mapping { // contents of name_mapping.rs here }

Yes mapped field should also be included in name_mapping.rs. You can move everything from mapped_fields to name_mapping

I still don't understand why we need to have the MappedFields field. In the pyiceberg implementation it doesnt use it (the initial implementation/review is here apache/iceberg-python#212). There isn't a usecase (unless there is) where we just index the first layer for MappedFields without having to later create another index based on a full traversal. cc @liurenjie1024 @Fokko

TBH I haven't come up with a case where Vec<MappedField> doesn't work well. I'm trying to keep things as consistent as possible with java implementation, as it's currently the refrence implementation.

I'm not quite sure what you mean, there is already a name_mapping.rs as a separate module. Or do you mean include everything in this file instead and use 👇 ?

Oh sorry for misclarification. I mean we could create following file structures:

name_mapping/ mod.rs visitors.rs

In rust, name_maping.rs is a module called name_mapping, and a dir name_maping + mod.rs is also a module called name_maping.

In PyIceberg we've have a visitor to traverse the name-mapping and schema at the same time: https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/name_mapping.py#L251-L372

This is because when looking up by column name, the . could make it ambiguous to determine if it is a nested field, or if the field has a dot in it:

table { foo: struct<bar: str> foo.bar: str }

crates/iceberg/src/spec/mapped_fields.rs

liurenjie1024 · 2025-03-21T02:52:54Z

crates/iceberg/src/spec/name_mapping.rs

 /// Iceberg fallback field name to ID mapping.
 #[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
 #[serde(transparent)]
 pub struct NameMapping {
-    pub root: Vec<MappedField>,
+    root: Vec<MappedField>,


This should contains a MappedFields

liurenjie1024 · 2025-03-21T02:53:14Z

crates/iceberg/src/spec/name_mapping.rs

    #[serde(default)]
    #[serde(skip_serializing_if = "Vec::is_empty")]
    #[serde_as(deserialize_as = "DefaultOnNull")]
-    pub fields: Vec<MappedField>,
+    fields: Vec<MappedField>,


This should be a MappedFields.

By changing this one to MappedFields, this alters all of the expected output JSON too.

I assume that is expected for now and I'll update the other tests 👍

We can't change json output since it's iceberg spec. I think we could use #serde[transparent] to do that?

@liurenjie1024 I've been taking another look at this again, although including #[serde(transparent)] onto the structure doesn't seem to produce the right value as it isn't a struct that contains only an internal Vec, since MappedFields contains other fields/state too.

Do you mind elaborating more on this? As I don't quite understand how to stop the field being serialised incorrectly when this is altered.

Seems #[serde(transparent)] doesn't work here. I think then maybe we should remove MappedFields struct and put everything in NamedMapping, just like what we do in pyiceberg. This maybe be harmful as this interface is not critical. Another approach is to have a standalone module for ser/de, just like what we did in other core data structures like TableMetadata But this may not deserve such complexity.

To clarify here, do you mean pull the information that used to be in the MappedFields struct, into the NameMapping instead?

E.g. the id_to_field etc?

It looks like that exists in pyiceberg via the _IndexByName class

To clarify here, do you mean pull the information that used to be in the MappedFields struct, into the NameMapping instead?

Exactly.

jonathanc-n · 2025-03-21T03:03:42Z

@jdockerty @liurenjie1024 I believe #1072 contains a lot of the functionality in this pr, this got split into #1082 being the first part of it.

liurenjie1024 · 2025-03-21T03:14:41Z

@jdockerty @liurenjie1024 I believe #1072 contains a lot of the functionality in this pr, this got split into #1082 being the first part of it.

Hi, @jonathanc-n I think #1072 added extra functionality into like visitor, indexing into NameMapping. I think this pr should happen before #1072 since it's a refactoring before exposing it to public.

crates/iceberg/src/spec/mapped_fields.rs

liurenjie1024 · 2025-03-25T02:03:57Z

crates/iceberg/src/spec/name_mapping.rs

    #[serde(default)]
    #[serde(skip_serializing_if = "Vec::is_empty")]
    #[serde_as(deserialize_as = "DefaultOnNull")]
-    pub fields: Vec<MappedField>,
+    fields: Vec<MappedField>,


We can't change json output since it's iceberg spec. I think we could use #serde[transparent] to do that?

liurenjie1024 · 2025-03-25T02:06:14Z

crates/iceberg/src/spec/mapped_fields.rs

+/// Utility mapping which contains field names to IDs and
+/// field IDs to the underlying [`MappedField`].
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+pub struct MappedFields {


I'm not quite sure what you mean, there is already a name_mapping.rs as a separate module. Or do you mean include everything in this file instead and use 👇 ?

Oh sorry for misclarification. I mean we could create following file structures:

name_mapping/ mod.rs visitors.rs

In rust, name_maping.rs is a module called name_mapping, and a dir name_maping + mod.rs is also a module called name_maping.

ZENOTME · 2025-04-10T19:03:25Z

crates/iceberg/src/spec/mapped_fields.rs

Should we put this in the name_mapping dir?🤔

Yes, I believe this was mentioned above as well

Seems this should be removed, see #1116 (comment)

liurenjie1024 · 2025-04-11T08:42:23Z

cc @jdockerty Are you still working on this? This is a blocker of 0.5.0 release

jdockerty · 2025-04-11T09:06:47Z

@liurenjie1024 I am! Sorry I should've mentioned, I've been finalising a work project but I can come back to this likely Tuesday morning.

Does that work or is there extra priority required as a release blocker?

liurenjie1024 · 2025-04-11T13:59:33Z

@liurenjie1024 I am! Sorry I should've mentioned, I've been finalising a work project but I can come back to this likely Tuesday morning.

Does that work or is there extra priority required as a release blocker?

Hi, @jdockerty Thanks for contributing. No worry, we can wait for a while.

jdockerty · 2025-04-15T10:13:02Z

@liurenjie1024 Can you take another look at this please?

I've removed the mapped_fields.rs in favour of introducing internal fields to NameMapping, similar to what was mentioned here and here 🙇

liurenjie1024

Thanks @jdockerty for this pr, left some suggestions to improve.

liurenjie1024 · 2025-04-16T12:17:36Z

crates/iceberg/src/spec/name_mapping/mod.rs

+    #[serde(skip)]
+    name_to_id: HashMap<String, i32>,
+    #[serde(skip)]
+    id_to_field: HashMap<i32, MappedField>,


Suggested change

id_to_field: HashMap<i32, MappedField>,

id_to_field: HashMap<i32, Arc<MappedField>>,

MappedField maybe deeply nested and expensive to copy.

crates/iceberg/src/spec/name_mapping/mod.rs

liurenjie1024 · 2025-04-16T12:22:00Z

crates/iceberg/src/spec/name_mapping/mod.rs

+        let mut name_to_id = HashMap::new();
+        let mut id_to_field = HashMap::new();
+
+        for field in &fields {
+            if let Some(id) = field.field_id() {
+                if id_to_field.contains_key(&id) {
+                    return Err(Error::new(
+                        crate::ErrorKind::DataInvalid,
+                        format!("duplicate id '{id}' is not allowed"),
+                    ));
+                }
+
+                id_to_field.insert(id, field.clone());
+                for name in field.names() {
+                    if name_to_id.contains_key(name) {
+                        return Err(Error::new(
+                            crate::ErrorKind::DataInvalid,
+                            format!("duplicate name '{name}' is not allowed"),
+                        ));
+                    }
+                    name_to_id.insert(name.to_string(), id);
+                }
+            }
+        }


This implementation is incorrect, id and name in top level should be global, and we should build it using a visitor. I would suggest to remove these two fields in this pr, and add them back later using visitor.

remove these two fields in this pr

By removing these fields, doesn't that also mean the Arc suggestion above is not to be implemented as well?

id_to_field: HashMap<i32, Arc<MappedField>> // <-- no need to Arc here, this field is to be removed now.

These fields could be deeply nested, so cloning them through an Arc makes this far cheaper

This should be replaced by a visitor pattern in a separate PR

crates/iceberg/src/spec/name_mapping/mod.rs

liurenjie1024

Thanks @jdockerty for this pr, LGTM!

chore: pub re-export name mapping

86db09c

jdockerty marked this pull request as ready for review March 19, 2025 13:34

jdockerty changed the title ~~chore: re-export name mapping~~ feat: re-export name mapping Mar 19, 2025

feat: add property name const

7cc1977

liurenjie1024 reviewed Mar 20, 2025

View reviewed changes

crates/iceberg/src/spec/name_mapping.rs Outdated Show resolved Hide resolved

crates/iceberg/src/spec/name_mapping.rs Outdated Show resolved Hide resolved

jdockerty added 5 commits March 20, 2025 10:06

refactor: add accessor methods

b57742d

feat: add MappedFields

6bdf66e

docs: typo

1c4cec0

docs: add header

abd4e51

docs: mapped field docstring

85b024e

jdockerty force-pushed the chore/expose-name-mapping branch from a9ff2b9 to 85b024e Compare March 20, 2025 12:24

liurenjie1024 reviewed Mar 21, 2025

View reviewed changes

jdockerty added 2 commits March 21, 2025 10:50

fix: remove value ref from id fn

3e766cc

feat: check for duplicates and produce error in MappedFields

dea509b

liurenjie1024 reviewed Mar 25, 2025

View reviewed changes

fix: remove owned string usage in id func

ea31484

liurenjie1024 added this to the 0.5.0 Release milestone Apr 1, 2025

refactor: add name_mapping mod

886ba3f

jonathanc-n mentioned this pull request Apr 9, 2025

Support adding parquet files to partitioned table #1035

Open

ZENOTME reviewed Apr 10, 2025

View reviewed changes

jdockerty added 3 commits April 15, 2025 10:05

refactor: move MappedField fields to NameMapping

d22edce

chore: move internal field tests to name mapping

140a42b

chore: remove mapped_fields module

c18338a

liurenjie1024 reviewed Apr 16, 2025

View reviewed changes

jdockerty added 2 commits April 16, 2025 15:01

fix: arc nested fields within MappedField

e93d846

These fields could be deeply nested, so cloning them through an Arc makes this far cheaper

refactor: remove internal fields from NameMapping

0cfa5f7

This should be replaced by a visitor pattern in a separate PR

liurenjie1024 reviewed Apr 18, 2025

View reviewed changes

crates/iceberg/src/spec/name_mapping/mod.rs Outdated Show resolved Hide resolved

Update crates/iceberg/src/spec/name_mapping/mod.rs

3bda87a

liurenjie1024 approved these changes Apr 18, 2025

View reviewed changes

liurenjie1024 merged commit 609b792 into apache:main Apr 18, 2025
17 checks passed

jonathanc-n mentioned this pull request May 7, 2025

feat: Add IndexByName and IndexById to Namemapping #1299

Open

	id_to_field: HashMap<i32, MappedField>,
	id_to_field: HashMap<i32, Arc<MappedField>>,

feat: re-export name mapping #1116

feat: re-export name mapping #1116

Uh oh!

Conversation

jdockerty commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdockerty Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdockerty Apr 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathanc-n commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liurenjie1024 commented Mar 21, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathanc-n Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 commented Apr 11, 2025

Uh oh!

jdockerty commented Apr 11, 2025

Uh oh!

liurenjie1024 commented Apr 11, 2025

Uh oh!

jdockerty commented Mar 19, 2025 •

edited

Loading

jdockerty Mar 21, 2025 •

edited

Loading

jdockerty Apr 13, 2025 •

edited

Loading

jonathanc-n commented Mar 21, 2025 •

edited

Loading

jonathanc-n Apr 10, 2025 •

edited

Loading

jdockerty Apr 16, 2025 •

edited

Loading