Skip to content

dedup _get_model_external_data_paths#2217

Merged
IlyasMoutawwakil merged 4 commits into
huggingface:mainfrom
xieofxie:xieofxie/dedup
Apr 15, 2025
Merged

dedup _get_model_external_data_paths#2217
IlyasMoutawwakil merged 4 commits into
huggingface:mainfrom
xieofxie:xieofxie/dedup

Conversation

@xieofxie
Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #2216

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@IlyasMoutawwakil
Copy link
Copy Markdown
Member

can you provide a concrete case where this happens ?

@xieofxie
Copy link
Copy Markdown
Contributor Author

xieofxie commented Apr 7, 2025

when using Olive, I could get models like this
model.zip

sample code

import onnx
from onnx.external_data_helper import ExternalDataInfo, _get_initializer_tensors
from pathlib import Path
from collections import Counter

model_path = Path("./model.onnx")

# _get_model_external_data_paths 
onnx_model = onnx.load(str(model_path), load_external_data=False)
model_tensors = _get_initializer_tensors(onnx_model)
# filter out tensors that are not external data
model_tensors_ext = [
    ExternalDataInfo(tensor).location
    for tensor in model_tensors
    if tensor.HasField("data_location") and tensor.data_location == onnx.TensorProto.EXTERNAL
]
data = [model_path.parent / tensor_name for tensor_name in model_tensors_ext]

data_count = Counter(data)
print(data_count)

Got Counter({WindowsPath('./weights.pb'): 685})

@IlyasMoutawwakil
Copy link
Copy Markdown
Member

okay I see

Comment thread optimum/onnx/utils.py Outdated
Comment thread optimum/onnx/utils.py Outdated
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@IlyasMoutawwakil
Copy link
Copy Markdown
Member

@xieofxie please run pip install .[quality] and then make style

@xieofxie
Copy link
Copy Markdown
Contributor Author

Internal Error 。。

@IlyasMoutawwakil
Copy link
Copy Markdown
Member

@xieofxie yeah those are okay, merging !

@IlyasMoutawwakil IlyasMoutawwakil merged commit 3239964 into huggingface:main Apr 15, 2025
24 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dedup _get_model_external_data_paths

3 participants