Skip to content

dev integration: add pyiceberg[pyarrow] to Dockerfile to resolve missing pyarrow import error #2174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 8, 2025

Conversation

kevinjqliu
Copy link
Contributor

Rationale for this change

Adding pyiceberg["pyarrow"] dep to the Dockefile resolves these warning log lines:

Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pyiceberg/io/__init__.py", line 317, in _import_file_io
    module = importlib.import_module(module_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/pyiceberg/io/pyarrow.py", line 62, in <module>
    import pyarrow as pa
ModuleNotFoundError: No module named 'pyarrow'

I also took the liberty to upgrade the Dockerfile python version from 3.9 to 3.12

Are these changes tested?

Are there any user-facing changes?

@kevinjqliu kevinjqliu requested a review from Fokko July 5, 2025 18:14
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

FROM python:3.9-bullseye
FROM python:3.12-bullseye
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feeling adventurous? :)

@Fokko Fokko merged commit fa71498 into apache:main Jul 8, 2025
10 checks passed
@Fokko
Copy link
Contributor

Fokko commented Jul 8, 2025

Thanks @kevinjqliu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants