Skip to content

open_dataset not closing NetCDF file (Windows) #1629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jennirinker opened this issue Oct 12, 2017 · 4 comments
Closed

open_dataset not closing NetCDF file (Windows) #1629

jennirinker opened this issue Oct 12, 2017 · 4 comments

Comments

@jennirinker
Copy link

MWE

Entering

test_ds = xr.open_dataset(path_to_nc)
os.remove(path_to_nc)

results in

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: (path_to_nc)

If I use the option autoclose=True, I can delete the file without this error, but then when I try to access test_ds I get OSError: No such file or directory. (This second part might be expected behavior depending on how autoclose is supposed to function.)

conda info

Current conda install:

               platform : win-64
          conda version : 4.3.27
       conda is private : False
      conda-env version : 4.3.27
    conda-build version : not installed
         python version : 3.6.1.final.0
       requests version : 2.14.2
       root environment : C:\Users\rink\AppData\Local\Continuum\Miniconda3  (wri
table)
    default environment : C:\Users\rink\AppData\Local\Continuum\Miniconda3
       envs directories : C:\Users\rink\AppData\Local\Continuum\Miniconda3\envs
                          C:\Users\rink\AppData\Local\conda\conda\envs
                          C:\Users\rink\.conda\envs
          package cache : C:\Users\rink\AppData\Local\Continuum\Miniconda3\pkgs
                          C:\Users\rink\AppData\Local\conda\conda\pkgs
           channel URLs : https://repo.continuum.io/pkgs/main/win-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/win-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/win-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/win-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://repo.continuum.io/pkgs/msys2/win-64
                          https://repo.continuum.io/pkgs/msys2/noarch
            config file : None
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.27 requests/2.14.2 CPython/3.6.1 Windows/7 W
indows/6.1.7601
          administrator : True
@fmaussion
Copy link
Member

Thank you for the report! This is expected behavior: users have the responsibility to open and close their files. You can either close it manually:

test_ds.close()

Or (better), use a context manager:

with xr.open_dataset(path_to_nc) as ds:
    # do stuff

@fmaussion
Copy link
Member

I just read the second part of you question: you can't delete the file and then still read data out of it. What is your use case?

@jennirinker
Copy link
Author

Thanks for response.

Basically I've inherited some legacy code that subclasses an xarray dataset, and all of the tests pass on Linux but one fails on window from this issue. I don't think the code ever closes a dataset, nor does it use context managers. We're loading data stored in a netcdf into the subclass instance and then doing a bunch of calculations on it.

Anyway, I agree that the current behavior is actually the ideal scenario, it was just weird at first when the tests passed on Linux and not Windows. I'll have to see if I can patch the code to use context managers, since manually closing the file will probably not always happen.

@shoyer
Copy link
Member

shoyer commented Oct 15, 2017

In my experience, Windows is certainly much pickier than Linux about deleting files that are still open.

LuukBlom added a commit to Deltares/hydromt_sfincs that referenced this issue Jan 8, 2025
roeldegoede added a commit to Deltares/hydromt_sfincs that referenced this issue Jan 10, 2025
* bugfix so that the .nc file is closed automatically when erroring or when leaving the context manager (with block). see pydata/xarray#1629 (comment)

* fixed all occurences of xr.opendataset with the safe open&close pattern

* review comments by Roel

* fix linting

---------

Co-authored-by: roeldegoede <[email protected]>
roeldegoede added a commit to Deltares/hydromt_sfincs that referenced this issue Jan 22, 2025
* potential bugfix havg

* undo previous commit and update computation of wet fractions (and increased readability)

* fix wet fractions and corresponding havg

* add dependencies

* add reading/writing quadtree netcdf files

* add read/write subgrid quadtree

* add quadtree io in main sfincs.py

* Solve some linting warnings

* make datashader optional dep

* add bounds property (since the original calls read_grid) and remove grid api

* fixing read_results and plot_basemap for quadtree models. Also fixed #133

* add quadtree IO tests

* extended tests for quadtree io, plot_basemap and read_results

* imrpoved downscaling methods (both bugfixes and allow for ugrids)

* pre-commit linting

* pre-commit test-data

* changed fix for plotting; not longer reprojecting to epsg4326 by default but using another cartopy projection

* fix pyflwdir version for now (to be investigated)

* fix typo in pyproject.toml

* test fixing docs workflows, since Mambaforge gets deprecated ...

* miniforge3?

* Delete tests/data/sfincs_test_quadtree/sfincs_log.txt

* bugfix so sfincs_his.nc files are closed correctly (#232)

* bugfix so that the .nc file is closed automatically when erroring or when leaving the context manager (with block). see pydata/xarray#1629 (comment)

* fixed all occurences of xr.opendataset with the safe open&close pattern

* review comments by Roel

* fix linting

---------

Co-authored-by: roeldegoede <[email protected]>

* added xu_open_dataset wrapper (#236)

* added xu_open_dataset wrapper.

* load_dataset -> open_dataset

* linting

---------

Co-authored-by: roeldegoede <[email protected]>

---------

Co-authored-by: DirkEilander <[email protected]>
Co-authored-by: Tim Leijnse <[email protected]>
Co-authored-by: LuukBlom <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants