Initial commit: D-MemFS v0.2.0

Author: nightmarewalker

.gitattributes

@@ -0,0 +1 @@
+* text=auto

.github/workflows/test.yml

@@ -0,0 +1,50 @@
+name: Tests
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ["3.11", "3.12", "3.13"]
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install uv
+      uses: astral-sh/setup-uv@v5
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Compile requirements
+      run: uv pip compile requirements.in -o requirements.txt
+    - name: Run tests with coverage
+      run: uvx --with-requirements requirements.txt --with-editable . pytest tests/ -v --timeout=30 --cov=dmemfs --cov-report=xml --cov-report=term-missing
+    - name: Upload coverage artifact
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: coverage-${{ matrix.os }}-py${{ matrix.python-version }}
+        path: coverage.xml
+    - name: Upload coverage to Codecov
+      if: ${{ vars.ENABLE_CODECOV == 'true' }}
+      uses: codecov/codecov-action@v4
+      with:
+        files: coverage.xml
+        fail_ci_if_error: true
+    - name: Generate API docs (pdoc)
+      if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
+      run: uvx --with-requirements requirements.txt pdoc dmemfs -o docs/api
+    - name: Upload API docs artifact
+      if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
+      uses: actions/upload-artifact@v4
+      with:
+        name: api-docs-pdoc
+        path: docs/api

.gitignore

@@ -0,0 +1,58 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.so
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Testing
+.pytest_cache/
+.hypothesis/
+.coverage
+coverage.xml
+htmlcov/
+*.lcov
+
+# Type checking
+.mypy_cache/
+.pytype/
+
+# Linters
+.ruff_cache/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+*.code-workspace
+
+# OS
+.DS_Store
+Thumbs.db
+ConsoleHost_history.txt
+
+# Lock files (library, not app)
+uv.lock
+
+# Generated API docs (HTML)
+docs/api/
+
+# Non-public directories
+plan/
+Article/
+
+# Backup files
+*.bak

BENCHMARK.md

@@ -0,0 +1,52 @@
+# Benchmark Guide
+
+This repository includes a minimal benchmark script to compare:
+
+- `MemoryFileSystem` (MFS)
+- `io.BytesIO` (single stream or dict of streams)
+- `PyFilesystem2` (`fs.memoryfs.MemoryFS`)
+- `tempfile`-backed real filesystem I/O
+
+> **Note on PyFilesystem2:** As of setuptools 82 (February 2026), `pyfilesystem2` fails to import due to its use of the deprecated `pkg_resources.declare_namespace()` API ([issue #597](https://github.com/PyFilesystem/pyfilesystem2/issues/597)). The project appears unmaintained and the fix has not been released. Benchmark results that include `PyFilesystem2` were measured in an environment with setuptools ≤ 81 and remain valid as historical comparison data, but the benchmark script will skip or error on PyFilesystem2 cases in current environments.
+
+## Run
+
+```bash
+uvx --with-requirements requirements.txt --with-editable . python benchmarks/compare_backends.py
+```
+
+## Common options
+
+```bash
+# More stable numbers
+uvx --with-requirements requirements.txt --with-editable . python benchmarks/compare_backends.py --repeat 10 --warmup 2
+
+# Heavier workload
+uvx --with-requirements requirements.txt --with-editable . python benchmarks/compare_backends.py --small-files 1000 --stream-size-mb 64
+
+# JSON output
+uvx --with-requirements requirements.txt --with-editable . python benchmarks/compare_backends.py --json
+
+# Save reports into benchmarks/results/
+uvx --with-requirements requirements.txt --with-editable . python benchmarks/compare_backends.py --save-md auto --save-json auto
+```
+
+When markdown/json reports are generated, the script also updates:
+
+- `benchmarks/results/benchmark_current_result.md`
+- `benchmarks/results/benchmark_current_result.json`
+
+You can link `benchmark_current_result.md` from README as the latest snapshot.
+
+## Cases
+
+- `small_files_rw`: write/read many small files
+- `stream_write_read`: write/read one larger stream in chunks
+
+Saved reports are written under `benchmarks/results/` when `--save-md auto` or `--save-json auto` is used.
+
+## Notes
+
+- `tracemalloc` reports Python-heap allocations; OS page cache and kernel-level effects are not fully represented.
+- `tempfile` results vary by OS, filesystem, and disk state. The included benchmark results were measured with the system `%TEMP%` directory located on a RAM disk. On a physical (SSD/HDD) disk, `tempfile` numbers will be significantly slower.
+- For fair comparisons, run on an idle machine and repeat multiple times.

CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- `MemoryFileSystem.is_file(path)` for API symmetry with `exists()` / `is_dir()`
+- `MemoryFileHandle` file-like methods: `truncate()`, `flush()`, `readable()`, `writable()`, `seekable()`
+- Async counterparts for new file-like methods and `AsyncMemoryFileSystem.is_file()`
+- `IMemoryFile._bulk_load(data)` method for internal use by `import_tree()` / `_deep_copy_subtree()` (improves encapsulation)
+
+### Changed
+- **[BREAKING]** `MFSStatResult.is_sequential` field removed. This internal implementation detail is no longer exposed in the public API.
+- **[BREAKING]** Package renamed from `memory-file-system` / `memory_file_system` to `D-MemFS` / `dmemfs`. Update imports: `from dmemfs import ...`
+- `stat()` directory support: previously raised `IsADirectoryError`; behavior unchanged in this release, but `is_dir` field added to `MFSStatResult` in future
+- `export_as_bytesio()` TOCTOU gap fixed: `_rw_lock.acquire_read()` now called inside `_global_lock` block
+- `exists()` / `is_dir()` now resolve paths under `_global_lock` for consistent thread-safety policy
+- `open(..., preallocate=...)` now performs preallocation under `_global_lock`
+- `import_tree()` quota accounting simplified to net-delta apply after successful write phase
+- `import_tree()` rollback now also cleans up auto-created parent directories
+- CI workflow now runs tests with coverage (`--cov`, `coverage.xml`) and uploads coverage artifact
+
+### Documentation
+- README / README_en updated with lock behavior notes (`_global_lock` hold, writer starvation)
+- README / README_en API tables updated for `is_file()` and new handle methods
+- Clarified that `stat()` is file-only and that `MFSStatResult.is_sequential` is implementation detail
+
+### Tests
+- Added tests for `is_file()`, handle file-like methods, async wrappers, and `import_tree()` parent-dir rollback
+
+## [0.2.0] - 2026-02-26
+
+### Added
+- **Directory Index Layer**: Internal refactoring from flat `dict[str, IMemoryFile]` to `DirNode`/`FileNode` tree structure
+  - `listdir()` complexity improved from O(N total entries) to O(children count)
+  - `rename()` / `rmtree()` no longer require prefix scanning
+- **`move(src, dst)`**: Move files/directories with automatic parent directory creation
+- **`copy_tree(src, dst)`**: Deep copy of directory subtrees with quota pre-check
+- **`stat(path)`**: Return `MFSStatResult` with size, timestamps, generation, storage type
+- **`glob("**")`**: Recursive glob pattern matching with `**` support
+- **File timestamps**: `created_at` / `modified_at` on `FileNode`, updated on write/truncate
+- **`MFSStatResult` TypedDict**: New typed return value for `stat()`
+- **`bytearray` shrink**: `RandomAccessMemoryFile.truncate()` reallocates buffer when new size ≤ 25% of old capacity
+- **`AsyncMemoryFileSystem` / `AsyncMemoryFileHandle`**: Async wrappers via `asyncio.to_thread()`
+- **`pytest-asyncio`** added to test dependencies
+
+### Changed
+- `IMemoryFile` is now pure data storage; `is_dir`, `generation`, `_rw_lock` moved to `DirNode`/`FileNode`
+- `wb` mode: truncate now executes **after** write-lock acquisition (prevents data corruption during concurrent reads)
+- `export_as_bytesio()`: entry lookup now protected by `_global_lock` (prevents race with concurrent `remove()`)
+- `MemoryFileHandle.__del__`: `stacklevel` changed from 2 to 1 for correct warning source location
+- `__version__` bumped to `0.2.0`
+
+### Tests
+- 288 tests (was 230): +58 new tests including `test_fs_coverage.py`
+- Coverage target documented at 99% (claim) with CI-side coverage XML generation in place
+- New test files: `test_timestamp.py`, `test_async.py`
+- Extended: `test_rename_move.py` (+13), `test_files_randomaccess.py` (+5), `test_concurrency.py` (+2), `test_open_modes.py` (+2), `test_export_import.py` (+1), `test_mkdir_listdir.py` (+2)
+
+## [0.1.0] - 2026-02-23
+
+### Added
+- Initial release of MemoryFileSystem (MFS)
+- In-process virtual filesystem with hard quota management
+- `MemoryFileSystem` class with full POSIX-like file operations:
+  - `open()` supporting modes `rb`, `wb`, `ab`, `r+b`, `xb` with optional preallocate
+  - `mkdir()`, `rename()`, `remove()`, `rmtree()`, `listdir()`, `exists()`, `is_dir()`
+  - `stats()` for quota and filesystem metrics
+  - `export_as_bytesio()`, `export_tree()`, `iter_export_tree()` with `only_dirty` support
+  - `import_tree()` with All-or-Nothing atomicity and rollback
+- `MemoryFileHandle` with full `io.RawIOBase`-compatible interface (read, write, seek, tell, truncate)
+- `MFSQuotaExceededError` — hard quota enforcement before any write
+- `ReadWriteLock` with configurable timeout — concurrent reads, exclusive writes
+- Auto-promotion from `SequentialMemoryFile` to `RandomAccessMemoryFile` on random-access writes
+- PEP 561 `py.typed` marker — full type annotation support
+- Zero external dependencies (Python 3.11+ standard library only)
+- 163 tests across Unit / Integration / Scenario / Property (Hypothesis) layers
+- CI matrix: 3 OS × 3 Python versions (3.11, 3.12, 3.13)

CONTRIBUTING.md

@@ -0,0 +1,52 @@
+# Contributing to D-MemFS
+
+Thank you for your interest in contributing! This document explains how to get involved.
+
+## Getting Started
+
+1. **Fork** the repository on GitHub.
+2. **Clone** your fork locally:
+   ```bash
+   git clone https://github.com/<your-username>/D-MemFS.git
+   cd D-MemFS
+   ```
+3. **Install dependencies** using [uv](https://github.com/astral-sh/uv):
+   ```bash
+   uv pip compile requirements.in -o requirements.txt
+   ```
+4. **Run the test suite** to confirm everything passes:
+   ```bash
+   uvx --with-requirements requirements.txt --with-editable . pytest tests/ -v --timeout=30
+   ```
+
+## Development Workflow
+
+- Create a **feature branch**: `git checkout -b feat/my-feature`
+- Write tests **before** or **alongside** code changes.
+- All tests must pass before submitting a PR.
+- Keep commits focused and use descriptive commit messages.
+
+## Code Style
+
+- Follow existing patterns — no additional linting tools are required.
+- Public APIs must have type annotations.
+- New public methods must be documented in the docstring and, if applicable, reflected in `README.md` / `README_ja.md`.
+
+## Submitting a Pull Request
+
+1. Push your branch to your fork.
+2. Open a Pull Request against the `main` branch.
+3. Describe **what** you changed and **why**.
+4. Link any related issues.
+
+## Reporting Bugs
+
+Please open a GitHub Issue with:
+- A minimal reproducible example
+- Python version and OS
+- Expected vs. actual behaviour
+
+## Design Principles
+
+Before proposing new features, please review the [Non-Goals](README.md) section.
+MFS intentionally avoids `os.PathLike` compatibility and does not aim to replace `os` or `pathlib`.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 D
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.