Skip to content

red-knot: Add directory support to MemoryFileSystem #11825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 13, 2024

Conversation

MichaReiser
Copy link
Member

@MichaReiser MichaReiser commented Jun 10, 2024

Summary

This PR adds support for handling directories to the MemoryFileSystem.
I'm adding this now because the module resolver contains logic that branches if path is a directory.

I used this PR to remove permissions support from the MemoryFileSystem. I think having support for it is misleading, it gives the impression that the file system would check permissions when reading/writing which it does not and I don't plan on adding.
The MemoryFileSystem is mainly intended for testing. We should use the real file system to test more advanced or even platform specific filesystem behavior. That's the only way to get that right.

Test Plan

Added tests

@MichaReiser MichaReiser requested a review from dhruvmanila as a code owner June 10, 2024 14:44
@MichaReiser MichaReiser added the ty Multi-file analysis & type inference label Jun 10, 2024
@MichaReiser MichaReiser changed the base branch from main to salsa-semantic-jar June 10, 2024 14:45
@MichaReiser MichaReiser deleted the branch main June 11, 2024 13:55
@MichaReiser MichaReiser changed the title red-knot: Add directory support to MemoryFileSystem red-knot[salsa part 4]: Add directory support to MemoryFileSystem Jun 11, 2024
@MichaReiser MichaReiser reopened this Jun 11, 2024
@MichaReiser MichaReiser changed the base branch from salsa-2-semantic-jar to salsa-source June 11, 2024 14:22
@MichaReiser MichaReiser removed the request for review from dhruvmanila June 11, 2024 14:22
@MichaReiser MichaReiser force-pushed the salsa-memory-fs-directories branch 2 times, most recently from bee11e7 to 0a7a310 Compare June 11, 2024 14:23
Copy link
Contributor

github-actions bot commented Jun 11, 2024

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

@MichaReiser MichaReiser force-pushed the salsa-memory-fs-directories branch from 0a7a310 to c5e4ec0 Compare June 12, 2024 07:16
Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there are some clippy/test problems to resolve before merging this.

Comment on lines +13 to +19
/// The implementation doesn't aim at fully capturing the behavior of a real file system.
/// The implementation intentionally doesn't support:
/// * symlinks
/// * hardlinks
/// * permissions: All files and directories have the permission 0755.
///
/// Use a tempdir with the real file system to test these advanced file system features and complex file system behavior.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to this approach. We should use in-memory file system for fast and easy testing of semantic analysis, not for testing filesystem behaviors.

struct MemoryFileSystemInner {
files: FxDashMap<FileSystemPathBuf, FileData>,
by_path: RwLock<FxHashMap<Utf8PathBuf, Entry>>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the switch from FxDashMap to an FxHashMap wrapped in a RwLock?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I don't think it was necessary as part of this PR. I first thought I would need a BTreeMap. I'll leave it this way because I do need a BTreeMap when adding support for removing directories.

Comment on lines +215 to +216
/// Normalizes the path by removing `.` and `..` components and transform the path into an absolute path.
///
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am very surprised that we have to implement such a thing ourselves instead of just using a battle-tested library method for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it exists, it's just not stabalized. @BurntSushi knows more ;)

Base automatically changed from salsa-source to main June 13, 2024 07:37
@MichaReiser MichaReiser force-pushed the salsa-memory-fs-directories branch from c5e4ec0 to 671d556 Compare June 13, 2024 07:44
@MichaReiser MichaReiser changed the title red-knot[salsa part 4]: Add directory support to MemoryFileSystem red-knot: Add directory support to MemoryFileSystem Jun 13, 2024
@MichaReiser MichaReiser enabled auto-merge (squash) June 13, 2024 07:45
@MichaReiser MichaReiser merged commit 22b6488 into main Jun 13, 2024
17 of 18 checks passed
@MichaReiser MichaReiser deleted the salsa-memory-fs-directories branch June 13, 2024 07:48
Copy link

codspeed-hq bot commented Jun 13, 2024

CodSpeed Performance Report

Merging #11825 will improve performances by 13.32%

Comparing salsa-memory-fs-directories (671d556) with main (d4dd96d)

Summary

⚡ 1 improvements
✅ 29 untouched benchmarks

Benchmarks breakdown

Benchmark main salsa-memory-fs-directories Change
linter/all-with-preview-rules[numpy/globals.py] 915.2 µs 807.6 µs +13.32%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ty Multi-file analysis & type inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants