-
Notifications
You must be signed in to change notification settings - Fork 232
Use Dart library to read and write tar files #2817
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@simolus3, this is AWESOME! 🚀 @walnutdust has some work in progress in a PR with a tar-stream reader like this in google/dart-neats#56 I wonder if we could combine efforts?
@walnutdust, do you recall why we went away from making transforming a |
Edit: I was a bit slow to see your email, I'll reply in more detail there. I didn't know about the existing effort (and sorry for snatching the package name :D).
That's true, but I like that I'm not sure what the best way forward is now. I can open a PR to add my writer to @walnutdust's implementation and transfer the pub package to you when that gets merged. I need a tar package that works without |
This might have been because we were trying to move away from @simolus3 I think making If we are certain that the relevant
|
I have tried unpacking all archives of pub.dev with this code and the code currently in pub,.dev (on linux), then running diff -rq on the results. I found no differences (after fixing for empty directories) except for obscure cases like |
Co-authored-by: Sigurd Meldgaard <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly nits.
I still think we should forbid symlinks, but that's something we can do on the server. That way we can roll it back if people decide we really need it.
@@ -216,6 +215,10 @@ Future<String> _createFileFromStream(Stream<List<int>> stream, String file) { | |||
}); | |||
} | |||
|
|||
void _chmod(int mode, String file) { | |||
runProcessSync('chmod', [mode.toRadixString(8), file]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we should use /bin/chmod
if it exists, so we are not affected by users tweaking PATH
.
According to FHS 3.4.2 /bin/chmod
should always exist on Linux.
I'm guessing it has a fixed location on Mac OS X as well. I think we should prefer the fixed location, if it exists that way we are not affected by changes to PATH
-- I think we did this when using tar :D
Perhaps we should also validate the mode
and check the exit code from running chmod
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See:
/// Find a tar. Prefering system installed tar.
///
/// On linux tar should always be /bin/tar [See FHS 2.3][1]
/// On MacOS it seems to always be /usr/bin/tar.
///
/// [1]: https://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.pdf
String _findTarPath() {
for (final file in ['/bin/tar', '/usr/bin/tar']) {
if (fileExists(file)) {
return file;
}
}
log.warning(
'Could not find a system `tar` installed in /bin/tar or /usr/bin/tar, '
'attempting to use tar from PATH');
return 'tar';
}
Let's do that to find chmod
, just in case someone decorates their shell with a different chmod
🙈
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could perhaps also use dart:ffi
to bind the syscall... But we a little care has to be taken that we don't bind symbols only exposed because the Linux build is in release mode and not product mode.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems to be /bin/chmod
on macOS too, so that should work.
Using dart:ffi
sounds interesting (especially because we could check the value of umask
at runtime too). AFAIK Dart doesn't support linking libc statically, so the symbols should be available
final resolvedTarget = path.joinAll( | ||
[parentDirectory, ...path.posix.split(entry.header.linkName)]); | ||
if (!path.isWithin(destination, resolvedTarget)) { | ||
// Don't allow links to files outside of this tar. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should print a warning when we ignore thing. I think it's the right call here.
lib/src/io.dart
Outdated
TarHeader( | ||
// Ensure paths in tar files use forward slashes | ||
name: path.url.joinAll(path.split(relative)), | ||
mode: stat.mode, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should do: mode: defaultMode | (stat.mode & executableMask);
final file = File(path.normalize(entry)); | ||
final stat = file.statSync(); | ||
|
||
return TarEntry( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it allowed to make file entries without first having the directory entries.
If allowed by tar format, then I think just having files is preferable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I couldn't find any documentation stating whether it is acceptable to exclude directory entries or not.
GNU tar has explicit support for it though (it recovers from ENOENT
by creating parent directories), and I'm not aware of any tar client that doesn't support this.
final relative = path.relative(entry, from: baseDir); | ||
// On Windows, we can't open some files without normalizing them | ||
final file = File(path.normalize(entry)); | ||
final stat = file.statSync(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we test if this is a symlink... If so maybe we should print a warning line that the symlink will be embedded as a file, and NOT a link.
IMO, it's probably preferable to keep the format simple and avoid fancy features like symlinks in distributed packages.
Uses
pkg:tar
to read and write tar files. That package is based on stream transformers and generally tries to minimize memory overhead. It supports the ustar format and extended PAX headers.For testing, I ran
tool/extract_all_pub_dev.dart
, but only with the latest version of each package. To test backwards-compatibility, I published some packages withpkg:tar
to a personal pub server and downloaded them with the regular pub on Linux and Windows. I don't have access to a Mac, but I can run more tests on the other OSes if needed.Full disclosure: I wrote the tar package.
Closes #1602 (and a bunch of related issues).