Symlinks on Modern Windows with latest node/npm.

[andrewfinnell]

Theree are not only two options to creating a symlink. Either Node should change it's default behavior for the symlink type 'dir', or module makes should start using 'junction' instead of 'dir' when on windows. The symlink 'dir' on Windows does not behave the same way as a POSIX style symlink.

The proper symlink for windows is a Junction which makes more sense when comparing it to a *nix style symlink.

symlink's of type 'Dir' cannot be created on windows without a Run As verb which is completely unacceptable. I've patched dep-linker to work on all platforms and not require the process to be ran as an actual Administrator.

This is the equivalent of REQUIRING a Linux user to run gulp as root. Not sudo'ing the process. But literally requiring them to run gulp as root user. I cannot imagine that would ever make it into a pull request, why does it make it into pull requests on the Windows side?

[yocontra]

@afinnell I don't think any of us use windows so oversights like this happen - no need to get angry at us. We try our best to support windows even though 99% of our users are on unix systems, but that does put some responsibility on the 1% who do use windows to report issues - so thank you for doing that.

If I'm understanding you right, you're saying this should be changed to use junction for directories on windows, right?

[andrewfinnell]

@contra My apologies. That Issue came across more hostile then intended. There's a bit of a closd-circle regarding npm's, fixes, and the ability to easily get them back onto a developers system using NPM. Meaning it's difficult to path modules ourselves and then use them across all machines. We have to wait for the author's to merge then release.

And it is also not so easy to create our own modules without going through the npmjs processs. At any rate I've been on the hunt to find something to easily symlink node_modules to a /lib dir and each symlink module always uses 'dir'.

At any rate I've submitted a pull request at #211

On Windows, the reason the DIR symlink requires such a heavy lock down is because it follows symlinks based on the LOCAL USER's relative path. So accessing symlinks across a shared drive can result in a User's own local directory getting resolved instead of the intended original link. Personally I think it's Microsofts fault for renaming it to Junction's.

But, in short, yes, what I have found is that Windows Junction === Linux Directory symlinks. If I've overlooked anything it would be very helpful for someone to point it out before I go too far down the Junction path.

Thanks!

[andrewfinnell]

@contra I updated my local copy of the vinyl-fs module with my fix and it works as is now. It creates the links as one would expect without requiring the Verb: Run As execution (Right Click, Run As Administrator). As for Linux the code should run exactly as it did before.

For users that setup their systems on Windows and truly want the non-Junction way, I've added a check for opts.disableFunction into the mix but I can't imagine there would be many Windows users wishing to do so.

This might be an opportunity to do some 'auto-magic' in the file system modules like graceful-fs and such. Where we can keep the modules as is ('dir' or 'file') and have the filesystem module take care of turning a 'dir' into a 'junction' if on windows. Thoguhts?

[yocontra]

@afinnell You'd have to bring that up with the people who maintain the graceful-fs module, not sure who that is now.

@phated Got a sec to review the PR? I know you've been deep on the symlink stuff lately.

[phated]

For Windows, we've been defaulting to no-ops for behavior node on Windows or the Windows platform does poorly (permissions, file times, etc). I'll look at the symlink stuff as discussed and PRed but I know npm was having a miserable time with it. Can't promise anything.

[phated]

Please make sure you are using the latest vinyl-fs also. Many people using the latest gulp on npm think they have the latest vinyl-fs but don't.

[andrewfinnell]

As products such as IntelliJ, Eclipse, etc. don't understand NTFS Junction Reparse Points, it becomes dangerous to Junction a Directory. Deleting the directory in that editor results in all the children being recursively deleted.

I've solved this issue (for now) by doing the following:

gulpfile.js

/**
 * Gulpfile.js
 * Symlinks node_module dependencies into any directory. This
 * file works on Windows and Linux machines by recursively
 * symlinking the files within a node module instead of symlinking
 * just the directory. This is vitally important as Editor's on Windows
 * do not understand Junctions and therefor will delete all the children
 * files first (i.e. the original files from the original directory), a
 * disasterous situation, especially if this is used with npm link.
 */

var gulp          = require('gulp'),
    clean         = require('gulp-clean'),
    path          = require('path'),
    fs            = require('graceful-fs'),
    source        = require('vinyl-source-stream'),
    through2      = require('through2'),
    prepareWrite  = require('vinyl-fs/lib/prepareWrite'),

    // Project package.json
    PACKAGE_JSON  = JSON.parse (fs.readFileSync ('./package.json')),

    // Project Structure 
    APP = {
      SRC: {
        BASE: "./src/main/webapp/js",
        DEST: "./src/main/webapp/dist",
        GLOBS: [
          "**/*.js"
        ],
        FILTER: () => APP.SRC.GLOBS.map(glob => path.join(APP.SRC.BASE, glob))
      },
      TESTS: {
        BASE: "./src/test/js",
        DEST: "./src/test/dist",
        GLOBS: [
          "**/*.js"
        ],
        FILTER: () => APP.TESTS.GLOBS.map(glob => path.join(APP.TESTS.BASE, glob))
      },
      DEPS: {
        BASE: "./node_modules",
        DEST: "./src/main/webapp/lib",
        // This is important to ensure that the files, not the directories get linked.
        GLOBS: Object.keys (PACKAGE_JSON.dependencies).map(dep => dep + "/**/*"),
        FILTER: () => APP.DEPS.GLOBS.map (glob => path.join(APP.DEPS.BASE, glob))
      }
    };

gulp.task ('default', ['build:deps', 'build:src']);

// Link the dependencies to the lib directory instead of requiring a copy.
gulp.task ('build:deps', function () {
  return gulp.src(APP.DEPS.FILTER(), { base: APP.DEPS.BASE})
             .pipe(symlink(APP.DEPS.DEST, {}));
});

// Just an example. Babel would go here.
gulp.task ('build:src', function () {
  return gulp.src(APP.SRC.FILTER(), { base: APP.SRC.BASE})
             .pipe(gulp.dest(APP.SRC.DEST));
});

// Clean project
gulp.task('build:clean', function () {
  return gulp.src([APP.SRC.DEST, APP.DEPS.DEST, APP.TESTS.DEST])
             .pipe(clean());
});

/**
 * Create a special symlink stream processor which will 
 * use junctions instead of dir on Windows. As the 'dir' symlink
 * on Windows is all but impossible to create as a User with Admin rights. 
 */
function symlink(outFolder, opt) {
  function linkFile(file, enc, cb) {
    var srcPath = file.path;
    var symType = file.isDirectory() ? ((process.platform === 'win32') ? 'junction' : 'dir') : 'file';
    prepareWrite(outFolder, file, opt, function(err, writePath) {
      if (err) {
        return cb(err);
      }
      fs.symlink(srcPath, writePath, symType, function(err) {
        if (err && err.code !== 'EEXIST') {
          return cb(err);
        }
        cb(null, file);
      });

    });
  }
  var stream = through2.obj(opt, linkFile);
  stream.resume();
  return stream;
}