cluster + rotating file + large log file = crash

[2fours]

Running on linux...see this in my journal at exactly midnight:

Dec 19 17:00:00 surespot-node-2 coffee[19539]: events.js:72
Dec 19 17:00:00 surespot-node-2 coffee[19539]: events.js:72
Dec 19 17:00:00 surespot-node-2 coffee[19539]: throw er; // Unhandled 'error' event
Dec 19 17:00:00 surespot-node-2 coffee[19539]: ^
Dec 19 17:00:00 surespot-node-2 coffee[19539]: throw er; // Unhandled 'error' event
Dec 19 17:00:00 surespot-node-2 coffee[19539]: Error: ENOENT, rename 'logs/surespot.log'
Dec 19 17:00:00 surespot-node-2 coffee[19539]: ^
Dec 19 17:00:00 surespot-node-2 coffee[19539]: Error: ENOENT, rename 'logs/surespot.log'

The log files now look like this:

-rw-r--r-- 1 surespot users 10966716 Dec 19 16:59 surespot.log.2
-rw-r--r-- 1 surespot users 0 Dec 19 17:00 surespot.log.0
-rw-r--r-- 1 surespot users 58868418 Dec 20 08:11 surespot.log

[2fours]

I think it's because I'm using cluster with multiple processes...it doesn't happen when I test with 1 process. How would you recommend using bunyan with cluster?

[2fours]

If I name the log files with the pid in them like so:

bunyanStreams = [{
type: 'rotating-file'
path: "logs/surespot_#{process.pid}.log"
count: 7
level: 'debug'
period: '6h'
}]

I don't have issues with multiple processes, but now I have three log files, one for each process - the master and the two forked children. This sort of defeats the purpose of having the pid queryable in the log for me. Suggestions?

[2fours]

Turns out naming the file with the pid in it doesn't work either under load. Suggestions?

[2fours]

Found some time to look into this some more...things work fine with small log files but once you start getting into bigger log (hundreds of MB) files you get the error. Looking into using some sort of lock file solution to sync across the processes but it's hard because each process sets its own log rotate time independently.

[2fours]

Couldn't get it to do what I want. Now I'm just sending bunyan output to stdout and using systemd service to send to journal.

[seanzx85]

+1 Started seeing this on our larger scale apps.

Cluster: 1 master, 4 children
Log Size: 23KB

{"name":"xxxx","hostname":"server1","pid":24054,"level":50,"msg":"uncaughtException: Error: ENOENT, rename '/logs/node/xxxx.log.4'","time":"2014-02-19T00:00:00.009Z","v":0}
{"name":"xxxx","hostname":"server1","pid":24054,"level":50,"msg":"Error: ENOENT, rename '/logs/node/xxxx.log.4'","time":"2014-02-19T00:00:00.010Z","v":0}

[dnndev]

I'm getting the same issue. It's causing my server to restart each time resulting in lost sessions.

[gabegorelick]

+1 This (or something very similar) just bit me with cluster and rotating-file:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: ENOENT, rename '/foo/bar.log.1'

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: ENOENT, rename '/foo/bar.log.1'

[ryanrolds]

Running in to this issue. Causing workers to crash and refork.

[larrymyers]

The best solution I've found so far is to just use a single file for logging, and then setup logrotate with the copytruncate option. Gives you the same end result without pissing off cluster workers.

[gabegorelick]

It looks like winston supports rotation. How do they avoid this issue?

[rosskukulinski]

@gabegorelick the last time I used winston their log-rotate didn't follow normal *nix file rotation standards. One of many reasons I'm using bunyan now.