profile
viewpoint
Mani Maghsoudlou manidlou California full stack dev, collaborator at node-fs-extra &-> mind explorer.

push eventjprichardson/node-fs-extra

Maxime Bargiel

commit sha 391fb7365e8fa53af5eadf8d673eda1fdda30db9

Add heuristics to stats.ino comparison when bigint is not available

view details

Mani Maghsoudlou

commit sha e3d1ab855a17a223e7e8be89d100a5d0ebbdf98c

Merge pull request #694 from mbargiel/feature/stats-heuristics-without-bigint Add heuristics to stats.ino comparison when bigint is not available

view details

push time in a month

PR merged jprichardson/node-fs-extra

Reviewers
Add heuristics to stats.ino comparison when bigint is not available

The resolution of https://github.com/jprichardson/node-fs-extra/issues/657 relies on Node >= 10.5, but actually leaves the issue not fixed on previous versions of Node (in particular, for the OP, who reported the issue on Node 8.11.3). The issue particularly problematic and frequent in automated setups, such as running tests that create files in sequence, where it is likely to happen.

This PR minimizes the frequency of this issue by comparing additional fs.Stats values when Bigint support is not available. Note that the check is still heuristic: different stats prove they are different files*, but while same stats do not prove they are the same, we must err on the side of caution and still treat them as the same file.

(*) Race condition: comparing file stats before performing a copy or other operation implicitly introduces a race condition. However the existing implementation already suffered from this, so we presume that this PR does not make the condition much worse (with the exception that unlike ino and vol, a malicious user could in theory manipulate some of the other stats). In practice, this race condition seems extremely hard to abuse, since it requires using a vulnerable system, and a means to change the file stats between the two stats calls made by getStats and getStatsSync.

+28 -5

6 comments

1 changed file

mbargiel

pr closed time in a month

pull request commentjprichardson/node-fs-extra

Add heuristics to stats.ino comparison when bigint is not available

@RyanZim do we wanna merge this directly into master? we have #633 and #698 that are apparently gonna be ready soon as well!

mbargiel

comment created time in a month

pull request commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

so this is my suggestion:

  • do not set the dest mode when we create the dest
  • use native fs.utimes*() and fs.futimes*() (instead of our internal utimes implementation)
  • if we do the above, then we don't need to check if the file is writable and we can simplify the setDestTimestampsAndMode function like this, which is so much easier to read and understand. All tests also pass with this implementation. I tried your example too and that one worked as well:
function setDestTimestampsAndMode (srcStat, src, dest, opts) {
  if (opts.preserveTimestamps) {
    const updatedSrcStat = fs.statSync(src)
    fs.utimesSync(dest, updatedSrcStat.atime, updatedSrcStat.mtime)
  }
  fs.chmodSync(dest, srcStat.mode)
}
function setDestTimestampsAndModeFd (srcStat, src, fdw, opts) {
  if (opts.preserveTimestamps) {
    const updatedSrcStat = fs.statSync(src)
    fs.futimesSync(fdw, updatedSrcStat.atime, updatedSrcStat.mtime)
  }
  fs.fchmodSync(fdw, srcStat.mode)
}
mbargiel

comment created time in a month

Pull request review commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

 function mayCopyFile (srcStat, src, dest, opts) { function copyFile (srcStat, src, dest, opts) {   if (typeof fs.copyFileSync === 'function') {     fs.copyFileSync(src, dest)-    fs.chmodSync(dest, srcStat.mode)     if (opts.preserveTimestamps) {-      return utimesSync(dest, srcStat.atime, srcStat.mtime)+      // Make sure the file is writable first+      if ((srcStat.mode & 0o200) === 0) {+        fs.chmodSync(dest, srcStat.mode | 0o200)

so this is my suggestion:

  • do not set the dest mode when we create the dest
  • use native fs.utimes*() and fs.futimes*() (instead of our internal utimes implementation)
  • if we do the above, then we don't need to check if the file is writable and we can simplify the setDestTimestampsAndMode function like this, which is so much easier to read and understand. All tests also pass with this implementation. I tried your example too and that one worked as well:
function setDestTimestampsAndMode (srcStat, src, dest, opts) {
  if (opts.preserveTimestamps) {
    const updatedSrcStat = fs.statSync(src)
    fs.utimesSync(dest, updatedSrcStat.atime, updatedSrcStat.mtime)
  }
  fs.chmodSync(dest, srcStat.mode)
}
function setDestTimestampsAndModeFd (srcStat, src, fdw, opts) {
  if (opts.preserveTimestamps) {
    const updatedSrcStat = fs.statSync(src)
    fs.futimesSync(fdw, updatedSrcStat.atime, updatedSrcStat.mtime)
  }
  fs.fchmodSync(fdw, srcStat.mode)
}
mbargiel

comment created time in a month

Pull request review commentjprichardson/node-fs-extra

Add heuristics to stats.ino comparison when bigint is not available

 function checkParentPathsSync (src, srcStat, dest, funcName) {     if (err.code === 'ENOENT') return     throw err   }-  if (destStat.ino && destStat.dev && destStat.ino === srcStat.ino && destStat.dev === srcStat.dev) {+  if (areApparentlyIdentical(srcStat, destStat)) {     throw new Error(errMsg(src, dest, funcName))   }   return checkParentPathsSync(src, srcStat, destParent, funcName) } +function areApparentlyIdentical (srcStat, destStat) {

This is good but the function name is a bit confusing. Will you please rename it to something simpler like just areIdentical?

mbargiel

comment created time in a month

push eventmanidlou/node-klaw-sync

Mani Maghsoudlou

commit sha 38ee0e3fb302313b8adb0018b40adc88d3103e70

update readme

view details

push time in a month

issue closedmanidlou/node-klaw-sync

Confusing description of traverseAll default value

Package version: 6.0.0

The README reads:

  • traverseAll <Boolean> default: true
    • traverse all subdirectories, regardless of filter option. (When set to true, traverseAll produces similar behavior to the default behavior prior to v4.0.0. The current default of traverseAll: false is equivalent to the old noRecurseOnFailedFilter: true).

First, it says it's true by default.

Then it says "The current default of traverseAll: false" implicating it's false by default.

In the code it is in fact undefined, which is equivalent to false in this case.

If the parameter is not present, klaw-sync behaves as if it was false.

What is the intended default value of this parameter, true or false? true seems more reasonable to me, but regardless, I believe default value should be indicated clearly in the docs and correspond to the actual behavior.

I also find it confusing that sister module node-klaw does not have this option at all.

closed time in 3 months

l1bbcsg

issue commentmanidlou/node-klaw-sync

Confusing description of traverseAll default value

Closed via https://github.com/manidlou/node-klaw-sync/commit/bc081f0b46a2795e59c88f758b5c46cc1bfe06dd

l1bbcsg

comment created time in 3 months

push eventmanidlou/node-klaw-sync

Mani Maghsoudlou

commit sha bc081f0b46a2795e59c88f758b5c46cc1bfe06dd

update docs

view details

push time in 3 months

Pull request review commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

 function copyFileFallback (srcStat, src, dest, opts) {     pos += bytesRead   } -  if (opts.preserveTimestamps) fs.futimesSync(fdw, srcStat.atime, srcStat.mtime)+  if (opts.preserveTimestamps) {+    const updatedSrcStat = fs.statSync(src)+    fs.futimesSync(dest, updatedSrcStat.atime, updatedSrcStat.mtime)+  } 

Also I just realized we forgot to chmod again here before closing fds. copy does that and we need to be consistent. So I think it is better we have the same function setDestModeAndTimestamps() like in copy so that we can call it after both native copy and fallback.

mbargiel

comment created time in 3 months

Pull request review commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

 function copyFileFallback (srcStat, src, dest, opts, cb) {     const ws = fs.createWriteStream(dest, { mode: srcStat.mode })     ws.on('error', err => cb(err))       .on('open', () => rs.pipe(ws))-      .once('close', () => setDestModeAndTimestamps(srcStat, dest, opts, cb))+      .once('close', () => setDestModeAndTimestamps(srcStat, src, dest, opts, cb))   }) } -function setDestModeAndTimestamps (srcStat, dest, opts, cb) {-  fs.chmod(dest, srcStat.mode, err => {+function setDestModeAndTimestamps (srcStat, src, dest, opts, cb) {+  if (!opts.preserveTimestamps) {+    return fs.chmod(dest, srcStat.mode, cb)+  }++  const next = (err) => {     if (err) return cb(err)-    if (opts.preserveTimestamps) {-      return utimes(dest, srcStat.atime, srcStat.mtime, cb)-    }-    return cb()-  })+    // Cannot rely on previously fetched srcStat because atime is updated by+    // the read operation: https://nodejs.org/docs/latest-v8.x/api/fs.html#fs_stat_time_values+    return fs.stat(src, (err, srcStat) => {+      if (err) return cb(err)+      return utimes(dest, srcStat.atime, srcStat.mtime, (err) => {+        if (err) return cb(err)+        return fs.chmod(dest, srcStat.mode, cb)+      })+    })+  }++  if ((srcStat.mode & 0o200) === 0) {+    // Make sure the file is writable first+    return fs.chmod(dest, srcStat.mode | 0o200, next)

This doesn't seem required! also this is a little hard to read. I think that'd be better we just run the functions without creating another inner function.

mbargiel

comment created time in 3 months

Pull request review commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

 function copyFileFallback (srcStat, src, dest, opts) {     pos += bytesRead   } -  if (opts.preserveTimestamps) fs.futimesSync(fdw, srcStat.atime, srcStat.mtime)+  if (opts.preserveTimestamps) {+    const updatedSrcStat = fs.statSync(src)+    fs.futimesSync(dest, updatedSrcStat.atime, updatedSrcStat.mtime)

fs.futimesSync() should use fdw and not dest.

mbargiel

comment created time in 3 months

Pull request review commentjprichardson/node-fs-extra

Update dest atime after copy/copy-sync

 function mayCopyFile (srcStat, src, dest, opts) { function copyFile (srcStat, src, dest, opts) {   if (typeof fs.copyFileSync === 'function') {     fs.copyFileSync(src, dest)-    fs.chmodSync(dest, srcStat.mode)     if (opts.preserveTimestamps) {-      return utimesSync(dest, srcStat.atime, srcStat.mtime)+      // Make sure the file is writable first+      if ((srcStat.mode & 0o200) === 0) {+        fs.chmodSync(dest, srcStat.mode | 0o200)

This seems unnecessary! we are chmoding dest at the end anyway.

mbargiel

comment created time in 3 months

more