using node to find all files in a directory

Turns out Node has readdir for exactly this purpose.

From the docs:

Asynchronous readdir(3). Reads the contents of a directory. The callback gets two arguments (err, files) where files is an array of the names of the files in the directory excluding ’.’ and ’..‘.

const path = require("path")
const fs = require("fs")

// hard-coding our directory path
const directoryPath = path.resolve(__dirname, "path/to/directory/with/files")

fs.readdir(directoryPath, function (err, files) {
  const allFiles = []
  if (err) {
    console.log("Error getting directory information.")
  } else {
    files.forEach(function (file) {
      console.log(`file -->`, file)
    })
  }
})

While, by default file types are excluded, there’s an option withFileTypes that can be turned on to ensure they’re included.

How could we make this slightly more usable though? Perhaps by converting it into a function that can receive a directory path…

At this point we might be tempted to do something like:

const path = require("path")
const fs = require("fs")

function getFiles(directoryPath) {
  const allFiles = []
  fs.readdir(directoryPath, function (err, files) {
    if (err) {
      console.log("Error getting directory information.")
    } else {
      files.forEach(function (file) {
        const filePath = directoryPath.concat(file)
        allFiles.push(filePath)
      })
      return allFiles
    }
  })
}

The problem, however, is that readdir is asynchronous (there is a readdirsync though).

The result is that if we were to run:

const directoryPath = path.resolve(__dirname, "../../data/localData")
const files = getFiles(directoryPath)
console.log(files) // undefined

At this point, there are two ways we can solve this:

1. Convert to the synchronous version of readdir (i.e. use readdirsync)
2. Pass in a callback into which our results can passed once they’ve resolved.

Looking at each in turn.

Asynchronous With Callbacks

Callbacks have been one of those parts of learning Javascript that’s given me fits for ages. It was all the more surprising then that not only did I quickly recognize the reason my files was undefined was due to asynchronicity of readdir but that I saw how to pass in a callback!

Here’s one way to look at it:

const path = require("path")
const fs = require("fs")

function getFiles(directoryPath, cb) {
  fs.readdir(directoryPath, function (err, files) {
    if (err) {
      console.log("Error getting directory information.")
    } else {
      cb(files)    }
  })
}

Now, once fs.readdir completes, it passes the files (our data received from the function) into our callback (cb by convention).

To use this new getFiles, we can do the following:

const directoryPath = path.resolve(__dirname, "../../data/localData");
function myCallback(value){console.log('files ->', value}
getFiles(directoryPath, myCallback)

You might notice that we’re just passing in the function, myCallback and not myCallback(files), which would also work. This reason we don’t need to is because of currying.

Synchronous Approach

Switching gears a bit, if we’re willing to pause your application and switch from the native asynchronous nature of javascript to synchronous, we can simplify this even further.

const path = require("path")
const fs = require("fs")

function getFiles(directoryPath) {
  return fs.readdirsync(directoryPath)
}

Which can then be used as:

const directoryPath = path.resolve(__dirname, "../../data/localData")
console.log(`files --> `, getFiles(directoryPath))

Wrapping Up

Because so much of Javascript is built on open sourced packages, it’s easy to forget how much we can do with the native APIs. Exploring Node is often fruitful - not only to find new solutions to problems, but to reinforce how programming paradigms like asynchronicity work.