Node Wait for File to Finish Reading

As a Node.js developer, at that place's a good hazard that at some point yous've imported the fs module and written some code that's interacted with the file organization.

What you might not know is that the fs module is a fully-featured, standards-based, cross-platform module that exposes not one, but 3 APIs that cater to synchronous and asynchronous programming styles.

In this article, we will thoroughly explore the world of Node.js file processing in Windows and Linux systems, with a focus on the fs module's hope-based API.

A note before we brainstorm

All examples in this commodity are intended to be run in a Linux environment, just many will work in Windows, too. Await for notes throughout the commodity highlighting examples that won't work in Windows. Regarding macOS — in most cases, the fs module works the aforementioned way it would on Linux, but in that location are some macOS-specific behaviors that are not covered in this article. Refer to the official Node.js documentation for macOS nuances.

The full source code for all examples is bachelor on my GitHub under briandesousa/node-file-procedure.

Introducing the fs module

The fs module is a cadre module built into Node.js. It has been effectually since the kickoff, all the way back to the original Node.js v0.ten releases.

Since its earliest days, the fs module has been aligned with POSIX file organisation standards. This means the code you write is somewhat portable across multiple operating systems, though especially between different flavors of Unix and Linux.

Although Windows is not a POSIX-compliant operating system, about of the fs module'south functions will still work. However, in that location are functions that are non portable simply because certain file arrangement capabilities do non exist or are implemented differently in Windows.

As we review the fs module's functions, keep in mind that the post-obit functions will render errors or volition take unexpected results on Windows:

• Functions to alter file permissions and ownership:
  • chmod()
  • chown()
• Functions to piece of work with hard and soft links:
  • link()
  • symlink()
  • readlink()
  • lutimes()
  • lchmod()
  • lchown()
• Some metadata is either non fix or displays unexpected values when using stat() and lstat()

Since Node v10, the fs module has included three unlike APIs: synchronous, callback, and promise. All 3 APIs expose the same ready of file organization operations.

This article will focus on the newer hope-based API. However, there may be circumstances where you desire or need to utilize the synchronous or callback APIs. For that reason, let'southward accept a moment to compare all three APIs.

Comparing the FS module APIs

Synchronous API

The synchronous API exposes a fix of functions that block execution to perform file arrangement operations. These functions tend to be the simplest to use when you're just getting started.

On the other hand, they are thread-blocking, which is very contrary to the non-blocking I/O design of Node.js. Even so, there are times where you lot must process a file synchronously.

Here is an example of using the synchronous API to read the contents of a file:

import * as fs from 'fs';  const data = fs.readFileSync(path); panel.log(information);        

Callback API

The callback API allows you to collaborate with the file system in an asynchronous fashion. Each of the callback API functions accept a callback function that is invoked when the performance is completed. For example, we can call the readFile function with an arrow role that receives an mistake if at that place is a failure or receives the data if the file is read successfully:

import * as fs from 'fs';  fs.readFile(path, (err, data) => {     if (err) {         console.error(err);     } else {         console.log(`file read complete, information: ${data}`);     } });        

This is a non-blocking approach that is usually more than suitable for Node.js applications, just information technology comes with its own challenges. Using callbacks in asynchronous programming often results in callback hell. If you're not careful with how you structure your code, y'all may end up with a complex stack of nested callback functions that can be difficult to read and maintain.

Promise API

If synchronous APIs should be avoided when possible, and callback APIs may non be ideal, that leaves usa with the promise API:

import * as fsPromises from 'fs/promises';  async office usingPromiseAPI(path) {     const hope = fsPromises.readFile(path);     panel.log('practice something else');     return expect promise; }        

The first thing you might notice is the divergence in this import argument compared to the previous examples: the promise API is available from the promises subpath. If you are importing all functions in the promise API, the convention is to import them as fsPromises. Synchronous and callback API functions are typically imported as fs.

If yous desire to proceed example code compact, import statements will be omitted from subsequent examples. Standard import naming conventions will be used to differentiate betwixt APIs: fs to access synchronous and callback functions, and fsPromises to access promise functions.

The promise API allows you to take advantage of JavaScript's async/await syntactic sugar to write asynchronous code in a synchronous way. The readFile() function called on line four above returns a promise. The code that follows appears to be executed synchronously. Finally, the promise is returned from the function. The expect operator is optional, but since we have included it, the function will wait for the file operation to consummate before returning.

It's fourth dimension to take the hope API for a test drive. Get comfortable. At that place are quite a few functions to cover, including ones that create, read, and update files and file metadata.

Working with files

Using file handles

The promise API provides ii different approaches to working with files.

The first approach uses a top-level ready of functions that accept file paths. These functions manage the lifecycle of file and directory resource handles internally. You don't need to worry about calling a close() office when you are done with the file or directory.

The second approach uses a set of functions available on a FileHandle object. A FileHandle acts as a reference to a file or directory on the file system. Here is how y'all can obtain a FileHandle object:

async function openFile(path) {     permit fileHandle;     try {         fileHandle = await fsPromises.open up(path, 'r');         console.log(`opened ${path}, file descriptor is ${fileHandle.fd}`);         const data = fileHandle.read()     } grab (err) {         panel.error(err.message);     } finally {         fileHandle?.shut();     } }        

On line 4 to a higher place, we employ fsPromises.open up() to create a FileHandle for a file. We laissez passer the r flag to indicate that the file should be opened in read-only mode. Any operations that try to modify the file volition neglect. (You tin can also specify other flags.)

The file'southward contents are read using the read() office, which is directly bachelor from the file handle object. On line 10, we need to explicitly close the file handle to avert potential retentivity leaks.

All of the functions available in the FileHandle class are too available as acme-level functions. Nosotros'll continue to explore meridian-level functions, but it is good to know that this approach is available as well.

Reading files

Reading a file seems like such a simple task. However, in that location are several dissimilar options that tin be specified depending on what you lot need to do with a file:

// example 1: simple read const data = await fsPromises.readFile(path);  // example ii: read a file that doesn't exist (creates a new file) const noData = await fsPromises.readFile(path, { flag: 'w'});  // case iii: read a file and render its contents every bit a base64-encoded string const base64data = wait fsPromises.readFile(path, { encoding: 'base64' });  // instance 4: read a file simply abort the performance before it completes const controller = new AbortController(); const { bespeak } = controller; const promise = fsPromises.readFile(path, { betoken: signal }); panel.log(`started reading file at ${path}`); controller.abort();      console.log('read functioning aborted before information technology could be completed') await hope;        

Case 1 is as simple as it gets, if all you desire to exercise is get the contents of a file.

In example two, we don't know if the file exists, so nosotros pass the w file system flag to create it kickoff, if necessary.

Example iii demonstrates how you alter the format of the information returned.

Example 4 demonstrates how to interrupt a file read functioning and abort it. This could be useful when reading files that are big or tiresome to read.

Copying files

The copyFile function tin can brand a copy of a file and requite you lot some control over what happens if the destination file exists already:

// example 1: create a copy, overwite the destination file if information technology exists already look fsPromises.copyFile('source.txt', 'dest.txt');  // instance 2: create a copy only fail because the destination file exists already await fsPromises.copyFile('source.txt', 'dest.txt', fs.constants.COPYFILE_EXCL); // Fault: EEXIST: file already exists, copyfile 'source.txt' -> 'dest.txt'        

Instance one will overwrite dest.txt if information technology exists already. In example 2, we laissez passer in the COPYFILE_EXCL flag to override the default behavior and fail if dest.txt exists already.

Writing files

There are three ways to write to a file:

• Suspend to a file
• Write to a file
• Truncate a file

Each of these functions helps to implement different utilize cases.

// example i: append to an existing file // content of information.txt earlier: 12345 await fsPromises.appendFile('information.txt', '67890'); // content of data.txt afterwards: 1234567890   // example ii: append to a file that doesn't exist yet await fsPromises.appendFile('data2.txt', '123'); // Error: ENOENT: no such file or directory, open 'data2.txt'  // example 3: write to an existing file // content of data3.txt before: 12345 await fsPromises.writeFile('data3.txt', '67890'); // content of data3.txt after: 67890  // instance 4: write to a file that doesn't be yet (new file is created) await fsPromises.writeFile('data4.txt', '12345');  // example 5: truncate information in an existing file // content of data5.txt before: 1234567890 await fsPromises.truncate('data5.txt', 5); // content of data5.txt later on: 12345        

Examples ane and 2 demonstrate how to use the appendFile function to append data to existing or new files. If a file doesn't exist, appendFile will create information technology first.

Examples 3 and iv demonstrate how to use the writeFile function to write to existing or new files. The writeFile part will also create a file if it doesn't be earlier writing to it. Yet, if the file already exists and contains data, the contents of the file is overwritten without alert.

Case 5 demonstrates how to apply the truncate function to trim the contents of a file. The arguments that are passed to this role can be confusing at first. You might expect a truncate part to have the number of characters to strip from the cease of the file, simply actually we need to specify the number of characters to retain. In the example above, yous can see that we entered a value of 5 to the truncate office, which removed the terminal 5 characters from the string 1234567890.

Watching files

The promise API provides a single, performant watch part that can lookout man a file for changes.

const abortController = new AbortController(); const { signal } = abortController; setTimeout(() => abortController.arrest(), 3000);  const watchEventAsyncIterator = fsPromises.sentry(path, { betoken });  setTimeout(() => {     fs.writeFileSync(path, 'new information');     panel.log(`modified ${path}`); }, 1000);  for look (const event of watchEventAsyncIterator) {     console.log(`'${event.eventType}' watch result was raised for ${outcome.filename}`); }  // console output: // modified ./data/watchTest.txt // 'alter' watch event was raised for watchTest.txt // sentinel on ./data/watchTest.txt aborted        

The sentinel office can watch a file for changes indefinitely. Each fourth dimension a alter is observed, a lookout man event is raised. The sentinel role returns an async iterable, which is essentially a way for the function to return an unbounded serial of promises. On line 12, we have reward of the for await … of syntactic sugar to await for and iterate each watch effect as it is received.

In that location is a good chance y'all don't want to endlessly watch a file for changes. The watch tin can be aborted by using a special signal object that can be triggered as required. On lines ane to 2, we create an case of AbortController, which gives us access to an instance of AbortSignal that is ultimately passed to the watch function. In this case, we telephone call the signal object's abort() role afterwards a fixed period of time (specified on line 3), simply you tin abort nevertheless and whenever you need to.

The lookout man part can as well be used to sentinel the contents of a directory. It accepts an optional recursive option that determines whether all subdirectories and files are watched.

File metadata

So far, we have focused on reading and modifying the contents of a file, but yous may also need to read and update a file'south metadata. File metadata includes its size, type, permissions, and other file system backdrop.

The stat role is used to retrieve file metadata, or "statistics" like file size, permissions, and ownership.

// go all file metadata const fileStats = await fsPromises.stat('file1.txt'); panel.log(fileStats) // console output: // Stats { //    dev: 2080, //    style: 33188, //    nlink: 1, //    uid: chiliad, //    gid: thou, //    rdev: 0, //    blksize: 4096, //    ino: 46735, //    size: 29, //    blocks: 8, //    atimeMs: 1630038059841.8247, //    mtimeMs: 1630038059841.8247, //    ctimeMs: 1630038059841.8247, //    birthtimeMs: 1630038059801.8247, //    atime: 2021-08-27T04:20:59.842Z, //    mtime: 2021-08-27T04:xx:59.842Z, //    ctime: 2021-08-27T04:20:59.842Z, //    birthtime: 2021-08-27T04:20:59.802Z //  } console.log(`size of file1.txt is ${fileStats.size}`);        

This case demonstrates the full list of metadata that tin can exist retrieved for a file or directory.

Continue in listen that some of this metadata is Os-dependent. For example, the uid and gid backdrop correspond the user and group owners — a concept that is applicative to Linux and macOS file systems, merely not Windows file systems. Zeroes are returned for these 2 backdrop when running in this function on Windows.

Some file metadata can be manipulated. For case, the utimes office is used to update the access and modification timestamps on a file:

const newAccessTime = new Appointment(2020,0,i); const newModificationTime = new Engagement(2020,0,1); wait fsPromises.utimes('test1.txt', newAccessTime, newModificationTime);        

The realpath office is useful for resolving relative paths and symbolic links to full paths:

// convert a relative path to a full path const realPath = wait fsPromises.realpath('./test1.txt'); console.log(realPath); // console output: /home/brian/test1.txt  // resolve the real path of a symbolic link pointing to /dwelling house/brian/test1.txt const symLinkRealPath = await fsPromises.realpath('./symlink1'); panel.log(symLinkRealPath); // console output: /habitation/brian/test1.txt        

File permissions and ownership

Please go along in listen every bit we go on in this section that file permission and buying functions are applicable to Unix, Linux and macOS operating systems. These functions yield unexpected results on Windows.

If you are non sure whether your awarding has the necessary permissions to access or execute files on the file organization, you tin utilize the access function to test it:

// case 1: check if a file can be accessed effort {   wait fsPromises.admission('test1.txt');   console.log('test1.txt can be accessed'); } grab (err) {   // EACCES: permission denied, access 'test1.txt' }  // case 2: cheque if a file tin can be executed (applies to Unix/Linux-based systems) try {   look fsPromises.access('test2.txt', fs.constants.X_OK); } take hold of(err) {   // EACCES: permission denied, access 'test2.txt' }        

File permissions can be modified using the chmod function. For instance, we can remove execute access from a file by passing a special mode string:

// remove all execute access from a file wait fsPromises.chmod('test1.txt', '00666');        

The 00666 style cord is a special five-digit number that is equanimous of multiple bit masks that draw file attributes including permissions. The last three digits are equivalent to the three-digit permission style you might exist used to passing to chmod on Linux. The fs module documentation provides a list of bit masks that can be used translate this fashion string.

File ownership tin besides be modified using the chown function:

// set up user and grouping ownership on a file const root_uid= 0; const root_gid = 0; expect fsPromises.chown('test1.txt', root_uid, root_gid);        

In this case, we update the file so that it is owned by the root user and root group. The uid of the root user and gid of the root grouping are always 0 on Linux.

Working with links

Tip: Link functions are applicable to Unix/Linux operating systems. These functions yield unexpected results on Windows.

The fs module provides a variety of functions you lot tin use to work with hard and soft, or symbolic, links. Many of the file functions we've already seen have equivalent versions for working with links. In most cases, they operate identically, too.

Before we start creating links, let'southward have a quick refresher on the two types of links we'll be working with.

Hard vs. soft links

Hard and soft links are special types of files that point to other files on the file system. A soft link becomes invalid if the file it is linked to is deleted.

On the other paw, a hard link pointing to a file will however exist valid and contain the file'southward contents fifty-fifty if the original file is deleted. Hard links don't point to a file, merely rather a file'south underlying data. This data is referred to as the inode on Unix/Linux file systems.

We tin easily create soft and hard links with the fs module. Employ the symlink office to create soft links and the link function to create hard links.

// create a soft link const softLink = await fsPromises.symlink('file.txt', 'softLinkedFile.txt');  // create a difficult link const hardLink = await fsPromises.link('file.txt', 'hardLinkedFile.txt');        

What if you want to determine the underlying file that a link points to? This is where the readlink role comes in.

>// read a soft link panel.log(await fsPromises.readlink('softLinkedFile.txt')); // output: file.txt  // read a difficult link... and fail console.log(look fsPromises.readLink('hardLinkedFile.txt')); // output: EINVAL: invalid argument, readlink 'hardLinkedFile.txt'        

The readlink function can read soft links, but not hard links. A hard link is duplicate from the original file it links to. In fact, all files are technically hard links. The readlink function essentially sees it as just another regular file and throws an EINVAL fault.

The unlink function tin can remove both hard and soft links:

// delete a soft link await fsPromises.unlink('softLinkedFile.txt');  // delete a hard link / file await fsPromises.unlink('hardLinkedFile.txt');        

The unlink function really serves as a general purpose function that can also be used to delete regular files, since they are essentially the same as a difficult link. Aside from the link and unlink functions, all other link functions are intended to be used with soft links.

You tin modify a soft link's metadata much similar y'all would a normal file's:

// view soft link meta information const linkStats = await fsPromises.lstat(path);  // update access and modify timestamps on a soft link const newAccessTime = new Date(2020,0,1); const newModifyTime = new Date(2020,0,1); await fsPromises.lutimes('softLinkedFile.txt', newAccessTime, newModifyTime);  // remove all execute access from a soft link expect fsPromises.lchmod('softLinkedFile.txt', '00666');  // set user and group buying on a soft link const root_uid= 0; const root_gid = 0; expect fsPromises.lchown('softLinkedFile.txt', root_uid, root_gid);        

Bated from each part being prefixed with an l, these functions operate identically to their equivalent file functions.

Working with directories

We can't but terminate at file processing. If you are working with files, it is inevitable that yous will need to piece of work with directories likewise. The fs module provides a diversity of functions for creating, modifying, and deleting directories.

Much like the open up function we saw earlier, the opendir function returns a handle to a directory in the form of a Dir object. The Dir object exposes several functions that can exist used to operate on that directory:

allow dir; try {   dir = await fsPromises.opendir('sampleDir');   dirents = await dir.read(); } take hold of (err) {   console.log(err); } finally {   dir.close(); }        

Be sure to call the close part to release the handle on the directory when you are done with it.

The fs module also includes functions that hide the opening and closing of directory resources handles for you. For instance, you can create, rename, and delete directories:

// example i: create a directory await fsPromises.mkdir('sampleDir');  // example ii: create multiple nested directories await fsPromises.mkdir('nested1/nested2/nested3', { recursive: true });  // example 3: rename a directory await fsPromises.rename('sampleDir', 'sampleDirRenamed');  // example 4: remove a directory await fsPromises.rmdir('sampleDirRenamed');  // example 5: remove a directory tree await fsPromises.rm('nested1', { recursive: true });  // example 6: remove a directory tree, ignore errors if it doesn't exist await fsPromises.rm('nested1', { recursive: true, forcefulness: true });        

Examples two, 5, and 6 demonstrate the recursive option, which is especially helpful if you lot don't know if a path volition exist before creating or deleting it.

At that place are two options to read the contents of a directory. By default, the readdir function returns a list of the names of all the files and folders direct below the requested directory.

Yous can laissez passer the withFileTypes choice to get a list of Dirent directory entry objects instead. These objects contain the proper noun and type of each file arrangement object in the requested directory. For example:

// example one: get names of files and directories const files = wait fsPromises.readdir('anotherDir'); for (const file in files) {   console.log(file); }  // example 2: become files and directories as 'Dirent' directory entry objects const dirents = await fsPromises.readdir('anotherDir', {withFileTypes: true}); for (const entry in dirents) {   if (entry.isFile()) {     panel.log(`file name: ${entry.name}`);   } else if (entry.isDirectory()) {     panel.log(`directory name: ${entry.name}`);   } else if (entry.isSymbolicLink()) {     console.log(`symbolic link name: ${entry.proper name}`);   } }        

The readdir role does not provide a recursive option to read the contents of sub-directories. You'll have to write your ain recursive function or rely on a third-party module like recursive-readdir]().

Close()

It'southward time to close() the resource handle for this article. We have taken a thorough expect at how to work with files, links, and directories using the Node.js fs module. File processing is available in Node.js out of the box, fully featured and prepare to apply.

200's only Monitor failed and wearisome network requests in production

Deploying a Node-based web app or website is the easy part. Making sure your Node example continues to serve resources to your app is where things get tougher. If y'all're interested in ensuring requests to the backend or 3rd party services are successful, try LogRocket. https://logrocket.com/signup/

LogRocket is similar a DVR for web and mobile apps, recording literally everything that happens while a user interacts with your app. Instead of guessing why problems happen, y'all can aggregate and report on problematic network requests to quickly understand the root cause.

LogRocket instruments your app to record baseline operation timings such as folio load time, time to first byte, tedious network requests, and also logs Redux, NgRx, and Vuex actions/country. First monitoring for gratis.

ortizitell1965.blogspot.com

Source: https://blog.logrocket.com/file-processing-node-js-comprehensive-guide/

Share this post