defiler/GUIDE.md

Concepts

Files

Defiler's concept of a file is something that can come from one of two places: a physical file on the disk, or a virtual file that is generated by your code. These two types of files differ very slightly in how they are treated, but for the most part Defiler handles them both the same.

Files have a path property containing the relative path to the file, as well as dir, filename, and ext properties containing portions of the path. All of these can be updated and will keep the others in sync.

For physical files, the stats property contains the fs.Stats object retrieved for the original file.

Files also have text and bytes properties, containing string and Buffer representations of the file's contents. Either can be updated and will keep the other in sync. (You shouldn't mutate bytes, only reassign it.) The enc property specifies the encoding to be used when converting between text and bytes, and can be changed.

See the API docs for more information.

The transform

Every file (physical and virtual) is run through the transform function you register with Defiler. The transform mutates the object representing the file in-place, and returns a promise indicating when it's finished.

The transform is called, for each file, with an object containing file (the File instance) and event (a string representing why this file is being run through the transform, see The event field). The transform should mutate the file object as it sees fit. It can also take whichever actions it wishes based on the file (including, for example, writing output to disk). Files' paths can be changed as they're transformed, but the main way to refer to them from other files will continue to be by their original paths (see Dependence).

The event field

The event passed to the transform is a string that can be one of four values:

• 'read' - This indicates that the file was just read in from the disk
• 'add' - This indicates that the file was just manually added (see Virtual files)
• 'delete' - This indicates that the file was just deleted from the disk. In this case, the file object will be the transformed version of the file that was just deleted, and not an untransformed file for you to transform
• 'retransform' - This indicates that the file was not changed, but that it is being re-transformed because one of its dependencies changed

At the very least, your transform should probably check whether event is 'delete', because in this case file isn't a file to transform, but is instead the fully-transformed version of the file that was just deleted.

Dependence

Files can be made to depend on other files, so that changes to a dependency cause the dependent to be re-transformed. For physical files, the file does not need to be re-read from the disk before it can be re-transformed, as the original version is kept in memory. When a file is being re-transformed because one of its dependencies changed, the transform will be called with the read, added, and deleted flags all set to false.

The defiler.get(path) method, when used inside the transform, lets you depend on and retrieve other transformed files. It should be passed the original path of a file, and will return a Promise resolving to the transformed File instance. If the requested file does not exist (or if you have a deadlock via a system of mutually-depending files, none of which will continue transforming until another one finishes), the Promise will resolve to undefined.

The defiler.get method can also be passed an array of (original) paths, in which case it will return a Promise resolving to an array of File instances (or undefineds).

It can also be passed a filter function, which should accept a path and return a boolean. In this case defiler.get will return a Promise resolving to an array of all File instances whose (original) paths match the filter.

See the API docs for more information.

Virtual files

During the transform's processing of a file, you can also create virtual files, which don't directly correspond one-to-one with physical files. The Defiler instance has a defiler.add(file) method, which you can pass a File instance to (or, more commonly, a POJO, which will be turned into a File). The virtual file will run through the transform and will thereafter be treated pretty much like a physical file. In particular, you can make other files depend on it with defiler.get(path).

Since Defiler has no way of knowing which virtual files will be created from transforming which files, when defiler.get is used to request a file that doesn't exist yet, Defiler waits until it does. Requesting a file that's never going to exist would cause a deadlock, so Defiler resolves this as a generalization of the above-mentioned deadlock resolution: If it ever happens that every in-process action is waiting for some other transformed file to exist, Defiler will resolve each of those pending Promises returned by defiler.get to undefined.

Similarly when retrieving files by a filter, there's no way to know which files will eventually exist. So Defiler will wait until all of the in-progress actions depend on filter responses or other in-progress files, at which point Defiler will resolve each pending filter Promise to the array of matching files that have been already completed in that moment.

Generators

Generators are an independent way of interacting with the Defiler instance, for things that do not fit well into the main transform. Each generator is a function that will be called without arguments. A generator would typically call defiler.get and/or defiler.add to retrieve dependencies and write new virtual files. Automatic dependence handling also works here, so when one of the files retrieved by defiler.get changes, that generator will be re-run.

It's beneficial to write multiple smaller generators, rather than a single large one. This helps ensure that unneeded recalculation is not done when a file changes.

Waves

Processing in Defiler happens in waves, sort of. During the first wave, all files have the transform run on them, all generators are run, and any virtual files added with defiler.add are also transformed. It's at the end of this first wave that the Promise returned by defiler.exec() resolves. Subsequently, each change to a watched file results in another wave, during which all dependent files are re-transformed, all dependent generators are re-run, and all added virtual files are re-transformed. If any watched file changes come in while a wave is still active, a new wave will be started immediately upon the completion of the previous one. Otherwise, a new wave will be started when a new file change event comes in.

It's only during the first wave that the special deadlock resolution behavior is relevant. On subsequent waves, any missing files requested via defiler.get(path) will return (a Promise resolving to) undefined. This does, however, establish a dependence relation between the two, and if the missing file ever does exist later (either as a physical or virtual file), the dependent file/generator will be re-transformed/re-run.

When performing production builds, you probably only want to have a first wave, and to not watch for subsequent file changes. This can be achieved with the watch option to the Defiler constructor.

Usage

First, create a new Defiler instance, initializing it with the directories to watch, the transform, and (optionally) the generators, a path resolver, and/or an error handler.

import { Defiler } from 'defiler';

const defiler = new Defiler({
	{ dir: '/path/to/input/directory' },
	{ dir: '/path/to/another/input/directory' },
	{
		transform: async ({ file, event }) => {
			// transform the file
		},
		generators: [
			async () => {
				// generator 1
			},
			async () => {
				// generator 2
			},
		],
		resolver: (base, path) => {
			// return path resolved from base
		},
		onerror: (error) => {
			// handle the error / abort the build / etc.
		},
	},
});

Then, call its exec() method to set everything in motion. This returns a Promise that will resolve when the initial wave of processing has completed.

await defiler.exec();

Useful things available on the Defiler instance for you to use in the transform, in the generators, or elsewhere are:

• defiler.paths - a Set of the paths of all of the physical files
• defiler.files - a Map of original paths to the transformed File instances
• defiler.get(path) - a method to retrieve one or more transformed Files based on their original paths
• defiler.add(file) - a method to add a virtual file, which is then transformed like a physical one is, and which can be depended on by other files
• defiler.resolve(path) - a method to resolve paths in defiler.get and defiler.add when these are called from the transform, using your resolver function

In closing

See the API docs for more information, and for a couple of other things not covered here.