Tue May 11 2021

Making a release toolkit - Part 3: ES modules

In my last post I talked about ES module incompatibility with jest, let's dive deeper into the land of modules!

Background

First some basic terminology: At the risk of oversimplification, a module is a JavaScript file, with a way to encapsulate code and expose it to other files. A major benefit is scoping, to avoid the need for global variables that can lead to confusing and buggy code.

For years there have been two syntaxes for writing JavaScript modules. Node has always used modules CommonJS modules, which use require() and module.exports to refer to other modules. ES module syntax uses the import and export keywords to share code with other modules. It was widely adopted thanks to tools like Babel, which transforms ES module syntax into more commonly understood JS (both in the browser and Node).

If syntax were the only difference, the two might have lived a long and happy life together but, well... it's complicated because ES modules operate fundamentally differently than CommonJS. Let me tell you about it.

1. Sync vs. Async

The first big difference: ES modules are loaded in an asynchronous three-step process: they're parsed, instantiated, and executed separately. A static dependency tree/graph is built before executing a single line of code.This way it can know how the code is structured and how to allocate memory.

CommonJS is evaluated and executed synchronously all in one go. It doesn't have a way to know the dependency graph before it starts running code, it evaluates and executes code as it's building the graph.

This has a few implications, which leads us to...

2. How shared variables are stored

When something is exported from a CommonJS module, the import and export are copies of each other, stored in different places in memory. In ES modules the import and export share the same memory location.

Among other things, this means each system handles cyclic dependencies differently - in short: ES modules can handle them, CommonJS can't.

Picture this:

// A.js import {second} from './B'; // (1) console.log(second); // Prints "second" export const first = 'first'; // (4) // B.js import {first} from './A'; // (2) - Memory is allocated, will be filled with export when (4) runs console.log(first); // Prints "first" export const second = 'second'; // (3)

In ES modules when A imports second from B, if B also imports first from A it will allocate a location in memory for first, and when parsing reaches the first export in A it will put it in that memory location.

// A.js const {second} = require('./B'); // (1) console.log(second); // Prints "second" exports.first = 'first'; // (4) // B.js const {first} = require('./A'); // (2) - exports.first has not yet been defined! console.log(first); // Prints "undefined" exports.second = 'second'; // (3)

In CommonJS, as execution makes its way from A to B, since the code hasn't reached first when it's importing, the import will be undefined. Then when the code reaches the export (if it doesn't error first), the export goes into a different place in memory and doesn't update the undefined import. Womp womp.

3. Dynamic module references

In CommonJS you can call require() wherever you want. It's just a function:

This will not fly in ES modules, import is a keyword and must be at the top level of the module.

Actually that's not exactly true anymore. Dynamic imports enable this, but they're asynchronous so they introduce promises into the flow:

let feeling; if (iceCream) { import('./happy').then(module => { feeling = module.default; }); } else { import('./sad').then(module => { feeling = module.default; }); }

Not the end of the world, but it can mean a lot of refactoring if you didn't write the code with promises in mind.

Back to the task at hand. This all came up because I decided to enable ES modules in Node. There has been a big push lately to make npm packages (most of which were built in CommonJS) compatible with ES modules. Maintainers are making their modules compatibile, and skypack is at the forefront of tools designed to ease compatibility woes.

One tool that relies directly on the CommonJS structure is jest, one of the most widely used JS testing frameworks. During this side project, I discovered that jest's mocking capability isn't yet working with ES modules, and it will likely never work in its current form since it relies on overriding CommonJS's require() with a version that replaces exports with mocks. If you've used jest.mock with import/export before, you've likely had to include babel-jest, which transforms the code behind the scenes.

Mocking in `jest` (very funny)

"Mock" is a loaded term, but in jest a mock can act as either:

A spy, which attaches to a function and allows you to inspect it after the code under test has run, doing something like expect(function).toBeCalledWith('foo').
A stub, which also attaches to a function and allows for inspection, but also overrides its behavior so you can, for example, return a specific value: mockFn.mockReturnValue('foo')

Module mocking is when you add mocks to every export in a module: jest.mock('./happy'). This is handy when you don't want to inject dependencies, or when you want to mock something nested deeply where you would need to inject through multiple layers.

I was writing this article I realized I don't know exactly how jest module mocking works - I figured it was somehow hooking into the require() cache. After some debugging and reading, I discovered that's basically what's going on. It's overriding the cache with its own module registry, but also hooking into the global require() function itself and overriding it with requireModuleOrMock. If there's a jest.mock in the file, it's going to replace all the module's exports with mocked functions.

Alas, jest can't hook into import in the same way, and so here we are injecting all our dependencies instead of mocking.

jest has this GitHub issue where you can track progress on ES module support, and this PR for tracking jest.mock progress. I love this about open source, you can see exactly how far along something is, and if you're able, help move it along yourself!

In short, it looks like something like jest.mockImport will be added to the API, and may require (heh) dynamic import syntax.

Learnings:

CommonJS and ES modules work very differently, more than just syntax

jest will probably add to the API rather than make the current jest.mock work for both CommonJS and ES modules

For more in-depth discussion of ES modules, Lin Clark has a great cartoon deep dive on how they work, and here's a closer look at the incompatibility between ES modules and CommonJS.

I hope you enjoyed this detour. I found the differences between CommonJS and ES modules confusing for a long time, so I hope this explanation has been helpful, or at least not made things more confusing. Next time we'll get back to work on the release toolkit, see you then!

<< Previous Next >>