Reliable Development Workflows for Local Node Dependencies (aka, Why We Wrote npm-pack-here)

So Many Modules

Here at Tableau we do a lot of web development, for our pure browser-based offerings (Tableau Public, Online, and Server), the Electron-based Tableau Prep, the native phone applications, and also for pieces of the desktop product’s UI. The code for these applications is split across a number of repositories. So in order to reuse code between them, shared TypeScript code is written in independently built and versioned GitLab projects. We have seen a steady growth of these modules over time, primarily because Web components have become the standard for writing reusable UI across our applications.

With hundreds of modules it is very common to end up with the applications depending on modules, depending on modules … When a Tableau developer works on one of these modules they used to run into issues when testing locally within another module (or the root application). To solve these local development issues in a straightforward, cross-platform, and consistent way, we wrote the tool npm-pack-here

npm-pack-here is publicly available for use and remixing! Check it out and let us know if you run into trouble by creating an issue. If you found it helpful, give the GitHub repository a ⭐️.

A Pain Point During Multi-Module Development

Say a developer is working on the edit-axis-dialog and they want to update the drop-down controls to handle some newly discovered edge case. The drop-down control is written within a separate widgets project (developed in partnership with our user experience team in order to have consistent styling, behavior, and accessibility support across our web products). This widgets project exports common React components (like buttons, text-boxes, and drop-downs) for use within the ever-growing list of UI projects.

In order to make the desired change to the drop-down control and test it out, this developer would perform the following steps:

1. Clone the widgets project locally and make the change (iterating quickly using TypeScript compile watch and test watch commands)
2. /build/css/style.css
3. Clone the edit-axis-dialog project locally
4. Using  yarn link or  npm link, set up a link between the projects so that their local version of edit-axis-dialog is using their local version of widgets(with the drop-down control changes)

At this point, when compiling edit-axis-dialog, it was super common to run into a TypeScript compile time error complaining about some React type incompatibilities. For example,

No overload matches this call. Overload 1 of 2, '(props: Readonly<DropdownProps>): DropdownWidget', gave the following error.Type 'React.CSSProperties' is not assignable to type 'import("/widgets/node_modules/@types/react/index").CSSProperties'....Overload 2 of 2, '(props: DropdownProps, context?: any): DropdownWidget', gave the following error.Type 'React.CSSProperties' is not assignable to type 'import("D:/gitlab/widgets/node_modules/@types/react/index").CSSProperties'.

😢

The root cause of this issue is that when using yarn link or npm link to do local testing, two semi-disconnected node_modules directory trees are present and the compiler ends up using two different React TypeScript type definitions that it cannot match against one another.

Are you making a Tableau extension and want the same look and feel as Tableau? Check out the public version of Widgets called Tableau UI.

The Problem in Detail

Until the as-of-yet-not-released React v17 is available, two versions of React cannot be shipped within an application. So, all of our modules are set up to list React in their peerDependencies in order to make sure that we only use a single React version in our applications (the version specified by the application). This leads to some pretty brutal pain when upgrading React to a new major version, but that is a story for another day …

Before running a link command, the edit-axis-dialog's node_modules directory looks something like this (abbreviated to be only the packages that matter for this example):

edit-axis-dialog

node_modules

widgets

@types

react

react

The widgets package includes the compiled TypeScript source code as JavaScript and the corresponding TypeScript type definitions.widgets assumes that any TypeScript module or application (edit-axis-dialog in this case) using widgets React components will also add the React TypesScript types to its devDependencies. So,edit-axis-dialog specifies widgets in its dependencies and the React types in its devDependencies.edit-axis-dialog's source then defines some React components that use the widgets components. When TypeScript finally compiles the edit-axis-dialog, all of the React components will end up referencing the same React TypeScript type definitions (the ones specified in the edit-axis-dialog's devDependencies).

As an aside, for those familiar with Node’s dependency declaration system, you might ask why in widgets we do not specify the React types in itsdependencies orpeerDependencies. Many open source projects do exactly this but in practice we found that it was confusing since the type dependency is not a runtime dependency requirement (only a compile time TypeScript requirement). So, we typically leave the React types unlisted (as people will see a compile time error if they don’t declare the React types in their devDependencies, prompting them to include them).

After linking widgets into& edit-axis-dialog,& dit-axis-dialog's node_modules directory ends up looking slightly different:

edit-axis-dialog node_modules widgets <symlink to local widgets directory> @types react react

Where the locally cloned widgets& directory also has its own node_modules& directory tree (since it specifies devDependencies on React and the React types in order to be able to compile and test itself):

widgets node_modules @types react react

The consequence of these two node_modules trees is that the widgets devDependencies are also present (when they normally would not be). Meaning that when the edit-axis-dialog is compiled, its source files will reference its own React types while the widgets exported components will end up referencing their own React types. We have found that the React types are complex enough that TypeScript is unable to match the types against another version of the same types (using its duck typing interface matching capabilities) even when the two versions of the React types are identical!

Initial Workarounds

A naive solution would be to just delete the widgets/node_modules directory, if it doesn’t have its own React types then it will use the edit-axis-dialog’s, right? Unfortunately, this doesn’t work because symlinks are unidirectional. When the Node require algorithm runs it doesn’t know that it jumped through a symlink when navigating back up the file tree. This results in a different TypeScript compile error, “cannot find types for React” at the source locations that import widgets React components.

What if instead of linking, the developer used a file path dependency reference? With this configuration yarn install (or npm install) will copy over the local widgets& directory into the edit-axis-dialog’s node_modules directory tree. This also does not work, since arn and npm will do a direct copy without cleaning up the copied idgets’ node_modules directory — so it will still include all of widgets’ devDependencies, bringing in the second set of React types.

What if we combined the two approaches above? Delete the node_modules directory AND use a file path dependency reference? This approach actually works! After running install in edit-axis-dialog, yarn/npm will copy over the widgets's project directory (now without any contained node_modules directory) and then install any dependencies that are specific to widgets into the copied directory. But this approach has an undesired side effect — every time a change is needed in widgets the developer has to run the install command, make the change, build, and then delete the node_modules directory. After all of that is done, they still need to rerun the install command from within edit-axis-dialog to pull in the changes. This routine gets exceedingly tedious when attempting to rapidly iterate.

Okay what if the developer just simulates what yarn and npm do when publishing by using npm pack(or yarn pack) and then references the packed version of their project using afile path dependency reference? This also works as expected, it also allows for more rapid iteration than the previous approach because it removes the need to delete the node_modules directory in widgets and reinstall every time a change is needed. If& edit-axis-dialog is configured in this way, then to see an update to widgets inside of edit-axis-dialog the developer would, make their change, build, and run npm pack from within widgets. Then to pull that change into edit-axis-dialog they would run the install command from within edit-axis-dialog.

This last approach works pretty well, and many developers at Tableau used this method for years. But it requires intimate knowledge of how yarn/npm install dependencies and how the Node require algorithm works. With a large development organization knowledge ubiquity is not usually possible. More often than not we would see developers flounder trying to use one of the initial workarounds, or get frustrated and just publish their work in progress changes (sometimes using a pre-release version number, if they knew how). A group of us observing this confusion and slow cycle times, asked the question, how can we automate this in a straightforward, fast, and predictable way?

Our Solution

Introducing npm-pack-here, a Node script that automates the last approach listed above. It usesnpm-packlist to get the list of files npm would pack and then efficiently copies over only those files to another project’s node_modules directory. We designed npm-pack-here to perform the minimal necessary file copies in a way that is friendly to bundler watch scripts. Also it supports its own watch mode, so can be started once and then will continually copy over any changes.

So how would it work in the above example?

The developer making a change to widgets would do the following actions:

1. Clone the widgets project locally and make the desired drop-down control change
2. Clone the& edit-axis-dialog project locally
3. From within edit-axis-dialog& run npm-pack-here< --target <path to their local widgets director/code>y>
4. Using yarn or npm to set up a file path dependency reference to ensure the node_modules directory has any dependency changes specified by the local widgets

At this point the developer’s local version of widgets will have been copied into& edit-axis-dialog’s& node_modules directory. If the developer makes further changes to widgets they can rerun pm-pack-here--target <path to their local widgets directory >to pull those new changes in or start the watch command npm-pack-here watch -target <path to their local widgets directory >to continually pull in any changes.

When to Use npm-pack-here

While the example problem described above is specific to TypeScript, Tableau developers have seen similar problems occur in other non-TypeScript contexts. It usually occurs when the project being linked in specifies peerDependencies or expects some of its dependencies to be “lifted” (moved up the node_modules directory tree to become a shared dependencies).

We recommend only using& yarn link(or npm link) if the project being linked in does not have any of its own dependencies. If it does have dependencies, it is safer to usenpm-pack-here. This will ensure that the dependencies resolve in the exact same way as they would if the project’s package was downloaded from the binary repository.

Prior Work and Some Alternative Solutions

The idea for npm-pack-here was inspired by  yalc. yalc can also be used to solve the described problem but we found that it did a bit too much magic. The folks who tried it out found that it took months to use the tool without having to reference the documentation. Since we only needed to solve the problem of “put the packed version of this project in this other project’snode_modules directory,”npm-pack-here forgoes yarn workspace support (which yalc does support). Also the majority of our developers are already familiar with yarn and npm so we keep those commands separate (less magic). Finally, the name of yalc is super unfortunate when one uses& yarn all day; people reported mistyping it daily (if not hourly).

Many people upon first hearing about npm-pack-here ask “Why not use yarn plug’n’play?” (aka yarn v2). For those unfamiliar,yarn plug’n’play is an ambitious solution to the problem “yarn install takes forever…”. It solves the dependency problem by creating a JavaScript file that exports the dependency tree for a given project, referencing the downloaded packages in its global cache in order to cut down on file copies. With this approach referencing a local package is fundamentally the same as referencing a package being downloaded to the global cache so this whole class of problems no longer exists. Unfortunately, TypeScript support is not there yet. So, to use plug’n’play with most TypeScript IDEs requires a developer to set up and maintain workarounds. While using this approach might make sense in some circumstances, with hundreds of developers and a many year-old codebase, we opted to not take on the uncertainty.

The Node team also had a competing solution to yarn’s plug’n’play, called tink. Unfortunately, they decided to discontinue this work.

The last alternative solution that we considered (but have not had time to really dig into yet) is pnpm, which solves the node_modules dependency problem through extensive use of symlinks and clever node_modules directory structures. It is similar to yarn plug’n’play because it always references packages in the global cache using symlinked directories in a project’s node_modules directory. Downloaded dependencies are treated the same as local file system dependencies, so it potentially could resolve this whole class of issues. We have plans to further investigate if this solution will work better for us than what we do currently.

Reduce Your Multi-Module Development Pain

Tableau developers have been using npm-pack-here successfully for over a year, now you can use it too! We hope that it will also reduce your own module development pains. Cheers 😄