What is a monorepo?
This article will cover monorepos that maintain Node.js-based projects. A monorepo is a single repository that manages multiple projects. Google, Facebook, and Microsoft (the founder of this article’s proposed solution) have taken the leap toward monorepo architecture to battle massive code and commit volume. These projects can be isolated components or utilities that are published to package repositories or full-blown framework-based projects like Next.js, Vue, and Angular applications.
The traditional multirepo model, where multiple VCS repositories are used to maintain isolated codebases, often starts as a simple solution. This approach can quickly become cumbersome to work with as codebases grow in size and complexity or as the number of code repositories becomes unmanageable. This is where a monorepo solution would excel.
Consider the case where a development team is working on a single web application, but they now need to share code for consumption by another team. These might be core utilities or even smart components. The team successfully factors code into a separate repository, complete with a CI/CD system that can publish packages. PR reviews are now split across repositories. A developer working on the main web application now needs to wait for a new package to be published so that they can use their own library in the next web release. This problem is compounded as the number of these repositories grows.
In a monorepo solution,
- There is already a package publishing system in place;
- Code that must change in a package is easily unit tested along with the consuming application(s);
- A package release is not mandatory to start using adjusted packaged code in other projects, and
- The code author adjusting a factored package can now be easily responsible for ensuring that their changes don’t disrupt related projects.
Key Benefits
Modular Refactoring
We’ve all worked on projects that have tightly coupled, monolithically architectured implementations. One major benefit of a monorepo solution is that it can provide a healthy starting point for modularizing your application.
Is there a well-designed component styling solution for your React application from which other teams could benefit? Would having the styling factored out make styling changes easier by isolating its code in a single package? Factor out your styling into a package. Do you have a linting configuration that could be propagated throughout your organization? Do you maintain an analytics implementation that could be consumed by multiple third parties?
When the proper tooling is in place, monorepos make it easy to start publishing packages that would otherwise require heavy lifting to coordinate into a separate repository and configure CI/CD processes. When designed properly, publishing a new package for the first time can be as easy as setting ”shouldPublish”: false to ”shouldPublish”: true.
Code Reuse
Internal utilities or implementations are commonly duplicated across different projects and repositories, especially when a packaging solution is not immediately available. A monorepo approach makes it possible to factor these into shared packages without the need to publish and bump these references down a long chain of dependencies. In particular, Rush can make use of a variety of versioning policies to ease the strain of cascading version bumps.
Dependency Management
A transitive dependency is any dependency in the dependency tree that is pulled in by direct dependencies and is not specified as a direct dependency in a project’s package.json. When our Node applications grow, it is not uncommon to see transitive dependencies in the order of thousands. Install times can be reduced by caching shared dependencies across all monorepos. By virtue of PNPM or Yarn workspaces, Rush can:
- Leverage a common dependencies folder to look at when multiple projects share the same dependency, and
- Help enforce consistent versioning across direct dependencies for all projects in a monorepo.
Why Rush?
Rush, a monorepo solution created by the platform team for Microsoft SharePoint, enters the conversation as a relatively underutilized solution when it comes to monorepo management. Tools such as Lerna have become mainstream, but there are growing concerns around its longevity as a project, given that a single developer has been contributing most of its code over the last three years. Consumers have been wondering if PNPM support will be introduced since 2018! It may not be introduced soon, given the project’s health, even though it has well-documented speed improvements over NPM and Yarn.
Package Manager Agnosticism
Web developers and architects alike enjoy having options when it comes to any solution. Package management is no different in this regard—the three most popular package managers today are:
- NPM
- Yarn
- PNPM
Rush supports each of these agnostically, although each package manager’s performance impacts Rush itself. Yarn and PNPM work especially well with Rush because they both support workspaces, which allows the package manager to install and collect dependencies for all projects in a single pass. PNPM is a particularly interesting option with Rush, as it provides extra safety around phantom dependencies and the NPM doppelgangers issue.
Build Caching
Different transpilation, linting, and bundling tools (Typescript, ESLint, Webpack, Rollup, etc.) often have their own caching layer to reduce incremental build times. Rush takes this a step further with their own build caching solution.
With build caching, Rush will create a tarball of each project’s build output. When subsequent builds occur, Rush can restore this build output instead of rebuilding the project, skipping the overhead that a no-op incremental build would have incurred.
This feature becomes especially powerful when combined with remote storage integration. Rush can leverage an Amazon S3 or Azure blob storage container to cache builds across any machine with the appropriate credentials. In such a configuration, developers would be able to pull PRs or branches that were built by CI and run code with practically zero build time.
Note that build caching with Rush is experimental as of the time of writing.
Consistent Dependency Versioning
When working with many dependencies spread across multiple projects, it can be helpful to ensure that only one version of a package is in use within any given project, which implies the following:
- Developers can safely assume that, when using lodash, for example, in one project, the API won’t change when working with lodash in a different project;
- When different projects consume factored out utilities, peer dependencies, or external module resolutions, it is ensured that version mismatches will not cause unforeseen issues, and
- The number of transitive dependencies in a monorepo can be reduced to improve install times.
Rush can enforce consistent versioning when performing dependency installations or additions.
Automatic Changelog Generation
With Rush, it becomes easy to communicate and track changes across publishable packages. When publishable packages undergo source file changes on a branch separate from the main branch, they are flagged by the rush change --verify command. This instructs the developer to run rush change, if they haven’t already, and for each modified package this utility guides them through:
- Writing a descriptive message detailing the changes made, and
- Assigning a version bump type that should occur based on these changes (major, minor, or patch).
This is easily integrated into a PR trigger to ensure that no changes go undocumented. When the next package is triggered, each project gets an up-to-date CHANGELOG.md file which compiles a listing of tracked changes.
Active Support Channels
Rush has an active Zulip chat room where questions and concerns that may not be covered in documentation or GitHub issues can be addressed often more quickly than one may expect of other monorepo tooling teams.
Migrating to Rush
Following from a minimal Rush example provided by the Rush Stack team, a standard directory structure may look as such:
│── apps
│ └── my-app
│ ├── package.json
│ └── src
│── common
│ │── config
│ │ └── rush
│ │── git-hooks
│ │ └── commit-msg.sample
│ │── scripts
│ └── ...
│── libraries
│ └── my-controls
│ ├── lib
│ ├── package.json
│ └── src
├── LICENSE
├── README.md
├── rush.json
└── tools
└── my-toolchain
├── package.json
└── src- apps may contain deployable but non-publishable projects like a web frontend or a GraphQL server.
- common is a Rush-owned directory for maintaining configuration, resolved dependencies and lockfiles, and utility scripts.
- tools may contain toolchain configuration that you would like to reuse across multiple projects that use similar build or bundling systems.
- libraries may contain your packages publishable or used internally.
- rush.json at the repository root contains the core Rush configuration—everything from the version of Rush being used, to the type and version of the underlying package manager, to the listing of projects that you wish to publish as packages, and much more.
This directory structure separates concerns in a comfortable manner. Note that each project within the monorepo has a familiar structure as well: src for source code, a package.json for specifying dependencies, and potentially a lib directory for build outputs.
Once configured, moving an existing web application into a Rush directory structure may be as simple as dropping tracked files into apps/<name-of-your-app>. From there, you can consider what is useful to share or reuse as the monorepo grows and you begin populating the libraries directory.
Conclusion
Monorepo solutions serve as a step toward a more collaborative development experience. Code reuse becomes easy. Dependency management is safer. This shouldn’t be the default for all Node projects (especially smaller ones), but if you need it, it’s a lifesaver.
From the rush --help command itself:
Rush makes life easier for JavaScript developers who develop, build and publish many packages from a central Git repo. It is designed to handle very large repositories supporting many projects and people. Rush provides policies, protections, and customizations that help coordinate teams and safely onboard new contributors. Rush also generates changelogs and automates package publishing. It can manage decoupled subsets of projects with different release and versioning strategies. A full API is included to facilitate integration with other automation tools. If you are looking for a proven turnkey solution for monorepo management, Rush is for you.
If you’re interested in learning how a monorepo solution could improve your team’s workflow, read the Rush Stack documentation for more information. They have guides geared towards developers and repository maintainers.
