Better Express Routing & Architecture for NodeJS
I have been working on the backend with NodeJS for a while now as both a hobbyist and as a professional. Thus, I've been exposed to both open-source code as well as enterprise level Javascript code for NodeJS. Last year I led the rearchitecture for one of Intuit's products from .NET to NodeJS, which you can read about here. For those of you who have worked in the NodeJS environment, you may also have had some experience with some of their web frameworks; Hapi, Koa, and Express are some examples. No matter what web framework you are familiar with, I've seen some common anti-patterns that I think should be stopped. So for this post, I'm going to talk about how to write a more modular, extensible, testable, and scalable implementation of your routing. Whether you're writing an API or the backend for your front-end, you may find this useful.
Given that Express is one of the most popular web frameworks for NodeJS, I am going to use this as the main basis for this post. Even though this post will be using Express for its examples, these design methodologies can be applied to any other web framework.
So what are the problems
I wouldn't necessarily call this a problem, but these web frameworks are minimalistic by nature. This is done to allow developers the flexibility to implement their own design and architecture around these frameworks. This is great (in theory....)! However, because there is no "opinionated" methodology on how to organize or set up your framework or routes, one of the unintended outcomes is that developers will simply design their environment against simple examples they see online. So you'll often see Express routes set up like:
...
import express from 'express';
const app = express();
...
//
// Register Node.js middleware
// -----------------------------------------------------------------------------
app.use(express.static(path.join(__dirname, 'public')));
app.use(cookieParser());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
//
// Authentication
// -----------------------------------------------------------------------------
app.use(expressJwt({
secret: auth.jwt.secret,
credentialsRequired: false,
getToken: req => req.cookies.id_token,
}));
app.use(passport.initialize());
if (process.env.NODE_ENV !== 'production') {
app.enable('trust proxy');
}
app.get('/login/facebook',
passport.authenticate('facebook', { scope: ['email', 'user_location'], session: false }),
);
app.get('/login/facebook/return',
passport.authenticate('facebook', { failureRedirect: '/login', session: false }),
(req, res) => {
const expiresIn = 60 * 60 * 24 * 180; // 180 days
const token = jwt.sign(req.user, auth.jwt.secret, { expiresIn });
res.cookie('id_token', token, { maxAge: 1000 * expiresIn, httpOnly: true });
res.redirect('/');
},
);
//
// Register API middleware
// -----------------------------------------------------------------------------
app.use('/graphql', expressGraphQL(req => ({
schema,
graphiql: process.env.NODE_ENV !== 'production',
rootValue: { request: req },
pretty: process.env.NODE_ENV !== 'production',
})));
//
// Register server-side rendering middleware
// -----------------------------------------------------------------------------
app.get('*', async (req, res, next) => {
try {
...
const html = ReactDOM.renderToStaticMarkup(<Html {...data} />);
res.status(route.status || 200);
res.send(`<!doctype html>${html}`);
} catch (err) {
next(err);
}
});
//
// Error handling
// -----------------------------------------------------------------------------
const pe = new PrettyError();
pe.skipNodeFiles();
pe.skipPackage('express');
app.use((err, req, res, next) => { // eslint-disable-line no-unused-vars
console.log(pe.render(err)); // eslint-disable-line no-console
const html = ReactDOM.renderToStaticMarkup(
...
);
res.status(err.status || 500);
res.send(`<!doctype html>${html}`);
});
//
// Launch the server
// -----------------------------------------------------------------------------
/* eslint-disable no-console */
models.sync().catch(err => console.error(err.stack)).then(() => {
app.listen(port, () => {
console.log(`The server is running at http://localhost:${port}/`);
});
});
/* eslint-enable no-console */
This code is actually much longer, but I condensed it for you. As you can see in this code, this is one server file that does everything from setting up your server, initializing all the routes, set up both API routes and view routes, error handling routes, etc. etc. Long story short, it does a bunch of shit
Why is this an anti-pattern? Well for starters, this file is....a bigass file. It is not necessarily easy to follow, read, modify, or test without breaking a whole bunch of other stuff. Say you wanted to write tests for your facebook auth route. That implementation logic is contained within this entire file and module. If you made another change somewhere in this module that breaks the entire thing, then kaboom, your test for a totally different feature just failed. What if you wanted to change the Facebook route to just /login instead /login/facebook. Well now you gotta find every instance of /login/facebook and replace it with /login.
Damn why am I being so critical to this random block of code? You're probably thinking I just pulled it out of some random project. But nah breh. This is straight from the react-starter-kit, a project with 12,500+ stars on Github.
So how do we make this better?
We make this better by taking some inspiration from some mature Web technologies and frameworks. For example, let's take a look at the .NET framework and .NET MVC/Web API.
I'm not going to go into detail on how .NET Web API solves for this, but the TL;DR is that in .NET Web API, they have a concept of controllers, route configs, and views. There is a clear separation of responsibilities. Route configs are responsible for initializing and associating routes with controllers. Controllers are responsible for taking the inputs from the route and invoking the appropriate actions to execute. These actions can involve rendering a view or calling a service if this particular controller is responsible for an API endpoint.
With the same design principles in which .NET separates responsibilities throughout their framework, we want to do the same within our NodeJS web frameworks to enable extensible and modular code that is testable.
Let's look at better code
To help demonstrate these design principles, I wrote a project to Github that is a sample environment using Express. The rest of this blog post will be using this project moving forward, so I'd recommend giving it a download if you want to follow along.
Download the example code here: https://github.com/kelyvin/express-env-example
Clear folder structure
The first thing you may notice is the clear folder structure. There is an app folder specifically for anything involving the front-end and UI, whereas there is a server folder specifically for all things related to the server. This is already a good first step since we are not intermingling view code with our server code.
For anyone familiar with working in NodeJS, usually your first starter point when pulling any new code is to look at the package.json and/or index.js. The first thing you'll see is that the top level index.js only has one job - to start the server.
What is the difference between "controllers" and "services"
"Controllers" are used to define how the user interacts with your routes. Whether your route's purpose is to render a view page or interact with an API, it should be handling both. For example, if your route is supposed to serve an API (e.g. RESTful API's), it's supposed to answer these following questions: Is it a POST request? GET request? What is expected in the request body? Does the user need to be authenticated? Is he/she authorized to access this API? etc. If your route is supposed to serve a view page, it also has to answer similar questions.
Now if your controller serves a view, then most of the time you don't need to worry about interacting with "services". But if you're writing an API, "services" hold your actual business logic or CRUD operations. There are many different opinions/methodologies, but in an essence, a service can holds two purposes:
- Operate as a micro-service - perform some CRUD (create, read, update, delete) against some data or database
- Orchestrate several micro-services - combine and interact with different micro-services to perform some sort of business logic or complex situation
Why have both when I can put all the logic into the controller?
There are many reasons to make this separation, but the main reason is that you don't want to muddle the API's that the user interacts with (i.e. controllers) with how your backend actually operates (i.e. services). Ignoring principles like "versioning" or the validity of how RESTful your API's are, there will undoubtably be use cases where your business logic entirely changes, but the user never realizes you made these changes because the API's they've been interacting with never changes. The separation is important to reduce coupling and helps you follow the single responsibility principle. Obviously this will also help with testing and writing tests, but I won't get there.
Server
In the server folder, you'll see that there is a top-level index.js and three subfolders: controllers, routes, and services. This is all a clear separation of concerns. Again, following NodeJS conventions, we know that our app initializes the server through this index.js.
create = function(config) {
let routes = require('./routes');
// Server settings
server.set('env', config.env);
server.set('port', config.port);
server.set('hostname', config.hostname);
server.set('viewDir', config.viewDir);
// Returns middleware that parses json
server.use(bodyParser.json());
// Setup view engine
server.engine('.hbs', expressHandlebars({
defaultLayout: 'default',
layoutsDir: config.viewDir + '/layouts',
extname: '.hbs'
}));
server.set('views', server.get('viewDir'));
server.set('view engine', '.hbs');
// Set up routes
routes.init(server);
};
You'll see that the responsibility of this index.js is to simply set up the server. It initializes all the middleware, sets up the view engine, etc. The last thing is done is set up routes by deferring that responsibility to the index.js within the routes folder.
Routes
The routes directory is only responsible for defining our routes. Within index.js in this folder, you'll see that its responsibility is to set up our top level routes and delegate their responsibilities to each of their respective route file. Each respective route file will define further define any additional subroutes and controller actions for each one.
// routes/index.js
server.get('/', function (req, res) {
res.redirect('/home');
});
server.use('/api', apiRoute);
server.use('/home', homeRoute);
server.use('/error', errorRoute);
Controllers
Controllers are responsible for invoking the appropriate action. If a controller's responsibility is to render a view, it will render the appropriate view from the app/views directory.
// controllers/home.js
function index (req, res) {
res.render('home/index', {
title: 'Home'
});
}
If a controller is associated with an API endpoint, it will call a respective service within the services directory.
// controllers/apis/dogs/index.js
const
express = require('express'),
dogService = require('../../../services/dogs');
let router = express.Router();
router.get('/', dogService.getDogs);
router.get('/:id', dogService.getDogWithId);
module.exports = router;
Why is all of this valuable?
This improved architecture is extremely valuable because it enables you to change anything on the fly.
Example 1: Scalability
Right now you'll see that there is a /home endpoint defined. What if tomorrow, your Product Manager comes to your engineering team and says that they no longer want to use /home and want all the pages within that route to now be /brand. With the current architecture, you can open the route index.js file and change the top level /home to /brand to change all of its child pages.
// routes/index.js
// server.use('/home', homeRoute);
server.use('/brand', homeRoute);
Then what if he also says he wants /brand/info and /brand/more to point to the same page view? Then open the appropriate brand routes file and add a new /more route to point to the same controller function as /info.
// routes/home.js
let router = express.Router();
router.get('/', homeController.index);
router.get('/info', homeController.info);
router.get('/more', homeController.info);
You no longer have to find every instance and declaration of a particular route to make changes
Example 2: Modularity
Each directory and file have succinct responsibilities. As a result, each file is its own module that can be used wherever it is needed. For example, in the project there is a sample API that I wrote that is associated with the /api route. If you follow a RESTful API standard, you may have different versions of the API that you may want to support. Thus, we expose both a v1 and v2 that are defined within the api directory's index.js file.
// routes/api/index.js
let router = express.Router();
router.use('/v1', v1ApiController);
router.use('/v2', v2ApiController);
Each api may expose different URL endpoints. In this case, one of v1's top level resource may be "dogs" whereas we want the top level resource of v2 to be "animals".
// routes/api/v1/index.js
const
express = require('express'),
dogsController = require('../../../controllers/apis/dogs');
let router = express.Router();
router.use('/dogs', dogsController);
// routes/api/v2/index.js
const
express = require('express'),
animalsController = require('../../../controllers/apis/animals');
let router = express.Router();
router.use('/animals', animalsController);
However, one of the v2 API's endpoints (/animals/dogs) may still do the same thing as the v1 API's (/dogs) endpoint. So instead of trying to rewrite the implementation, we simply reference the same module used for /dogs in /animals/dogs.
// controllers/apis/dogs/index.js
const
express = require('express'),
dogService = require('../../../services/dogs');
let router = express.Router();
router.get('/', dogService.getDogs);
router.get('/:id', dogService.getDogWithId);
// controllers/apis/animals/index.js
const
express = require('express'),
dogController = require('../dogs');
let router = express.Router();
router.use('/dogs', dogController);
As you can see, our v2 API can still leverage the same modules as v1 while continuing to maintain backwards compatibility. This is all done without any duplicate code and a clear separation of logic.
Key Takeaways
If you download the example Express environment that I set up, you can clearly see that the improved folder and module structure enables for more modular routing that can easily scale. By simply making a few tweaks and changes, your code is lot more digestable and can help other developers contribute without worrying about breaking or modifying another feature. In fact, this is a prime example of writing code with low coupling. Each individual module is self-contained and unaware of who might be using it, but each is responsible for one particular job.
There are many opinions and ideas on how to properly set up your web framework for NodeJS - this may not even be the best solution for you. However, this is built with the same inspiration and ideas that currently exist within "tried and true" backend technologies that have been around for years before. Express and other web frameworks for NodeJS are minimal by design, to enable you to form your own opinions on how the architecture should be structured. So, if you have any better thoughts, ideas, or something that's worked well for you, please share them!
