How to use Gulp, Webpack, Node.js, and Codeship effectively for Heroku deployment

Kelvin Nguyen -

Over the past several weeks I've been working on building an ideal developer environment that I can use to build and launch a variety of my side projects. Because this environment will only be used by me, expect a small number of sessions/users, and since I'm a cheapass, I wanted the environment to solve the following problems:

  1. Provide both an API server and web server for my web apps
  2. Be easily deployable
  3. Support both front-end and backend development with all the latest tools
  4. Enable future decomposition and modular design
  5. Ensure I only need to pay for one server/instance/dyno/etc for all my web apps

Architecture

To make this happen and if you couldn't already tell, a lot of my previous work/expertise has been within the Javascript space, so Node.js is the perfect solution to help me solve these problems. It would help me solve problems (1) and (3) right off the bat. It also solves (5) because I can separate all my apps into different subdomains using the same backend. I already wrote about how to do this on my post: Setting up subdomains on one express app. So I will be using that code as the starting point for my new environment.

So let's go through what the architecture and folder structure looks like.

This project contains a couple of key folders. The app directory contains all of our front-end and UI code. Each folder within it represents a different side project that will be assigned its own subdomain. The server directory contains all of our server-side code that Nodejs will use to launch our web server. It contains our API server and hosts our apps using express. The configs directory contains all of our custom configs for each environment (developer, production, etc). Finally, the build-utils directory contains all of our build tasks and will house any other build instructions. So broken down, the contents within these directories will look something like this:

I'm not going to go into the details of each file, the tools I'm using, or their responsibilities. That'll further discussed in a different post. Instead, I'll be spending time going over the build system, how this environment enables me to work efficiently, and how it is easily deployed out to production.

Build system

This project uses a combination of gulp + webpack. I am using gulp for task management and webpack for asset bundling for client-side files.

Because this project contains both a node web server and the front-end code to be used by the browser, I am leveraging both systems to handle each respectively. Every build will bring all the relevant assets into the dist directory. This includes both the server code, cient code, and any environment configs. More on that later.

Gulp

I am using gulp instead of npm scripts because I want to create more complex tasks. My Gulp tasks are responsible for transpiling all of my non-client side code, copying files to the respective directory, and any tasks that help improve the developer workflow. Gulp is helping to ensure that I can use the latest tools to streamline my developer workflow - I can write all of my code (on both the frontend and backend) in ES6, my local web server restarts automatically, any files I change are recompiled as needed, and etc.

Webpack

Webpack is specifically used for asset management for my front-end UI code. Thus, it is responsible for transpiling, bundling, and minifying + uglifying any code within the app directory and output the results into the dist directory. Webpack is especially good at asset bundling, which is why we're using this over setting up complex Gulp tasks.

dist

As mentioned earlier, every build will bring all the deployable, relevant code into the dist directory. This includes the server code, client code, and any environment configs.

When I want to deploy to production, I would only deploy this dist directory and any other files required by the production server. In this case, because I will be deploying to Heroku, I will also want to deploy the Procfile and package.json. Heroku will respect the Procfile and use it determine how to run the app. So in the Procfile, you'll see the following code:

web: node dist/index.js

Deploying to Heroku using Codeship

From this point on, you can choose to deploy to Heroku directly from your local codebase, but I want to make this as streamlined and easy to deploy as possible, so I will be leverging Codeship, a CI/CD platform. Why Codeship and not Travis CI, Circle CI, or etc? Cause Codeship's free tier allows for private repo's and has Gitlab integration (cause I use Gitlab's free unlimited private repos...because I'm cheap).

I'm not going to go over how to set up a project on Codeship, you can find out how to set up your job here. I will be setting up the build job like so

These commands will make sure I am running my build job against Node v8, install all the node_modules, install gulp-cli on the machine, and run the build job. As I mentioned earlier, when the build runs, everything gets compiled into the dist directory and I want Codeship to deploy the following:

  1. package.json
  2. Procfile
  3. /dist
  4. yarn.lock

I only need to include these files because once the files are deployed on Heroku, Heroku's Node.js buildpack will run yarn install before running the Procfile. In Heroku I have already specified the NODE_ENV environment variable to production. When that is the case, yarn will only install the production node_modules required for the project.

IMPORTANT Because yarn will only install the production dependencies when NODE_ENV=production, make sure that Codeship does not do this. Since we want Codeship to run the build job, we need it to install our devDependencies as well. To make sure that happens, I set a NODE_ENV environment variable on Codeship to test.

To make sure that I only deploy those four key things, Heroku supports something called a .slugignore. This file acts like a .gitignore file and enables you to specify which files/directories to exclude from the deployment.

NOTE
.slugignore will handle all file names and regex's recursively, so if you input something like *.js or index.js, it will exclude all JS files or JS files with the name index.js respectively. So make sure you are careful with how you specify the files to exclude from the deployment.

Use Heroku's Platform API for deployment

You may notice that Codeship has Heroku deployment integration, as documented here. However, we will not be using this. If you deploy using Codeship's built-in Heroku deployment (or even when you deploy from your own local environment), it performs the deployment through Heroku's GIT deployment instructions. This is great and easy most of the time, but keep in mind that it not only respects the .slugignore file, but the .gitignore file as well.

Most likely, you have included the /dist directory in your .gitignore to ensure that it does not get pushed to your Git repo, but as a result, it will also not deploy your directory to Heroku! So how do you solve this?

Heroku provides a Platform API that is optimized for non-interactive CI setups. Thankfully, Codeship has open-sourced some of their deployment scripts for custom builds, one of which supports the Heroku Platform API. You can find that script here and their other scripts here! If you follow the instructions on the script, you'll be able to deploy to Heroku through Codeship with the exact files that you want! As you can see, I am using both the deploy script as well as the check url script to make sure the website returns a successful 200 after deploying.

This deployment problem solves (2) of my list above. Once all this setup is done, whenever I commit code to the master branch, Codeship will automatically pull the code, run the build, and auto deploy to Heroku.

Alright now where's the code?

But what about (4)? Well, in its current state, it has already lended itself to prepare for modular design. If I want to host the front-end code for one of my apps in a different GIT repo, deploy them to a CDN, or simply want to separate my front-end and server code so that I don't have to restart my Heroku instance every time I make a UI change, the code/folder structure will allow me to easily pull these out as needed with little hiccups.

However, this environment is still a work in progress for me and I'm still in the process of optimizing certain aspects of the architecture. I had to do a lot of research to figure out how to get Codeship + Gulp + Webpack + Heroku to play well together. Some similar scenarios posted on StackOverflow or any existing documentation and blog posts did not provide an ideal solution that I desired. It was all kind of hacky. As a result, I wrote this to help some people who are trying to solve similar problems (e.g. deploying a dist or public folder to Heroku when it's part of a .gitignore, how to make sure your deployment doesn't install unneeded dependencies, how to make one Express app support multiple subdomains, etc.)

Expect the sample code to be released in the near future, stay tuned...