Dev Environments — Why and How

Cloud Env = Happy Dev :)

Introduction

In this article, we will discuss why you need a developer environment in your organization, what you should strive for, and how to go about implementing one.

This is quite a large topic to cover, so we will not go into any implementation. Instead, I will try to guide you through the thought process of integrating a good tool while utilizing known solutions, like a puzzle.

Recently, while working at LayerX, our dev environment tool Raftt was acquired and stopped maintaining the product. We had to make a decision — swap to a different solution, or build one in-house.

What we want

As a developer, I want to be able to run a copy of my application as I develop it. I don’t care where it runs or how it works. It should run all the services, wire them up, and make me feel like I’m serving it locally.

The problem is, the application often consists of multiple services, frontend and databases. You may think docker is a good solution, but as your codebase and system complexity grows, your machine will not be able to handle it, and will start to sound like an apache helicopter taking off.

Requirements

• Each developer should have a separate environment
• Run in the cloud to allow scaling the system
• Hot reload — When changes are made, they should be reflected in the environment
• Port forwarding — the system should feel as though it’s running locally
• Database seeding — The database should be pre-loaded with application data, and we should be able to run migrations against it.

Existing solutions

I always prefer to use a SaaS solution when possible, and advice you to do the same. At the time of writing, both Okteto (expensive), or Tilt (runs locally) are viable solutions — please take a look at them before starting to develop and maintain you in-house solution.
Keep in mind, that an in-house solution may be superior, as it allows you to make assumptions to optimize the process. All solutions require a lot of work to integrate!

A word about cost

You might feel discouraged from running these environments in the cloud — it’s expensive and unnecessary. Or, you might feel like this is unnecessary, and you can just serve your services using node.

Consider how your devs will thrive with such a tool. No more discrepancies between environments, your devs can verify everything E2E while developing without a hassle. You have less bugs, deliver faster features. You encourage E2E features, ownership and confidence within your R&D.

Still think it’s not worth it? Contrast the cost with the developer salary, and how much time they’re wasting on bugs that could’ve been found before deployment. I’m not exaggerating when saying a dev environment is one of the most important tools for your R&D.

Designing a Solution

Let’s make a few assumptions that will help us create a solution that is not as complicated as Okteto -

• Our backend consists of single language. We’ll focus on nodejs in this article.
• We have a working Kubernetes with autoscaling to utilize for remote compute
• We have built docker images of our base commit from main, which lack our code but have the base tools and most packages to run the app (node and node_modules).

Solution, outlined in two flows

Initial Setup

Initial setup flow

• Deploy a database (optional) and seed it. You can skip this if you don’t want to seed the database, but it’s recommended you implement this at some point to create a better developer experience.
• Deploy the latest docker images
• Setup port forwarding

Local Changes Syncing

Syncing local changes

After the initial setup, we should have a working replica of the staging environment. However, we lack the local changes we made.

Every time we save, and initially, we’d like to sync those local changes to our environment. This consists of:

• Watching for changes
• Building the app (using webpack or any other build tool)
• Sync packages (copy the local package.json file and run npm)
• Hot reload — Copy the built file main.js to the remote, and reload the process. This can be achieved with nodemon.

Implementation Details

Implementing a solution will change between companies, depending on your tech stack. Our tech stack includes some useful tools that made the job easier, which you may decide to utiliize -

Tool Usability

Aim for a good developer experience — The developer should be able to run dev up and dev down or something similar. Provide good errors, and help the developer understand if something is wrong.

For example, if a project fails to bulild, it should be easily diagnosed using a log file and something like dev status .

Commands you should consider (look at okteto for more) —

• dev status — Provide status for deploying, building, syncing and port forwarding
• dev port-forward — Ensure and fix the port forwarding
• dev sync — Force the syncing process to restart, in case of build errors and such.

To achieve this, you want some sort of CLI component to spawn the environment. Then, decide between a web UI and a CLI to debug and use the tool.

ArgoCD / Flux

ArgoCD allows you to manage your application on Kubernetes using GitOps. Luckily, we utilized this tool from day one. We already had all our cluster definitions, and creating a ‘dev’ environment was as easy as tuning the resources and HPAs.

If you’d like to learn more about ArgoCD and why it’s amazing, I recommend this tutorial.

If you’d like to use GitOps for your solution, you could use the following flow —

• When spawning an environment, create a branch for the developer in the GitOps repo.
• Make the necessary changes to the YAML definitions — Reduce the resources, scale down the deployments, and duplicate any secrets. You may want to replace the container command (from node to nodemon for example)
• Create an ArgoCD ApplicationSet on the remote cluster to track the remote branch, for example dev/koby . Deploy the application set to a separate namespace per developer, for example dev-koby
• ArgoCD will deploy the changes. If you’re using seeding, you might want to apply a subset of the app (DBs only) first, seed, then deploy the entire app.
• Port forward using a kubernetes client (or kubectl).
• When deleting the environment, simply delete the ApplicationSet. ArgoCD will take care of the rest.

Change Management

Take inspiration from your language build tools to watch for changes. We use nx monorepo, so we had the watch functionality out of the box.

• Watch functionality — Detect what project has changed
• Detect if package syncing is needed
• Copy the files to the remote container (using kubernetes client or kubectl)
• Reload the process (prefer existing tools like nodemon) — If you decide to kill the process, it may kill the container so be careful