How the hell do we run all these microservices? This was one of the painful questions that we asked ourselves after we started working on our cloud platform Oblax (at the time called BoosterCloud). It was easy to run the services when it was just one or two of them, we would simply launched another terminal or use tmux and do a split for the new service. But the number of services grew, so we had to split it again, and then again, and then a… you get the point.

After a while we came to a point where this way of doing things became a problem. Too many terminals and panes spitting print-lines and errors. It was getting pretty hard to track what’s where and what it do. So, there is a day in every platform engineer’s life when they have to start developing their own tooling.

Doing it the BASH way

The easiest way around the problem was to just create couple of bash scripts that would run all the services at once. The script looped through all our microservice directories and did a compile+run… So, nada especial… We consolidated the service output into the same terminal, and added file logging, that was it, we were ready to roll.

All services were run natively because it was easier to debug. Each of them ran on a different port, and since we did not want to hardcode the port numbers, we created a env exports file (script?) containing the port numbers for each of the services which we then manually sourced just before the service were fired-up.

This worked pretty good, at least for a while.

Accessing the services from a front-end app

We stabilized most of the planned services, and started working on some front-end apps that would utilize them. You know standard stuff, back-office and admin consoles.

And then… we were calling 20+ different services by their port number, which to be honest didn’t really echo a well planned system. We needed a reverse proxy. And what did we do? Use Nginx? No! Use HAProxy, Caddy, Traefik? Hell no! We built the reverse proxy by ourselves, because in this house, that’s how it’s done!

To the untrained eye it may look like we did a real badass thing by doing this, but before you say that out loud just check out httputil.NewSingleHostReverseProxy in the golang docs. In short (if you were too lazy to google it), you don’t need to write a lot of code to get a reverse proxy going. Sure, there are things that you’ll need to handle manually, but the possibility to write a reverse proxy in one day still stands.

So now we had the bash scripts, the env exports and the proxy. We were ready to go! And so we did for couple of years… Until…

The Oblax rewrite

Ask any decent developer about rewrites, and they will tell you “You never go full rewrite!”.

But we did!

The reason we did this is a part of a completely different story (for which we reserve the right to tell in the future), what’s important was that “when in Rome”… You also rethink the way you run and manage your services locally.

I sat one day and created a list of things that this new “runner” should have. It should:

• build/run all the services
• serve SPAs
• inject env variables
• serve static (html/css) content
• reverse-proxy the services
• have single config file
• pass build flags (ldflags)
• distinct separate build groups

and the most important one

• detect changes and auto-rebuild only the affected services

It wasn’t a particularly scarry list. I already had an idea how to do most of the things. What was left was just building it.

The Wingman

Every cool project needs a cool name. And what’s cooler that being cool?! I know, ICE COLD! In this case it was, drumroll please… Wingman.

Yes, like, you are a developer, and this guy right here helps you run your code, rebuilds when needed, not much maintenance… You get me? A true wingman!

Wingman is written in go (not a big surprise, eh?) It’s open source and freely available for download (and PRs) at it’s GitHub repo.

At the moment you can install it by running

$ go install github.com/oblax/wingman@latest

It has a single config file called wingman.yaml, which you can easily generate by executing wingman init. This will generate a very basic version of the config whih you’ll need to fill in with your project’s details.

Few keypoints to get you started

• Add the “include dirs”. This tells the file-change watchers where to watch. The path is relative and starts at the directory where the wingman.conf file is placed.
• Add the “include file”. It basically tells the watchers what kind of files should be monitored for change. Usually it’s *.go files, because “go”, but maybe you’ll need monitor some html templates for change or even some configs.
• Add any environment variables that are shared between services.
• Define a service.

Here comes the important part. There are proxied and unproxied services. Usually unproxied service means a GRPC or an asynchronous service that listens for orders from an MQ. Proxied services come in two flavours, service and static. Proxy type “service” usually means that it’s a REST/HTTP service expecting http data on a certain port. This kind of proxy can also be used if you are running a React (or similar) frontend in dev mode (running on a port, served by nodejs). The “static” service is used for serving static files. You’ll notice that instead of a port number, there is a “proxy static dir” that needs to be defined.

Let’s take the following directory structure as an example.

.
├── bin
├── go.mod
├── go.sum
├── pkg
│   ├── libone
│   │   └── libone.go
│   ├── libtwo
│   │   └── libtwo.go
│   └── shared
│       └── shared.go
├── public
├── services
│   ├── obx-test-service-one
│   │   ├── handlers
│   │   │   └── handler.go
│   │   └── main.go
│   └── obx-test-service-two
│       ├── handlers
│       │   └── handler.go
│       ├── public
│       │   ├── image.jpg
│       │   ├── script.js
│       │   └── style.css
│       └── main.go
└── wingman.yaml

There are two main directories to be watched:

• pkg - containing all the shared code between the services
• services - containing the actual service code

If you look at the services, there are two distinct services, obx-test-service-one, and obx-test-service-two. The first one we’ll configure as a GRPC service, which means no proxying, just the service location, ldflags, environment. We define the entrypoint, which is the directory where the service code resides, and the executable which is the name of the built service binary.

For the REST service we repeat the steps from the previous one, but now with some added flavor. There are four more definitions that need to be added so it’ll be proxied correctly. First is the proxy type, which is “service”. Second, we set the proxy handle. This is the part of the URL string that identifies to which service a request should be proxied to. If the url is http://myproject.com/api/v1/test-service-two/blah/blah it will identify the /api/v1/test-service-two part and know that this request needs to be proxied to obx-test-service-two. The proxy address is the address of the service, which usually would be 127.0.0.1, and the proxy port is the port on which the service is running on.

The last service that we’ll define it actually no service at all, it’s a static file server. Similar to the previous two, but with a lot less configuration. Here we specify the proxy type as “static”, we define a proxy handle which is the path that “identifies the service”, and instead of an address and a port number we just define a proxy static dir, which is the place where the files that want to serve are located.

To make all this work, this is the config you’ll need.

version: 1 # The config version number. For now it's 1
module: oblax.io # The name of the go module
build_dir: bin # The build directory for the services
watchers:
  include_dirs: ["pkg", "services"] # Directories to be watched
  exclude_dirs: ["vendor", "modules"] # Directories to be excluded from watching
  include_files: ["*.go"] # Types of tiles to be watched
  exclude_files: ["test_*.go"] # Types of files not to be watched

env: # Environment variables available to all services at start
  GODEBUG: 'x509sha1=1' 
  OBLAX_REST_ERROR_MODE: 'development'

service_groups: 
  testing: ["obx-test-service-one", "obx-test-service-two"] # A service list

services: 
  obx-test-service-one: # An example of a GRPC/Protobuf service
    entrypoint: services/obx-test-service-one # Service location directory
    executable: obx-test-service-one # Name of the built service
    ldflags: # Build flags
      oblax.io/services/obx-test-service-one.Version: 'v0.1'
      oblax.io/services/obx-test-service-one.Build: 'dev-build'
      oblax.io/services/obx-test-service-one.Name: 'obx-test-service-one'
    env: # Service specific environment variables
      PORT: 10001

  obx-test-service-two: # An example of a REST service with reverse proxy
    entrypoint: services/obx-test-service-two
    executable: obx-test-service-two
    proxy_type: service 
    proxy_handle: /api/v1/test-service-two # When someone asks for this route
    proxy_address: 127.0.0.1 # ...proxy to this address
    proxy_port: 10002 # ...and this port
    ldflags:
      oblax.io/services/obx-test-service-two.Version: 'v0.1'
      oblax.io/services/obx-test-service-two.Build: 'dev-build'
      oblax.io/services/obx-test-service-two.Name: 'obx-test-service-two'
    env:
      PORT: 10002

  obx-test-web-storage: # An example of a static file handler
    proxy_type: static
    proxy_handle: /public/platform # Whenever someone asks for this route ...
    proxy_static_dir: services/obx-test-service-two/public # ...static files

That should be enough to get you started. Wingman is a real dogfed tool that is saving us at Beyond Basics and Oblax a lot of manual work, and makes microservice development really easy without the bloat of containerizing services and running them behind “a real reverse proxy”.

So, what now? Now, we’d like to hear from you. Do you find Wingman useful? Is there anything that you’d like to see added in the next version?

Over and out!