Generating and publishing Go gRPC stubs as separate modules via GitLab CI/CD

Setting up gRPC stub generation for Go and connecting them as a module

Keeping proto contracts in a single repository is convenient, but pulling the entire thing into every service is not. Let's walk through how to automatically generate Go stubs from proto files, version them as standalone Go modules, and publish them via GitLab CI/CD. Bonus: Swagger documentation and GitLab Pages.

Everything described here targets private free-tier GitLab. For self-hosted, paid plans, or public repositories the setup is considerably simpler — a number of the limitations simply don't apply.

Notes before you start

The approach requires some dev environment setup: standard SSH access to the repository and a few environment variables on each developer's machine:

GOPRIVATE=gitlab.com/your-group
GOPROXY=direct
GONOSUMDB=gitlab.com/your-group

Services that consume the stubs will need a vendor directory — both for local runs and in pipelines during linting, testing, and builds. This is because Go can't download private modules from GitLab without additional auth configuration bypassing the proxy.

In CI environments, access is handled via CI_JOB_TOKEN. A typical before_script for services consuming the stubs looks like this:

default:
  image: golang:${GO_VERSION}
  cache:
    key: go-mod
    paths:
      - go/pkg/mod
      - .cache/go-build
  before_script:
    - go version
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf "https://gitlab.com/"
    - go env -w GOPRIVATE=gitlab.com/chichilaki
    - go env -w GONOSUMDB=gitlab.com/chichilaki
    - go mod vendor

Repository structure

Create the following groups and repositories:

group/proto — proto contracts; the only repository you'll edit by hand.
group/proto-stubs — a group for generated stubs; create the repositories inside it ahead of time, empty.
group/proto-stubs/common, group/proto-stubs/user, group/proto-stubs/admin, etc. — one repository per module.
group/service — any service that consumes the stubs.

Stub repositories

Each stub gets its own Go module repository. In my case there's a contract for shared entities (common) and contracts for services: user, admin, runner.

Configuring access for CI_JOB_TOKEN

Since every service uses CI_JOB_TOKEN in its pipeline to access stub repositories, you need to explicitly allow that access.

For each stub repository, go to Settings -> CI/CD -> Job token permissions and add the proto contracts repository and the group containing services that consume the stubs.

One important nuance: if some stubs depend on others — for example, admin imports common — you also need to add the stub repositories group to the access settings of admin.git. Otherwise go get in CI will fail with an authentication error, and the reason won't be obvious at all.

Paid and self-hosted GitLab lets you grant access at the group level — everything is configured in one place.

Why .git in the module name

The module name includes a .git suffix:

gitlab.com/chichilaki/mgs/proto-go-stubs/common.git

This is intentional. When resolving a module path from GitLab, Go can't figure out on its own where the repository path ends and the in-repo package path begins. The .git suffix gives it an explicit hint. Without it, go get will try to fetch module metadata from the wrong path and fail.

Proto contracts repository

This is the repository that triggers the stub generation and publishing pipeline.

Versioning

The publish job runs only for branches named vX.X.X. On publish, a matching tag is created in the stub repository. This means the branch name in the proto repo directly becomes the Go module version. Want to release v1.2.3 — create a branch called v1.2.3 and run the pipeline.

Setting up a Personal Access Token

Note: GitLab Next recently added the ability to configure CI_JOB_TOKEN access for pushing to repositories — worth checking, this step may soon become unnecessary.

In the free tier of GitLab you can't issue an access token scoped to a group of repositories, so we use a Personal Access Token instead.

Go to Profile Settings -> Access Tokens and create a token with the following permissions: api, read_api, read_repository, write_repository. The token is shown only once — save it immediately.

Then in the proto contracts repository: Settings -> CI/CD -> Variables. Create a variable with:

type: Variable
visibility: Masked and hidden
flag: Protected

If there are multiple developers on the team — each person creates their own token and their own variable, e.g. CI_PUSH_TOKEN_USERNAME. In the pipeline, the right token is selected via rules with a condition on GITLAB_USER_LOGIN — more on that below.

Setting up Branch Rules

To make protected variables available outside the main branch: Settings -> Repository -> Branch Rules -> Add rule. The mask v* gives any branch starting with v access to protected variables and tokens. Without this step, the pipeline on branch v1.0.0 simply won't see the token.

buf configuration

buf.yaml describes dependencies and linting rules; buf.gen.yaml covers the generation plugins.

In buf.gen.yaml we use local with go run instead of a globally installed protoc and plugins — Go will download the required versions on first run. This eliminates version mismatch issues between developer machines and CI.

Both files are included in full in the appendix.

Proto file examples

The go_package option must include the .git suffix — that's how Go will resolve the module import:

// common.proto
syntax = "proto3";
package common.v1;
option go_package = "gitlab.com/chichilaki/mgs/proto-go-stubs/common.git/v1";

For service contracts with grpc-gateway HTTP annotations and Swagger options, the structure looks like this:

// admin.proto
syntax = "proto3";
package admin.v1;
option go_package = "gitlab.com/chichilaki/mgs/proto-go-stubs/admin.git/v1";

import "common/v1/common.proto";
import "google/api/annotations.proto";
import "protoc-gen-openapiv2/options/annotations.proto";

option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {
  info: { title: "Control Plane ADMIN API"; version: "1.0"; };
  security_definitions: {
    security: {
      key: "Bearer";
      value: {
        type: TYPE_API_KEY;
        in: IN_HEADER;
        name: "Authorization";
        description: "Admin Bearer token";
      };
    };
  };
  security: { security_requirement: { key: "Bearer"; } };
};

service AdminService {
  rpc ListRunners(ListRunnersRequest) returns (ListRunnersResponse) {
    option (google.api.http) = { get: "/v1/runners" };
  }
  rpc GetRunnerStatus(GetRunnerStatusRequest) returns (GetRunnerStatusResponse) {
    option (google.api.http) = { get: "/v1/runners/{runner_id.runner_id}/status" };
  }
  // ...
}

CI/CD pipeline

The proto repository pipeline has three stages: lint, generate, commit. The full version is in the appendix.

The generate job

Runs buf generate, then creates a go.mod with the correct module name (including .git) for each module. Artifacts are passed to the next job via artifacts.paths.

Key detail: go mod init is only called if go.mod doesn't already exist, to avoid overwriting the file on re-runs. Otherwise the module would be re-initialized on every run and lose its dependencies.

The commit-generated job

Clones each target repository, replaces its contents with the generated files, copies swagger.json, updates dependencies via go mod tidy, commits, and creates a version tag.

Token selection per user is handled via rules with a condition on GITLAB_USER_LOGIN — this lets each developer use their own Personal Access Token without sharing it with teammates:

rules:
  - if: $CI_COMMIT_BRANCH =~ /^v\d+\.\d+\.\d+$/
    variables:
      PROTO_VERSION: $CI_COMMIT_BRANCH
  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    when: never
  - if: $GITLAB_USER_LOGIN == "End1essRage"
    variables:
      PUSH_TOKEN: $CI_PUSH_TOKEN
  - if: $GITLAB_USER_LOGIN == "stivvhuys"
    variables:
      PUSH_TOKEN: $CI_PUSH_TOKEN_GENGAVSOV
  - when: never

If PUSH_TOKEN ends up empty — the job fails in before_script with an explicit error rather than a cryptic 403 somewhere mid-script. This is an intentional check.

For modules that depend on common, the script explicitly adds a require with the correct version:

if [ "$module" != "common" ]; then
  go get gitlab.com/chichilaki/mgs/proto-go-stubs/common.git@$PROTO_VERSION
fi

This matters: without an explicit go get with the tag, go mod tidy will pull in latest instead of the version we just published.

The service consuming stubs

Let's look at the consumer side — a service that wants to host its own Swagger documentation, serve it over HTTP, and interact with the API through Swagger UI.

Embedding swagger.json via go:embed

A docs package is created at the project root:

package docs

import _ "embed"

//go:embed control-plane-admin.swagger.json
var AdminSwaggerJSON []byte

//go:embed control-plane-user.swagger.json
var UserSwaggerJSON []byte

Imported in main.go as a side-effect:

import _ "gitlab.com/chichilaki/mgs/control-plane/docs"

mux.HandleFunc("/swagger/admin/doc.json", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    _, _ = w.Write(docs.AdminSwaggerJSON)
})

mux.HandleFunc("/swagger/user/doc.json", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    _, _ = w.Write(docs.UserSwaggerJSON)
})

go:embed compiles the files directly into the binary — no separate static file serving needed, and no risk of the documentation going out of sync with the code.

Syncing swagger.json

Documentation is synced in three places: a pre-commit hook, a Taskfile, and the service CI pipeline. The logic is the same in all three: clone the stub repository into a temp directory, grab v1/<service>.swagger.json, and put it in docs/.

Locally we use SSH; in CI — CI_JOB_TOKEN. The corresponding Taskfile and CI configs are in the appendix.

Local stack: Traefik + Swagger UI

Brought up via Compose. Note: the BASE_URL in the Swagger UI config and the PathPrefix in the Traefik rule must match — otherwise the UI will load but won't be able to resolve its own assets.

swagger-ui:
  image: swaggerapi/swagger-ui:v5.9.0
  environment:
    BASE_URL: /swagger
    URLS: >
      [
        { "url": "/cp-swagger/swagger/admin/doc.json", "name": "ADMIN control-plane" },
        { "url": "/cp-swagger/swagger/user/doc.json", "name": "USER control-plane" }
      ]
  labels:
    - "traefik.http.routers.swagger.rule=Host(`localhost`) && PathPrefix(`/swagger`)"
    - "traefik.http.routers.swagger.entrypoints=web"
    - "traefik.http.services.swagger.loadbalancer.server.port=8080"

Service route with /cp prefix stripping:

control-plane:
  labels:
        # ===== API =====
      - "traefik.http.routers.cp-api.rule=Host(`localhost`) && PathPrefix(`/cp`)"
      - "traefik.http.routers.cp-api.entrypoints=web"
      - "traefik.http.routers.cp-api.service=cp"

      - "traefik.http.middlewares.cp-strip.stripprefix.prefixes=/cp"
      - "traefik.http.routers.cp-api.middlewares=cp-strip"

      - "traefik.http.services.cp.loadbalancer.server.port=9100"
      # Enable health check for the load balancer
      - "traefik.http.services.cp.loadbalancer.healthcheck.path=/health"
      - "traefik.http.services.cp.loadbalancer.healthcheck.interval=10s"
      - "traefik.http.services.cp.loadbalancer.healthcheck.timeout=3s"
      - "traefik.http.services.cp.loadbalancer.healthcheck.scheme=http"
      # Swagger proxy via Traefik
      # ===== SWAGGER =====
      - "traefik.http.routers.cp-swagger.rule=Host(`localhost`) && PathPrefix(`/cp-swagger`)"
      - "traefik.http.routers.cp-swagger.entrypoints=web"
      - "traefik.http.routers.cp-swagger.service=cp-swagger"

      - "traefik.http.middlewares.cp-swagger-strip.stripprefix.prefixes=/cp-swagger"
      - "traefik.http.routers.cp-swagger.middlewares=cp-swagger-strip"

      - "traefik.http.services.cp-swagger.loadbalancer.server.port=9100"

Bonus: GitLab Pages with Swagger UI

It's handy to have documentation available not just locally but at a permanent URL. GitLab Pages solves this without any separate hosting — just put the artifacts in a public folder.

The pages job creates the public folder, places the swagger files there, and generates an index.html with a CDN-hosted Swagger UI. It runs on master and dev branches. The full job is in the appendix.

CI: fetch-swagger + pages

fetch-swagger:
  stage: prepare
  image: alpine:latest
  before_script:
    - apk add --no-cache git
  script:
    - mkdir -p docs
    - |
      for REPO in admin user; do
        git clone --depth 1 \
          "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/chichilaki/mgs/proto-go-stubs/${REPO}.git" \
          "/tmp/${REPO}"
        cp "/tmp/${REPO}/v1/${REPO}.swagger.json" "docs/control-plane-${REPO}.swagger.json"
        rm -rf "/tmp/${REPO}"
      done
  artifacts:
    paths:
      - docs/*.swagger.json

pages:
  stage: pages
  image: alpine:latest
  script:
    - mkdir -p public/docs
    - cp docs/*.swagger.json public/docs/ || echo "No swagger files"
    - |
      cat > public/index.html << 'EOF'
      <!DOCTYPE html>
      <html>
      <head>
        <title>API Documentation</title>
        <meta charset="UTF-8">
        <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui.css">
      </head>
      <body>
        <div id="swagger-ui"></div>
        <script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui-bundle.js"></script>
        <script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui-standalone-preset.js"></script>
        <script>
          window.onload = function() {
            window.ui = SwaggerUIBundle({
              urls: [
                { url: "./docs/control-plane-admin.swagger.json", name: "Admin API" },
                { url: "./docs/control-plane-user.swagger.json", name: "User API" }
              ],
              dom_id: '#swagger-ui',
              deepLinking: true,
              presets: [SwaggerUIBundle.presets.apis, SwaggerUIStandalonePreset],
              layout: "StandaloneLayout"
            });
          }
        </script>
      </body>
      </html>
      EOF
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
    - if: $CI_COMMIT_BRANCH == "dev"

Conclusion

Here's what we end up with: proto contracts in a single repository, automatic stub generation and publishing triggered by creating a branch like v1.0.0, separate Go modules per service, Swagger documentation as part of the same pipeline, and Pages with a UI for browsing it.

The main challenge with free-tier GitLab is token management. On a paid plan or self-hosted, most of that overhead disappears: you can grant access at the group level and use a single token. But even as described, everything works reliably.

推荐订阅源

DEV Community