Üllar Seerme

Uploading GoReleaser snapshot releases in GitLab

April 14, 2023

I’m wary about doing a short write-up about this as I don’t want anyone to eat Carlos Alexandro Becker’s lunch in any way. He seems to be an absolute lad and rockstar when it comes to his open source work, and I’m hoping this lands softly in regards to people still shelling out for the pro version of GoReleaser.

So, the nitty-gritty. Setting up GoReleaser for GitLab is trivial using the official documentation, but that only takes care of the tag-based releases. I wanted a system wherein people could create merge requests that would result in archives being created in GitLab’s Generic Packages Repository.

At first I thought I would just be able to use the HTTP upload functionality that GoReleaser has, but I didn’t see a way I could do so without creating multiple .goreleaser.yml files. Not that multiple configuration files are inherently bad, but if I can I try to avoid having too many of them. Next I looked over the Generic Packages Repository documentation and saw that one can just use curl to upload files using --upload-file. A lightbulb moment if there ever was one!

By default, however, GoReleaser’s snapshots functionality creates archives that contain the current commit ID as well as the currently released tag in its name. This would have meant that every new commit in relation to a merge request would result in an entirely new package (in terms of GitLab) being created which has the potential to balloon the space requirements for a single Git repository. Luckily, the name can be changed. The example to change the name of a snapshot release though is as follows:

1
snapshot:
2
# Default is `{{ .Version }}-SNAPSHOT-{{.ShortCommit}}`.
3
# Templates: allowed
4
name_template: '{{ incpatch .Version }}-devel'

The ‘incpatch’ common field bumps the patch segment of a given version by one. So, for example, if version 1.0.0 is currently released and someone creates a merge request with proposed changes, then a job would kick off that would create 1.0.1-devel. Seems perfect. However, what if multiple people create merge requests? They would collide with each other! And in GitLab any file uploads for the same version would result in duplicates being appended under the same version.

For a given merge request within a project, its own ID (i.e. the merge request’s ID) seems to be a unique enough value to be used within the name of the package in order to differentiate versions. GitLab has a bunch of predefined variables. One of them is CI_MERGE_REQUEST_IID, which is:

The project-level IID (internal ID) of the merge request. This ID is unique for the current project.

For local usage of GoReleaser I wanted to keep the devel string within the name, but for snapshot releases happening in merge requests I wanted to use the value of the environment variable CI_MERGE_REQUEST_IID. GoReleaser, again, has thought of pretty much everything and has global environment variables, which I set up as follows:

1
env:
2
- ENV_MR_IID={{ if index .Env "CI_MERGE_REQUEST_IID" }}{{ .Env.CI_MERGE_REQUEST_IID }}{{ else }}devel{{ end }}
3
4
snapshot:
5
name_template: "{{ incpatch .Version }}-{{ .Env.ENV_MR_IID }}"

This would result in a package version like 1.0.1-1 if this was the first merge request in the project. For GitLab’s CI it’s necessary to create a job that only triggers for merge request events. I usually use YAML anchoring for rules as then I can just define them at the top and use aliases to reference them elsewhere. The following sets up two variables - one for the packages API and one for the project-specific package registry - and one rule for triggering upon commits to an active merge request:

1
stages:
2
- publish
3
4
variables:
5
PACKAGE_API_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages"
6
PACKAGE_REGISTRY_URL: "${PACKAGE_API_URL}/generic/${CI_PROJECT_NAME}"
7
8
.rules:
9
rules:
10
- if: &merge-request-criteria $CI_PIPELINE_SOURCE == "merge_request_event"

Now the actual job1:

1
...
2
3
.publish:
4
image:
5
name: goreleaser/goreleaser:v1.17.0
6
entrypoint: [""]
7
variables:
8
GIT_DEPTH: 0
9
10
merge-release:
11
stage: publish
12
extends: .publish
13
before_script:
14
- apk add jq
15
script:
16
- goreleaser release --snapshot --clean
17
- cd dist/
18
- |
19
echo "Finding out current release version"
20
export RELEASE_VERSION=$(find . -type f -iname "*checksums.txt" | cut -d "_" -f 2)
21
- |
22
echo "Finding out package ID for current merge request at version '${RELEASE_VERSION}'"
23
export MR_PACKAGE_ID=$(curl -s --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "$PACKAGE_API_URL" |\
24
jq -r ".[] | select(.version==\"$RELEASE_VERSION\").id")
25
- |
26
echo "Deleting potentially existing package for current merge request at package ID '${MR_PACKAGE_ID}'"
27
curl --request DELETE --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${PACKAGE_API_URL}/${MR_PACKAGE_ID}" > /dev/null 2>&1
28
- |
29
find . -type f -iname "*${RELEASE_VERSION}*" -exec bash -c \
30
'echo "Uploading \"{}\" to package registry"; \
31
curl -s --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" --upload-file {} ${PACKAGE_REGISTRY_URL}/${RELEASE_VERSION}/{}; \
32
echo -e "\n"' \;
33
rules:
34
- if: *merge-request-criteria

First I created a generic .publish job, which by itself can never run (it can only be referenced through ‘extends’). The merge-release job uses it as its base to get the correct image and any variables. In before_script I install jq just for the purposes of being able to reliably query for the correct package ID (as shown on line no. 25). I’m not enthused about having it download and install anything at the start of every job, but I’m not at the point yet where I want to create a separate Docker image just for this.

Afterwards 5 steps are performed by the job:

  1. it uses the goreleaser release --snapshot command to create a release, but not publish it
  2. it finds out the current release version (e.g. 1.0.1-1)
  3. for that release version it founds out the package ID it needs to wipe beforehand2
  4. it deletes the package with the found ID or does nothing if no package existed at that ID
  5. it uploads every file from the dist/ directory, which includes the release version in its name3

The astute among you may have noticed the GITLAB_TOKEN variable. This is different from a CI_JOB_TOKENvariable, which would effectively mimic you or whoever has permissions to make changes in the repository. As explained by GoReleaser’s documentation this specific variable is a project access token with api scope. I also added the Maintainer role to this token so it would have enough permissions to perform the deletions required in the merge-release job.

Here’s what an example run of this job would look like:

1
$ apk add jq
2
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
3
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
4
(1/2) Installing oniguruma (6.9.8-r0)
5
(2/2) Installing jq (1.6-r2)
6
Executing busybox-1.35.0-r29.trigger
7
OK: 472 MiB in 71 packages
8
$ goreleaser release --snapshot --clean
9
• starting release...
10
• loading config file file=.goreleaser.yml
11
• loading environment variables
12
• using token from "$GITLAB_TOKEN"
13
• getting and validating git state
14
• couldn't find any tags before "v1.0.0"
15
• building... commit=067622c920472ae1266dcfc2def20ce9174630d2 latest tag=v1.0.0
16
• pipe skipped reason=disabled during snapshot mode
17
• parsing tag
18
• setting defaults
19
• snapshotting
20
• building snapshot... version=1.0.1-1
21
• checking distribution directory
22
• loading go mod information
23
• pipe skipped reason=not a go module
24
• build prerequisites
25
• writing effective config file
26
• writing config=dist/config.yaml
27
• building binaries
28
• building binary=dist/client_windows_amd64_v1/client.exe
29
• building binary=dist/client_linux_amd64_v1/client
30
• took: 2m17s
31
• archives
32
• creating archive=dist/project_name_client_1.0.1-1_linux_amd64.tar.gz
33
• creating archive=dist/project_name_client_1.0.1-1_windows_amd64.zip
34
• took: 6s
35
• calculating checksums
36
• storing release metadata
37
• writing file=dist/artifacts.json
38
• writing file=dist/metadata.json
39
• release succeeded after 2m22s
40
$ cd dist/
41
$ echo "Finding out current release version" # collapsed multi-line command
42
Finding out current release version
43
$ echo "Finding out package ID for current merge request at version '${RELEASE_VERSION}'" # collapsed multi-line command
44
Finding out package ID for current merge request at version '1.0.1-1'
45
$ echo "Deleting potentially existing package for current merge request at package ID '${MR_PACKAGE_ID}'" # collapsed multi-line command
46
Deleting potentially existing package for current merge request at package ID '252'
47
$ find . -type f -iname "*${RELEASE_VERSION}*" -exec bash -c \ # collapsed multi-line command
48
Uploading "./project_name_1.0.1-1_checksums.txt" to package registry
49
{"message":"201 Created"}
50
Uploading "./project_name_client_1.0.1-1_linux_amd64.tar.gz" to package registry
51
{"message":"201 Created"}
52
Uploading "./project_name_client_1.0.1-1_windows_amd64.zip" to package registry
53
{"message":"201 Created"}
54
Cleaning up project directory and file based variables 00:00
55
Job succeeded

Footnotes

  1. Always pin your Docker images, kids.

  2. This wiping is necessary to combat the aforementioned append-only upload functionality.

  3. Since only the archives and the checksum include the version, this seemed like a simple enough heuristic.

« PreviousNext »