infrastructure/gitlab-ci
2
0
Fork 0
Code Issues 17 Pull Requests 1 Actions Projects Releases 21 Wiki Activity

docs: expand

Browse Source 
!59
This commit is contained in:
Jon Lockwood
2023-06-10 15:55:44 +09:30
parent 1a03324ecc
commit d9b303edc1
7 changed files with 414 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

126
docs/projects/gitlab-ci/ansible_playbook.md
View File

@ -11,29 +11,22 @@ This job enables you to run an Ansible playbook within the Gitlab CI/CD environm
There is also an additional job defined that enables you to specify a project to run an automated update of it's git submodules. This is useful if the project you create the job in, is used as a git submodule in another project. On pushing to the development branch, this job triggers the `Git.Submodules.Update.Chores` job that updates the specified projects git submodules.
This job provides the following badge:
- None
## Dependencies
- **Mandatory** file `.nfc_automation.yaml` see [Documentation](../git_configuration/submodule/) for file details.
- **Mandatory** In addition to the previous dependency, this requirement is for the project specified in the trigger job. The following must be added to the `.gitlab-ci.yml` file in that project.
``` yaml
include:
- project: nofusscomputing/projects/gitlab-ci
ref: master
file:
- .gitlab-ci_common.yaml
- automation/template/automagic.gitlab-ci.yaml
## your .gitlab-ci.yml changes
To use this job add the following to your `.gitlab-ci.yml` file
- **Mandatory** In addition to the previous dependency, this requirement is for the project specified in the trigger job. The following must be added to the `.gitlab-ci.yml` file in that project.
``` yaml
include:
- project: nofusscomputing/projects/gitlab-ci
ref: master
file:
- .gitlab-ci_common.yaml
- automation/template/automagic.gitlab-ci.yaml
```
To use the ansible playbook job add the following to your `.gitlab-ci.yml` file.
``` yaml
@ -56,7 +49,7 @@ Ansible Job:
```
To run the trigger job to update another projects git submodules.
To run the trigger job to update another projects git submodules. add the following to your `.gitlab-ci.yml` file
``` yaml
@ -77,26 +70,91 @@ Docker_Mail.Submodule.Deploy:
```
!!! Tip
You can optionally override the stage by specifying the job you define
In addition to the variables below, you can also specify any additional Environment variables for use by Ansible within the container. Refer to the [Ansible configuration documentation](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#common-options) for further details
## CI/CD Variables required
## Job: `.ansible_playbook`
| var name | Description |
|:----:|:----|
| ansible_playbook | ***Mandatory** The ansible playbook to run.* |
| ansible_tags | ***Optional** Tags to limit task scope* |
| PIPELINE_RUN_TRIGGER | ***Optional** if the job can be triggered by pipeline.* |
| PIPELINE_RUN_SCHEDULE | ***Optional** if the job can be triggered by schedule.* |
| SUBMODULE_UPDATE_TRIGGER_PROJECT | ***Mandatory** and only used for the Submodule.Deploy job. and is the project to trigger the `git.submodule.update.chores` job.* |
!!! Tip
In addition to the variables above, you can also specify any additional Environment variables for use by Ansible. Refer to the [Ansible configuration documentation](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#common-options) for further details
This job runs an Ansible playbook using the `nofusscomputing/ansible-ee:dev` Docker image.
## Job Workflow
### Stage: Chores
This job will run the specified Ansible playbook using our [Ansible Execution Environment](../execution_environment) container. This container contains all of our playbooks and roles.
This job is responsible for executing an Ansible playbook. It can be customized by setting the following variables:
#### Variables
- `ansible_inventory`: The Ansible inventory file.
- `ansible_playbook`: The name of the Ansible playbook file.
- `ansible_tags`: The tags to be applied during playbook execution.
### Rules
- Rule 1: If the `NFC_AUTO_JOBS` variable is set to `"false"`, the job will never run.
- Rule 2: If the pipeline is triggered by a schedule and `PIPELINE_RUN_SCHEDULE` is set to `"true"`, the job will run only if the `.nfc_automation.yaml` file exists.
- Rule 3: If the pipeline is triggered by an API call, another pipeline, a trigger, or a parent pipeline, and `PIPELINE_RUN_TRIGGER` is set to `"true"`, the job will run only if the `.nfc_automation.yaml` file exists.
- Rule 4: If the pipeline is triggered by a push to the `development` branch, the job will run only if the `.nfc_automation.yaml` file exists. see [Documentation](../git_configuration/submodule/) for file details.
- Rule 5: This rule prevents the job from running under any circumstances.
## Job: `.ansible_playbook_git_submodule`
This job extends the `.ansible_playbook` job and is specifically used for running the `git_configuration.yaml` playbook with the `submodule` tags.
### Stage: Chores
This job is responsible for executing the `git_configuration.yaml` playbook with the `submodule` tags.
#### Variables
- `ansible_playbook`: The name of the Ansible playbook file (`git_configuration.yaml`).
- `ansible_tags`: The tags to be applied during playbook execution (`submodule`).
### Rules
- Rule 1: If the `NFC_AUTO_JOBS` variable is set to `"false"`, the job will never run.
- Rule 2: If the pipeline is triggered by a schedule and `PIPELINE_RUN_SCHEDULE` is set to `"true"`, the job will run only if the `.nfc_automation.yaml` file exists.
- Rule 3: If the pipeline is triggered by an API call, another pipeline, a trigger, or a parent pipeline, and `PIPELINE_RUN_TRIGGER` is set to `"true"`, the job will run only if the `.nfc_automation.yaml` file exists. see [Documentation](../git_configuration/submodule/) for file details.
- Rule 4: If the pipeline is triggered by a push to the `development` branch, the job will run only if the `.nfc_automation.yaml` file exists.
- Rule 5: This rule prevents the job from running under any circumstances.
## Job: `.submodule_update_trigger`
This job triggers a pipeline in another project.
### Stage: Publish
This job is responsible for triggering a pipeline in another project.
#### Variables
- `PIPELINE_RUN_TRIGGER`: The flag to indicate if the triggered pipeline should run (`true`).
### Rules
- Rule 1: If the pipeline is triggered by a push to the `master` or `development` branch, and there is no associated tag, the job will run on successful completion.
- Rule 2: This rule prevents the job from running under any other circumstances.
## Artifacts

69
docs/projects/gitlab-ci/conventional_commits.md
View File

@ -47,7 +47,7 @@ Run these commands once you have the information above.
``` bash
git format-patch {original_commit}..HEAD -o diff-patches
git format-patch {original_commit}..HEAD -o ../diff-patches
git reset {source_commit} --hard
@ -57,7 +57,7 @@ Now, navigate to the `diff-patches` folder, open up the offending patch (commit)
``` bash
git am diff-patches/*.patch
git am ../diff-patches/*.patch
```
@ -72,18 +72,9 @@ Now push your changes upstream.
| *Ensure that all of your commits were exported prior to reseting the branch and when re-applying, that all of your commits were applied correctly* |
## Conventional Commits Admin Manual Manual
## GitLab CI Template - `.conventional_commit`
This job checks commit messages on a branch and the merge request title for validity against the [conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/)
This job provides the following badge:
- None
### Dependencies
- None
This GitLab CI template, named `.conventional_commit`, is designed to validate conventional commits within a GitLab CI/CD pipeline. It follows predefined rules to ensure that commit message conventions are met.
### your .gitlab-ci.yml changes
@ -105,20 +96,55 @@ include:
```
## CI/CD Variables required
### Job Description
| var name | Description |
|:----:|:----|
| MR_ACCESS_TOKEN | *only required if you are accessing a private repository.* <br>This token is a user access token that as a minimum requires read-only access to the api to fetch the projects merg requests. |
In addition to the required variables above, the commitizen config file `.cz.yaml` must exist for the job to run.
The `.conventional_commit` job performs various tasks related to validating conventional commits. It runs in the `validation` stage of the pipeline and uses the `python:3.6-slim` Docker image.
### Job Workflow
### Variables
- `DEFAULT_ROOT_DIR`: The default root directory is set as `./gitlab-ci`.
- `MR_ACCESS_TOKEN`: The access token for the merge request. If not defined, it falls back to `CI_JOB_TOKEN`.
- `JOB_ROOT_DIR`: The root directory for the job. If not defined, it falls back to `DEFAULT_ROOT_DIR`.
- `MY_PROJECT_ID`: The custom project ID. If not defined, it falls back to `CI_PROJECT_ID`.
### Stages
- `validation`: The job is assigned to the `validation` stage.
### Script
The script section contains the actions performed during the job execution. These actions include creating directories, setting up variables, preparing the Python environment, and executing commands related to validating conventional commits.
The complete script can be found in the GitLab CI template file.
### Artifacts
The job generates artifacts that are stored for a period of 3 days. The artifacts include the following paths:
- `$CI_PROJECT_DIR/artifacts/*`
- `$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/*.junit.xml`
### Rules
The job follows certain rules to determine when it should be executed:
- If the variable `$JOB_STOP_CONVENTIONAL_COMMITS` is true, the job will not run.
- If the branch is pushed and a commit is made, and there is no tag associated with the commit, and the pipeline source is "push", the job will run if `.cz.yaml` file exists.
- In all other cases, the job will not run.
This GitLab CI template provides a convenient way to validate conventional commits and enforce commit message conventions within your CI/CD pipelines.
## Gitlab job Definition
@ -129,6 +155,3 @@ When you include this definition the following makes up the job definition
--8<-- "conventional_commits/.gitlab-ci.yml"
```
!!! Note
Docs Still under development

56
docs/projects/gitlab-ci/docker_build.md Normal file
View File

@ -0,0 +1,56 @@
---
title: docker build Gitlab CI/CD jobs Template
description: How to use No Fuss Computings gitlab-ci template for auto creation of CI/CD joobs.
date: 2023-06-10
template: project.html
about: https://gitlab.com/nofusscomputing/projects/gitlab-ci
---
This documentation provides an overview and explanation of the GitLab CI/CD YAML template. The template defines a job named `.build_docker_container` that builds a Docker container using the specified configuration.
### Stage: build
This job builds a Docker container using the `nofusscomputing/docker-buildx-qemu:dev` image and the `docker:23-dind` service.
#### Variables
- `DOCKER_IMAGE_BUILD_NAME`: The name of the Docker image to be built (`$CI_PROJECT_NAME`).
- `DOCKER_IMAGE_BUILD_REGISTRY`: The registry where the Docker image will be pushed (`$CI_REGISTRY_IMAGE`).
- `DOCKER_IMAGE_BUILD_TAG`: The tag to be applied to the Docker image (`$CI_COMMIT_SHA`).
#### Rules
- Rule 1: The job runs if the pipeline is triggered by a push to the `master` branch and a `dockerfile` or `dockerfile.j2` file exists.
- Rule 2: The job runs if the pipeline is triggered by a push to the `development` branch and a `dockerfile` or `dockerfile.j2` file exists. It also checks for changes in the `dockerfile`, `dockerfile.j2`, or `includes/` directory compared to the `master` branch.
- Rule 3: The job runs if the pipeline is triggered by a push to a branch other than `master` or `development` and a `dockerfile` or `dockerfile.j2` file exists. It also checks for changes in the `dockerfile`, `dockerfile.j2`, or `includes/` directory compared to the `development` branch.
- Rule 4: The job never runs.
#### Script
The script performs the following steps:
1. Updates the binary formats and enables execution of other binary formats in the kernel.
2. Creates a Docker buildx builder and sets it as the active builder.
3. Builds a multi-arch Docker image if the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is specified. It applies labels to the image and pushes it to the specified registry. It also inspects the image and performs cleanup by removing additional unknown images from the container registry.
4. Builds a Docker image if the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is not specified. It applies labels to the image and pushes it to the specified registry.
## gitlab-ci.yml definition
``` yaml title=".gitlab-ci.yml" linenums="1"
--8<-- "docker/build.gitlab-ci.yaml"
```

102
docs/projects/gitlab-ci/docker_publish.md Normal file
View File

@ -0,0 +1,102 @@
---
title: docker publish Gitlab CI/CD jobs Template
description: How to use No Fuss Computings gitlab-ci job template to publish a docker image to docker hub.
date: 2023-06-10
template: project.html
about: https://gitlab.com/nofusscomputing/projects/gitlab-ci
---
This GitLab CI template is designed to publish Docker images to Docker Hub. It contains a job called "Publish Docker Image to Docker Hub" that handles the image publishing process.
## Stage
- `publish`
## Variables
The following variables are used in the job:
- `DOCKER_IMAGE_BUILD_NAME`: The name of the Docker image to build. By default, it uses the GitLab CI project name.
- `DOCKER_IMAGE_BUILD_REGISTRY`: The registry for the Docker image build. By default, it uses the GitLab CI registry image.
- `DOCKER_IMAGE_BUILD_TAG`: The tag for the Docker image build. By default, it uses the GitLab CI commit SHA.
- `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS`: A comma-separated list of available platforms for the Docker image build. Supported platforms include: `linux/amd64`, `linux/amd64/v2`, `linux/amd64/v3`, `linux/arm64`, `linux/riscv64`, `linux/ppc64`, `linux/ppc64le`, `linux/s390x`, `linux/386`, `linux/mips64le`, `linux/mips64`, `linux/arm/v7`, `linux/arm/v6`.
- `CI_REGISTRY_USER`: The username for logging in to the GitLab CI registry.
- `CI_REGISTRY_PASSWORD`: The password for logging in to the GitLab CI registry.
- `NFC_DOCKERHUB_USERNAME`: The username for logging in to Docker Hub.
- `NFC_DOCKERHUB_TOKEN`: The access token or password for logging in to Docker Hub.
- `DOCKER_IMAGE_PUBLISH_REGISTRY`: The registry for publishing the Docker image to Docker Hub.
- `DOCKER_IMAGE_PUBLISH_NAME`: The name of the Docker image for publishing to Docker Hub.
- `DOCKER_IMAGE_PUBLISH_URL`: The URL to access the published Docker image on Docker Hub.
## Services
- `docker:23-dind`: Runs Docker in Docker (DinD) service with version 23.
## Script
The job executes the following steps in the `script` section:
1. Logs in to the GitLab CI registry using the provided credentials (`CI_REGISTRY_USER` and `CI_REGISTRY_PASSWORD`).
2. If the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is specified, it iterates over the platforms and inspects the Docker image using `docker buildx imagetools inspect`.
3. If the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is not specified, it pulls the Docker image from the specified registry.
4. Logs in to Docker Hub using the provided credentials (`NFC_DOCKERHUB_USERNAME` and `NFC_DOCKERHUB_TOKEN`).
5. Lists the Docker images using `docker image ls`.
6. Determines the appropriate tag for the Docker image based on the pipeline source and commit tag.
7. If the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is specified, it creates multi-arch images using `docker buildx imagetools create` and tags them with the appropriate tag.
8. If the `DOCKER_IMAGE_BUILD_TARGET_PLATFORMS` variable is not specified, it tags the Docker image with the appropriate tag based on the pipeline source and commit tag.
9. Pushes the Docker image to Docker Hub.
10. Logs out of Docker Hub.
## Environment
The job sets the following environment variables:
- **Name**: DockerHub
- **URL**: The URL to access the published Docker image on Docker Hub.
## Rules
The job is controlled by the following rules:
- Runs when the pipeline is triggered by a Git tag and there is no associated branch.
- Runs when the pipeline is triggered by a push to the `master` branch and there is a Dockerfile present.
- Runs when the pipeline is triggered by a push to the `development` branch, there are changes in the Dockerfile or the `includes/` directory compared to the `master` branch, and it allows failure.
- Never runs explicitly.
## gitlab-ci.yml definition
``` yaml title=".gitlab-ci.yml" linenums="1"
--8<-- "docker/build.gitlab-ci.yaml"
```

159
docs/projects/gitlab-ci/gitlab_release.md
View File

@ -12,7 +12,7 @@ about: https://gitlab.com/nofusscomputing/projects/gitlab-ci
All commit messages must be in [conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/) and have a footer with a gitlab reference. The reference **must** be either a merge request or a gitlab issue. (format i.e. `!1` or `#2` *using the correct reference number*).
### fixing commit messages (suggestion)
### fixing commit messages
If only the last commit is the commit with an error just use `git commit --amend` and edit your commit message to be in the correct format and save. now push your changes.
@ -27,7 +27,7 @@ Run these commands once you have the information above.
``` bash
git format-patch {original_commit}..HEAD -o diff-patches
git format-patch {original_commit}..HEAD -o ../diff-patches
git reset {source_commit} --hard
@ -37,7 +37,7 @@ Now, navigate to the `diff-patches` folder, open up the offending patch (commit)
``` bash
git am diff-patches/*.patch
git am ../diff-patches/*.patch
```
@ -52,31 +52,65 @@ Now push your changes upstream.
| *Ensure that all of your commits were exported prior to reseting the branch and when re-applying, that all of your commits were applied correctly* |
## Gitlab Release - Developer Manual
## GitLab CI Template - Developer Manual
## Job: gitlab_release
This job bumps the version, updates the changelog, creates a git tag and creates a gitlab release. The git tag and release title use [semantic versioning](https://semver.org/). for this job to function correctly a `.cz.yaml` is required in the root of the repository. this file contains the [commitizen](https://github.com/commitizen-tools/commitizen) config and the version details.
This job has the following workflow:
- `master` Branch
> Automatically increment the version
- `development` Branch
> Manual CI job made available to increment the version. (release-candidate increment only)
| :octagonal_sign: Danger |
|:----|
| *If prior to merging to the master branch you do a version increment, and there are no commits prior to merging. the job will not increment the version and the job will fail. it is recommended that you only do a version increment on the `development` branch if you are going to commit further changes to the `development` branch* |
!!! Alert
*If prior to merging to the master branch you do a version increment, and there are no commits prior to merging. the job will not increment the version and the job will fail. it is recommended that you only do a version increment on the `development` branch if you are going to commit further changes to the `development` branch*
This job provides the following badge:
### Stage
- None
`release`
### Dependencies
### Image
- None
The job uses the `registry.gitlab.com/gitlab-org/release-cli:latest` image.
### Variables
The job does not use any additional variables.
### Explanation
The `gitlab_release` job is responsible for creating releases and tags for the GitLab repository. It follows a specific release workflow and utilizes the `release-cli` tool to automate the release process.
### Steps
1. Set ROOT_DIR variable: Sets the `ROOT_DIR` variable based on the value of `JOB_ROOT_DIR`.
2. Create necessary directories: Creates necessary directories for storing artifacts and tests.
3. Install dependencies: Updates the package manager and installs Git and Python 3. Sets up the Python environment by installing required packages.
4. Clone repository: Clones the repository using the provided authentication token and checks out the `development` branch.
5. Configure Git: Configures Git settings for the release process.
6. Perform release steps: Executes release-related steps, such as running a custom command (`$MY_COMMAND`), generating the release changelog, and tagging the release.
7. Push changes: Pushes the changes to the GitLab repository.
8. Cleanup: Removes the cloned repository.
### Rules
- The job is never triggered when `$JOB_STOP_GITLAB_RELEASE` is true.
- The job is never triggered when the commit author is `nfc_bot <helpdesk@nofusscomputing.com>`.
- If the commit is pushed to the `master` branch, the job is only triggered on successful pipeline execution and failure is not allowed.
- If the commit is pushed to the `development` branch, the job is triggered manually and failure is allowed.
### your .gitlab-ci.yml changes
@ -105,40 +139,70 @@ Gitlab Release:
> if you wish to run any commands you can add them to variable `MY_COMMAND`. The custom command will run under shell `/bin/sh`. This command is set to run before the version bump commit is conducted so any changes you wish to add as part of the version bump, you can do here as long as you `git add {changed file name}`.
### CI/CD Variables required
| var name | Description |
|:----:|:----|
| GIT_COMMIT_TOKEN | *this must be a personal token that has write access to the repository* |
| CHANGELOG_FOOTER_REFERENCES | ***Optional** If set to `False` the changelog will not output gitlab references for each entry of the changelog. If this variable is set globally, it will also prevent the creation of the CI job to validate a users commits as having gitlab references.* |
## Job: commit_footer_refs
### Job Workflow
### Stage
This CI job's workflow is:
1. updates the changelog from the commits
1. commit the changelog to git
1. adds a `git tag` to the changelog commit.
1. pushes the change back to the repo
1. creates a git release from the `git tag`
| :octagonal_sign: **NOTE** |
|:----|
| *If the user has forked the branch, they must keep the development brnach synced with the main repo. If they **don't** the CI job 'commit footer refs' will fail as it will not be able to fetch the parent (`development`) hash of the branch.* |
`validation`
### Artifacts
### Image
- `ci commit footer`
> $CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/$CI_JOB_NAME.junit.xml
The job uses the `python:3.6-slim` image.
- `Gitlab Release`
> None
### Variables
- `DEFAULT_ROOT_DIR`: The default root directory path.
### Explanation
The `commit_footer_refs` job validates the commit footer references in the GitLab repository. It checks if the commit messages adhere to the conventional commit format and generates a JUnit XML report.
### Conventional Commits
Conventional commits follow a specific format for commit messages, consisting of a type, optional scope, and a message. The format is as follows:
```
<type>(<scope>): <message>
```
- The `<type>` represents the nature of the changes, such as `feat` for a new feature, `fix` for a bug fix, `docs` for documentation changes, and so on.
- The `<scope>` (optional) provides additional context for the commit, indicating the module, component, or area of code being modified.
- The `<message>` contains a concise and descriptive summary of the changes.
The commit footer can contain additional information, such as references to issues, feature requests, or pull requests.
### Steps
1. Create necessary directories: Creates necessary directories for storing artifacts and test results.
2. Set ROOT_DIR variable: Sets the `ROOT_DIR` variable based on the value of `JOB_ROOT_DIR`.
3. Install dependencies: Updates the package manager and installs Git and the required Python packages for the commit footer validation.
4. Clone repository: Clones the repository and checks out the specified branch.
5. Run commit_footer script: Executes the `commit_footer` script to validate the commit footer references.
6. Generate artifacts: Generates a JUnit XML report for the test results.
### Rules
- The job is never triggered when `$JOB_STOP_CONVENTIONAL_COMMITS` is true.
- The job is never triggered when `CHANGELOG_FOOTER_REFERENCES` is false.
- The job is always triggered when the commit is not pushed to the master or development branch, and a .cz.yaml file is present, indicating the usage of conventional commits.
- The job is never triggered otherwise.
## Gitlab job Definition
@ -150,6 +214,3 @@ When you include this definition the following makes up the job definition
--8<-- "gitlab_release/.gitlab-ci.yml"
```
!!! Note
Docs Still under development

3
docs/projects/gitlab-ci/templates/automagic.md
View File

@ -18,10 +18,11 @@ This template is designed to autodetect which jobs should be created. By includi
Docs Still under development
## gitlab-ci.yml definition
``` yaml title=".gitlab-ci.yml" linenums="1"
--8<-- "template/automagic.gitlab-ci.yaml"
```
```

6
mkdocs.yml
View File

@ -20,6 +20,8 @@ nav:
- Gitlab CI:
- Build:
- projects/gitlab-ci/docker_build.md
- projects/gitlab-ci/mkdocs-build.md
@ -33,6 +35,10 @@ nav:
- projects/gitlab-ci/yaml_lint.md
- Publish:
- projects/gitlab-ci/docker_publish.md
- Templates:
- projects/gitlab-ci/templates/index.md