infrastructure/gitlab-ci
2
0
Fork 0
Code Issues 17 Pull Requests 1 Actions Projects Releases 21 Wiki Activity

Merge branch 'update-readme' into 'development'

Browse Source 
docs(readme): added how to use repo

See merge request nofusscomputing/projects/gitlab-ci!5
This commit is contained in:
Jon Lockwood
2021-08-11 06:37:28 +00:00
parent f4ee55daa5 0653766c93
commit a629e39f59
20 changed files with 547 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
.gitlab-ci.yml
View File

@ -8,9 +8,24 @@ stages:
- sync
include:
- local: $JOB_ROOT_DIR/yaml_lint/.gitlab-ci.yml
- local: $JOB_ROOT_DIR/conventional_commits/.gitlab-ci.yml
- local: $JOB_ROOT_DIR/git_push_mirror/.gitlab-ci.yml
- local: $JOB_ROOT_DIR/gitlab_release/.gitlab-ci.yml
- local: $JOB_ROOT_DIR/python/.gitlab-ci.yml
PyLint:
extends:
- .PyLint
image: python:3.6-slim
gilab-ci.yml Lint (python 3.6):
variables:
YAML_LINT_PATH: "/*/.gitlab-ci.yml"
extends:
- .yaml_lint_defaults
image: python:3.6-slim
Gitlab Release:
@ -20,7 +35,6 @@ Gitlab Release:
- .gitlab_release
Github (Push --mirror):
variables:
GIT_SYNC_URL: "https://$GITHUB_USERNAME_ROBOT:$GITHUB_TOKEN_ROBOT@github.com/NoFussComputing/gitlab-ci.git"

0
.gitlab/.gitkeep Normal file
View File

0
.gitlab/issue_templates/.gitkeep Normal file
View File

84
.gitlab/issue_templates/new_job.md Normal file
View File

@ -0,0 +1,84 @@
## :new: Ansible Role - {new ansible role name}
<!--
- provide a summary of the role and what you propose including what the role will do. ensure all explainations are detailed.
- include any templates here using the collapsable sections. remove/add as many as is required and name as to what it is.
- you can add/delete any details section as required.
- emojis ref: https://github.com/yodamad/gitlab-emoji
-->
<details>
<summary>:clipboard: template filename.extension</summary>
``` yaml
your yaml playbook here
```
</details>
### Playbook usage
<!--
provide examples on how this role would be used in a playbook.
include a playbook example i.e.
<details>
<summary>:clipboard: example playbook</summary>
``` yaml
your yaml playbook here
```
</details>
-->
<!-- provide a summary here -->
<details>
<summary>:clipboard: example playbook</summary>
``` yaml
your yaml playbook here
```
</details>
### :ledger: Documentation
<!-- provide a summary of what is required or even better contribute by writing the docs here -->
<details>
<summary>:clipboard: contribution - Documentation</summary>
<!-- if you are contributing docs write them here. -->
</details>
## :link: Links
<!-- Add a list of links to items relevant to this issue-->
- link 1
- issue #number
## :ballot_box_with_check: Tasks
<!--
don't delete tasks from the list strike them through to denote not-applicable use double tidle '~~'. i.e. '~~[ ] a random task~~'
tasks are used as part of the development stage
-->
- [ ] ~Documentation Added
- [ ] (dev) Merge request opened and linked to this issue
<!-- don't adjust the below slash commands -->
/label ~"workflow::not started"
/label ~"category::feature"

0
.gitlab/merge_request_templates/.gitkeep Normal file
View File

34
.gitlab/merge_request_templates/default.md Normal file
View File

@ -0,0 +1,34 @@
## :books: Summary
<!-- your summary here emojis ref: https://github.com/yodamad/gitlab-emoji-->
-----
| :link: **Related Issue(s)** | :link: **Related Merge Request(s)** |
|:----:|:----:|
| <!-- example #1 <br> #2 <br> #3 etc. if none enter *none* --> | <!-- example !1 <br> !2 <br> !3 etc. if none enter *none* --> |
### :construction_worker: Tasks
<!-- dont remove tasks strike through including the checkbox by enclosing in double tidle '~~' -->
- [ ] merging to correct branch?
- [ ] issue linked to this MR?
- related issues:
- [ ] closed in a commit message
- [ ] Manually closed
- [ ] All related issues tasks complete?
<!-- don't adjust the below slash commands -->
/label ~"workflow::not started"
/label ~"category::feature"

40
.templates/README.md Normal file
View File

@ -0,0 +1,40 @@
# {CI Job Name}
Summary of job here
This job provides the following badge:
{A badge here}
## Dependencies
- {dependent job name}
## your .gitlab-ci.yml changes
To use this job add the following to your `.gitlab-ci.yml` file
``` yaml
variables:
VARNAME: "a var value"
stages:
- {new stage name}
include:
- local: CI/{job name}/.gitlab-ci.yml
```
## CI/CD Variables required
| var name | Description |
|:----:|:----|
| NEW VAR | this var does this and bka |
## Job Workflow
## Artifacts
## License
To view the license for this folder and any sub-folders, refer [here](https://gitlab.com/nofusscomputing/projects/gitlab-ci)

94
README.md
View File

@ -1,20 +1,104 @@
# Ansible Roles
# No Fuss Computing - Gitlab-CI
[![Gitlab build status - stable](https://img.shields.io/badge/dynamic/json?color=ff782e&label=Build%20%5B%20Stable%20%5D&query=0.status&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fpipelines%3Fref%3Dmaster&logo=gitlab&style=plastic)](https://gitlab.com/nofusscomputing/projects/gitlab-ci)
<div align="center">
[![Open Issues](https://img.shields.io/badge/dynamic/json?color=ff782e&logo=gitlab&style=plastic&label=Open%20Issues&query=%24.statistics.counts.opened&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fissues_statistics)](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/issues)
[![Gitlab build status - development](https://img.shields.io/badge/dynamic/json?color=ff782e&label=Build%20[%20Development%20]&query=0.status&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fpipelines%3Fref%3Ddevelopment&logo=gitlab&style=plastic)](https://gitlab.com/nofusscomputing/projects/gitlab-ci)
| Stable Branch |
[![Gitlab build status - stable](https://img.shields.io/badge/dynamic/json?color=ff782e&label=Build&query=0.status&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fpipelines%3Fref%3Dmaster&logo=gitlab&style=plastic)](https://gitlab.com/nofusscomputing/projects/gitlab-ci) |
[![PyLint Score](https://img.shields.io/badge/dynamic/json?&style=plastic&logo=python&label=PyLint%20Score&query=%24.PyLintScore&url=https%3A%2F%2Fgitlab.com%2Fnofusscomputing%2Fprojects%2Fgitlab-ci%2F-%2Fjobs%2Fartifacts%2Fmaster%2Fraw%2Fartifacts%2Fvalidation%2FPyLint%2Fbadge_pylint.json%3Fjob%3DPyLint)](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/jobs/artifacts/master/file/artifacts/validation/tests/gl-code-quality-report.html?job=PyLint) |
|:----|:----|:----|
| development Branch |
[![Gitlab build status - development](https://img.shields.io/badge/dynamic/json?color=ff782e&label=Build&query=0.status&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fpipelines%3Fref%3Ddevelopment&logo=gitlab&style=plastic)](https://gitlab.com/nofusscomputing/projects/gitlab-ci) |
[![PyLint Score](https://img.shields.io/badge/dynamic/json?&style=plastic&logo=python&label=PyLint%20Score&query=%24.PyLintScore&url=https%3A%2F%2Fgitlab.com%2Fnofusscomputing%2Fprojects%2Fgitlab-ci%2F-%2Fjobs%2Fartifacts%2Fdevelopment%2Fraw%2Fartifacts%2Fvalidation%2FPyLint%2Fbadge_pylint.json%3Fjob%3DPyLint)](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/jobs/artifacts/development/file/artifacts/validation/tests/gl-code-quality-report.html?job=PyLint) |
|:----|:----|:----|
</div>
This repository is hosted on [gitlab.com](https://gitlab.com/nofusscomputing/projects/gitlab-ci) and has a read-only copy hosted on [github.com](https://github.com/NoFussComputing/gitlab-ci).
links:
- [Issues](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/issues)
- [![Open Issues](https://img.shields.io/badge/dynamic/json?color=ff782e&logo=gitlab&style=plastic&label=Open%20Issues&query=%24.statistics.counts.opened&url=https%3A%2F%2Fgitlab.com%2Fapi%2Fv4%2Fprojects%2F28543717%2Fissues_statistics) Issues](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/issues)
- [Merge Requests (Pull Requests)](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/merge_requests)
> *To Do: expand on how repo setup and how to use.*
## Using this repository for your Gitlab CI/CD Jobs.
This repository has been designed as a central point for your repositories CI/CD jobs. By simply linking this repository to your repository and configuring, your CI/CD jobs will run as part of the build process, whilst keeping any CI/CD commits limited within your git history.
Each CI/CD job is contained within its own sub-folder. Each sub-folder has a readme specific to the job, which includes the details on how to implement, use etc.
### gitlab-ci layout
We use the following branches:
- `master` - Considered as the stable branch
- `development` considered as unstable
We also tag each branch to denote the version of release. We use our own repo to do the version increment automagically in line with [semantic versioning](https://semver.org/).
### CI Stages
The CI stages for these jobs are as follows, and in the order expected by the jobs:
- validation
> validation of files, commits, Merge Request titles, code quality, license checks.
- build
> build any binaries or files that would be used in the later stages .
- prepare
> any jobs that must run after build but before testing. for example a docker image build that includes artifacts from the build job
- test
> unit, functional, integration and any other tests.
- release
> git tagging and creating a gitlab release. Also includes adding build artifacts to gitlab registry.
- sync
> repository synchronization, i.e. push mirror to github.
- publish
> placement of build objects to external sources
### Artifacts
Any artifacts by jobs will be created in folders named after the stage.
preference is placed on jobs to output JUnit.xml test reports. This is because they are visible in merge requests.
It is recommended that you set-up this repo as a git sub-module to your repo and that you configure it to a set commit/tag. This ensures that any change to `gitlab-ci` repo, does not effect your CI/CD jobs.
run the following commands:
``` bash
git submodule add -b {ref} https://gitlab.com/nofusscomputing/projects/gitlab-ci.git gitlab-ci
git submodule update --remote
```
| :bulb: Tip |
|:-----|
| NOTE: `{ref}` should be replaced with the branch name, `master` is the stable branch and recommended. by default the sub-module will be created in folder `gitlab-ci`, it is recommended that you **don't** change this folder name.
You can also substitute the gitlab url with the github url `https://github.com/NoFussComputing/gitlab-ci.git` for the submodule if you desire. this repo is auto-synced with github on each change to the repo. |
After each `git submodule update --remote` you will have to commit the sub-module update to your repo. Suggested commands as follows:
``` bash
git add .gitmodules
git add gitlab-ci
git commit -m "ci(gitlab-ci): updated to use version x
{your reason here or explain what is provided/changed}"
```
Then push the changes to your source.
## Contributing

2
git_push_mirror/.gitlab-ci.yml
View File

@ -18,5 +18,3 @@
- if: '$CI_COMMIT_BRANCH == "development"'
when: always

2
gitlab_release/python-module/commit_footer/commits.py
View File

@ -13,7 +13,7 @@ class Commits:
self._repository = git.Repo(os.getcwd())
self._failed = []
merge_base = self._repository.merge_base('development', 'changelog-footer-toggle')
merge_base = self._repository.merge_base('development', self._repository.active_branch)
self._merge_base = str(merge_base[0])
self.fetch_all()

49
python/.gitlab-ci.yml Normal file
View File

@ -0,0 +1,49 @@
.pylint_badge: &pylint_badge |
echo "{
\"PyLintScore\": \"$PyPIScore\"
}" > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/badge_pylint.json"
.PyLint:
variables:
PYLINT_PATH: "/*/python-module/*/*.py"
stage: validation
before_script:
- if [ "0$JOB_ROOT_DIR" == "0" ]; then ROOT_DIR=gitlab-ci; else ROOT_DIR=$JOB_ROOT_DIR ; fi
- echo "[DEBUG] ROOT_DIR[$ROOT_DIR]"
- mkdir -p "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME"
- mkdir -p "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests"
- export PYTHON_VERSION=`python -c 'import sys; version=sys.version_info[:3]; print("{0}.{1}.{2}".format(*version))'`
- apt update
- apt install --no-install-recommends -y git
- python3 -m venv env
- . env/bin/activate
- pip install --upgrade pip
- pip install -r $ROOT_DIR/python/requirements.txt
- mkdir $PYTHON_VERSION
script:
- PYLINT_PATH=$ROOT_DIR$PYLINT_PATH
- echo "[DEBUG] PYLINT_PATH[$PYLINT_PATH]"
- python3 -m pylint --exit-zero --output-format=pylint_gitlab.GitlabCodeClimateReporter $PYLINT_PATH > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/gl-code-quality-report.json"
- python3 -m pylint --exit-zero --output-format=pylint_gitlab.GitlabPagesHtmlReporter $PYLINT_PATH > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/gl-code-quality-report.html"
- PyPIScore=$(python3 -m pylint --exit-zero $PYLINT_PATH | sed -n 's/^Your code has been rated at \([-0-9./]*\).*/\1/p')
- *pylint_badge
after_script:
- echo deactivate
artifacts:
expire_in: 1 days
when: always
paths:
- "artifacts/*"
reports:
codequality: "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/gl-code-quality-report.json"
rules:
- if: '$CI_COMMIT_BRANCH'
when: always
- when: never

88
python/README.md Normal file
View File

@ -0,0 +1,88 @@
# Python
This folder (`python`) covers jobs for python development
These jobs provides the following badge:
- `PyLint` - code quality [![PyLint Score](https://img.shields.io/badge/dynamic/json?&style=plastic&logo=python&label=PyLint%20Score&query=%24.PyLintScore&url=https%3A%2F%2Fgitlab.com%2Fnofusscomputing%2Fprojects%2Fgitlab-ci%2F-%2Fjobs%2Fartifacts%2Fdevelopment%2Fraw%2Fartifacts%2Fvalidation%2FPyLint%2Fbadge_pylint.json%3Fjob%3DPyLint)](https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/jobs/artifacts/development/file/artifacts/validation/tests/gl-code-quality-report.html?job=PyLint)
Use the following MD to add a badge adjusting the variables and ensuring everything is on one line.
``` md
[![PyLint Score](https://img.shields.io/badge/dynamic/json?&style=plastic&logo=python&label=PyLint%20Score&query=%24.PyLintScore&url=https%3A%2F%2Fgitlab.com%2F
{project path}
%2F-%2Fjobs%2Fartifacts%2F
{branch}
%2Fraw%2Fartifacts%2Fvalidation%2FPyLint%2Fbadge_pylint.json%3Fjob%3D
{Job Name}
)](https://gitlab.com/
{project path}
/-/jobs/artifacts/
{branch}
/file/
artifacts/validation/tests/gl-code-quality-report.html
?job=
{Job Name}
)
```
| Variable | Description |
|:----|:----|
| `{project path}` | *project path, what's after gitlab.com/* |
| `{branch}` | *git branch to fetch the score from* |
| `{Job Name}` | *name of the gitlab-ci job for the linting* |
## Dependencies
- None
## your .gitlab-ci.yml changes
To add the `PyLint` job, add the following to your `.gitlab-ci.yml` file
``` yaml
stages:
- validation
include:
- remote: https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/raw/master/python/.gitlab-ci.yml
PyLint:
variables:
PYLINT_PATH: "/*/*.py"
extends:
- .PyLint
image: python:3.6-slim
```
## CI/CD Variables required
| var name | Description |
|:----:|:----|
| PYLINT_PATH | *The path you wish the linter to search for python files* |
## Job Workflow
- This job will lint any yaml file in the specified directory using the specified rules.
## Artifacts
- `$CI_PROJECT_DIR/artifacts` - Root artifact directory
- `$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/gl-code-quality-report.json` - Gitlab code quality report (displays in merge request)
- `$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/gl-code-quality-report.html` - html code quality report
## License
To view the license for this folder and any sub-folders, refer [here](https://gitlab.com/nofusscomputing/projects/gitlab-ci)

2
python/requirements.txt Normal file
View File

@ -0,0 +1,2 @@
pylint==2.9.6
pylint-gitlab==0.3.0

44
yaml_lint/.gitlab-ci.yml Normal file
View File

@ -0,0 +1,44 @@
.yaml_lint_defaults:
variables:
YAML_LINT_PATH: "/*.yml"
YAML_LINT_CONFIG: "/yaml_lint/.yamllint.yaml"
stage: validation
before_script:
- if [ "0$JOB_ROOT_DIR" == "0" ]; then ROOT_DIR=gitlab-ci; else ROOT_DIR=$JOB_ROOT_DIR ; fi
- echo "[DEBUG] ROOT_DIR[$ROOT_DIR]"
- mkdir -p "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME"
- mkdir -p "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests"
- export PYTHON_VERSION=`python -c 'import sys; version=sys.version_info[:3]; print("{0}.{1}.{2}".format(*version))'`
- apt update
- apt install --no-install-recommends -y git
- python3 -m venv env
- . env/bin/activate
- pip install --upgrade pip
- pip install -r $ROOT_DIR/yaml_lint/requirements.txt
- mkdir $PYTHON_VERSION
script:
- YAML_LINT_PATH=$ROOT_DIR$YAML_LINT_PATH
- echo "[DEBUG] YAML_LINT_PATH[$YAML_LINT_PATH]"
- YAML_LINT_CONFIG=$ROOT_DIR$YAML_LINT_CONFIG
- echo "[DEBUG] YAML_LINT_CONFIG[$YAML_LINT_CONFIG]"
- yamllint $YAML_LINT_PATH -f parsable -c $ROOT_DIR/yaml_lint/.yamllint.yaml > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/$PYTHON_VERSION-yaml-lint.log" 1>&1 || YAML_LINT=$?
- cat "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/$PYTHON_VERSION-yaml-lint.log" | ansible-lint-to-junit-xml > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/$PYTHON_VERSION-yaml-lint.junit.xml"
- if [ $YAML_LINT > 0 ]; then echo "YAML lint failed with $YAML_LINT"; exit $YAML_LINT; fi
after_script:
- echo deactivate
artifacts:
expire_in: 3 days
when: always
paths:
- "artifacts/*"
reports:
junit:
- "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/*.junit.xml"
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: never
- if: '$CI_COMMIT_BRANCH'
when: always
- when: never

26
yaml_lint/.yamllint.yaml Normal file
View File

@ -0,0 +1,26 @@
---
extends: default
rules:
braces: {max-spaces-inside: 1, level: error}
brackets: {max-spaces-inside: 1, level: error}
colons: {max-spaces-after: -1, level: error}
commas: {max-spaces-after: -1, level: error}
comments: disable
comments-indentation: disable
document-start: disable
empty-lines:
max: 2
max-start: 1
max-end: 1
level: error
hyphens: {level: error}
# indentation: disable
indentation:
spaces: consistent
key-duplicates: enable
line-length: disable
new-line-at-end-of-file: disable
new-lines: {type: unix}
trailing-spaces: disable
truthy: disable

52
yaml_lint/README.md Normal file
View File

@ -0,0 +1,52 @@
# Yaml Lint
This job does yaml linting when any commit is pushed to any branch.
This job provides the following badge:
- None
## Dependencies
- None
## your .gitlab-ci.yml changes
To use this job add the following to your `.gitlab-ci.yml` file
``` yaml
stages:
- validation
include:
- remote: https://gitlab.com/nofusscomputing/projects/gitlab-ci/-/raw/master/yaml_lint/.gitlab-ci.yml
Yaml Lint (python 3.6):
variables:
YAML_LINT_PATH: "roles/"
YAML_LINT_CONFIG: ".yamllint.yaml"
extends:
- .yaml_linter_defaults
image: python:3.6-slim
```
> You can use any python version you wish.
## CI/CD Variables required
| var name | Description |
|:----:|:----|
| YAML_LINT_PATH | *The path you wish the linter to search for yaml files* |
| YAML_LINT_CONFOG | *The path you have stored the yaml config file in* |
## Job Workflow
- This job will lint any yaml file in the specified directory using the specified rules.
## Artifacts
- `$CI_PROJECT_DIR/artifacts` - Root artifact directory
- `$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/tests/$PYTHON_VERSION-yaml-lint.junit.xml` - JUnit Test report
- `$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/$PYTHON_VERSION-yaml-lint.log` - Linter log
## License
To view the license for this folder and any sub-folders, refer [here](https://gitlab.com/nofusscomputing/projects/gitlab-ci)

2
yaml_lint/requirements.txt Normal file
View File

@ -0,0 +1,2 @@
yamllint
ansible-lint-to-junit-xml