What are CircleCi pipelines? Can they be used to trigger job with parameters?

问题

Documentation is spread out and it's a bit hard grasp how to make use of the pipeline concept in circle ci language? Also what's the point of pipelines and pipeline variables?

The following docs were useful but were far from enough for me to figure out how they actually work:

https://circleci.com/docs/2.0/pipelines
https://circleci.com/docs/2.0/build-processing (Enabling pipelines)
https://circleci.com/docs/2.0/pipeline-variables/#conditional-workflows (Pipeline variables and conditional workflows)
https://circleci.com/docs/api/v2/#circleci-api-pipeline (pipeline API docs)

回答1:

TLDR answer;

pipelines in circle ci are essentially triggers - triggering all workflows for a specific repo/branch/tag, including when circleci auto-triggers from a push/merge etc.

pipeline variables are apparently variables that require declaration in config.yml and default values. Their values can apparently only be set when triggering a "pipeline" via the 2.0 API.

Example trigger via 2.0 API [github]: (NOTE: Requires personal [not project] token)

curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{
  "branch": "feat",
  "parameters": {
    "image-tag": "4.8.2"
  }
}' https://circleci.com/api/v2/project/gh/<org>/<repo>/pipeline

Long answer

If you're anything like me, you might think of the word pipeline in the contect of CI as a hierarchy of jobs with dependencies between them and the ability to pass data from one step to the next. This feature exists in circle ci and is quite powerful (except that passing data is kind of awkward) but it's called workflow. Thus leaving the question of what circle ci mean with "pipeline", after some playing around with triggering it and looking at different parts of the docs, my conclusion is it should probably have been called "trigger" or "workflow execution" or something. It essentially describes the triggering of all workflows in a given branch/tag, including when that trigger is automatic through push/merge.

You can't use a pipeline to trigger a job with parameters, or even trigger a job at all, not unless you first wrap each such job in a pipeline and set up a conditional scheme to not run other workflows.

Why even go there?

I'm still not sure it's worth it to be honest, but basically the following drives us:

Don't stay on old tech (and soon to be discarded)
Orbs - a pretty nice way to DRY pipelines that only works on 2.1 config

The problem?

Use case 1: Essentially we have a job that needs to run after deployments in 3 different repos and instead of copy-pasting and maintaining the code in 3 places, we put the job in a 4th repo and using circleci API 1.1, triggered it with input parameters from different repos. Works great in 2.0 circleci config. It's also impossible to achieve in 2.1 config after circle ci introduced the regression to no longer support job triggering with parameters.

Use case 2: In some other cases it's useful to trigger via parameters if say: a proceeding job takes 2 hours and you don't want to wait to test out something in your pipeline.

Use case 3: Job 2 fails and you need to fix it before manually rerunning it with the output from job 1.

For simplicity lets look at a 2 job workflow:

+-------+      +-------+
| Job 1 |  ->  | Job 2 |
+-------+      +-------+

And we want to be able to:

Pass variables from job 1 to job 2
Execute job 2 via API, passing parameters to the job

In circleci API 1.1 it's a simple matter of passing the parameters to the job (via API) and they are automatically transformed to Environment variables. Simple.

With "pipelines" enabled and in 2.1 config there doesn't seem to be an elegant way of achieving this. Although this is somewhat alleviated by the existence of orbs and keeping full workflow in 1 repo (at least use case 1). There is however a bloated and hacky way of doing it with 2.1 pipelines which comes down to (POC example below):

Ensure there's a pipeline parameter that allows all "normal" workflows to not run.
Add a pipeline parameter for on-demand triggering of job 2
Add pipeline parameters for the actual parameters you need to pass to job 2.
Create a job 3 that takes pipeline parameters and passes them to job 2 as env vars.
Create a workflow that runs job 3 and then job 2 when the on-demand variable is set

Awkward? Oh yes. I can only guess that circle ci had some other use case in mind for their introduction of pipeline variables because this is just not very convenient.

Conclusion

I still can't really figure out how you are "supposed to" use pipeline variables. Maybe official docs will have more clarity on this in the future.

I really see the need for pipeline variables and they could be quite powerful but their limitations lead to some awkwardness, at least for our use cases. I find the following limitations the most annoying:

No way (I think) to set a pipeline variable in job 1, accessing it from job 2.
No way (I think) to set a pipeline variable in job definition.
Variables must be predefined.
No way to selectively just run one workflow
No way to selectively just run one job

Working example POC of config.yml for running job2 both using output from job1 or on-demand with parameters sent to custom pipeline, custom workflow and interim job3:

version: 2.1

# Pipeline parameters
parameters:
  workflow_ondemand:
    type: boolean
    default: false
  workflow_job2_ondemand:
    type: boolean
    default: false
  workflow_job2_param1_version:
    type: string
    default: "invalid version"

workflows:
  version: 2

  normal-workflow:
    unless: << pipeline.parameters.workflow_ondemand >>
    jobs:
      - job1
      - job2:
          requires: [job1]
  workflow-job2-ondemand:
    when: << pipeline.parameters.workflow_job2_ondemand >>
    jobs:
      - job3
      - job2:
          requires: [job3]

# Trigger with:
#
# curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{
#                             "branch": "feat",
#                             "parameters": {
#                                "workflow_ondemand": true,
#                                "workflow_job2_ondemand": true,
#                                "workflow_job2_param1_version": "version1"
#                              }
# }' https://circleci.com/api/v2/project/gh/<org>/<repo>/pipeline

jobs:
  job1:
    docker:
      - image: circleci/node:latest
    steps:
      - run:
          name: Fake build and generate random version number
          command: |
            echo export VERSION=$((1 + RANDOM % 100)) >> /tmp/.env
            source /tmp/.env
            echo "Version in job1: ${VERSION}"
      - persist_to_workspace:
          root: /tmp/
          paths: ['.env']

  job2:
    docker:
      - image: circleci/node:latest
    steps:
      - attach_workspace:
          at: /tmp
      - run:
          name: "Load and print version from previous step"
          command: |
            source /tmp/.env
            echo "Version in job2: ${VERSION}"

  job3:
    docker:
      - image: circleci/node:latest
    environment:
      VERSION: << pipeline.parameters.workflow_job2_param1_version >>
    steps:
      - run:
          name: "Save parameter value to .env"
          command: |
            echo export VERSION=${VERSION} >> /tmp/.env
            echo "Version in job3: ${VERSION}"
      - persist_to_workspace:
          root: /tmp/
          paths: ['.env']

来源：https://stackoverflow.com/questions/60390673/what-are-circleci-pipelines-can-they-be-used-to-trigger-job-with-parameters

标签

continuous-integration

pipeline

circleci