Argo Workflows 体验

作者: ygqygq2 分类: 自动化,虚拟化/容器 发布时间: 2022-04-27 18:59

1. Argo Workflows 简单介绍

Argo Workflows 是一个开源容器化原生工作流引擎,用于在 Kubernetes 中编排并行作业。Argo Workflows 实现为一个 Kubernetes CRD (自定义资源定义)。
其详细介绍和核心概念等查看官方文档即可,本文通过示例来体验 Argo Workflows。

2. Argo Workflows 安装

我使用 helm 安装 bitnami 的 charts 仓库中的 argo-workflows。安装过程略…

3. 官方示例体验

我们使用官方示例测试

3.1 CLI 安装

根据自己的环境,安装相应的 CLI。
https://github.com/argoproj/argo-workflows/releases

3.2 示例体验并简单总结

3.2.1 hello world

argo submit hello-world.yaml    # submit a workflow spec to Kubernetes
argo list                       # list current workflows
argo get hello-world-xxx        # get info about a specific workflow
argo logs hello-world-xxx       # print the logs from a workflow
argo delete hello-world-xxx     # delete workflow
argo submit hello-world.yaml -n argo
argo watch -n argo hello-world-59rtg
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
  annotations:
    workflows.argoproj.io/description: |
      This is a simple hello world example.
      You can also run it in Python: https://couler-proj.github.io/couler/examples/#hello-world
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
hello world
kubectl get pod -n argo
pod 完成
  • argo 命令用起来参考了 kubectl 的习惯,还是非常丝滑的;
  • argo 使用 submit 子命令创建 Workflow ;
  • Workflow 资源显示非常详细,包含运行状态、所用资源、运行时间等;

3.2.2 Parameters

argo submit arguments-parameters.yaml -p message="ygqygq2 is testing argo workflows" -n argo
argo watch -n argo arguments-parameters-fnbpl
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: arguments-parameters-
spec:
  entrypoint: whalesay
  # Parameters can be passed/overridden via the argo CLI.
  # To override the printed message, run `argo submit` with the -p option:
  # $ argo submit examples/arguments-parameters.yaml -p message="goodbye world"
  arguments:
    parameters:
    - name: message
      value: hello world

  templates:
  - name: whalesay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]
参数化构建
运行结果
argo logs -n argo  arguments-parameters-fnbpl

全局参数

argo submit -n argo global-parameters.yaml -p message="ygqygq2 is testing workflows"
argo logs -n argo global-parameters-jhwqj
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: global-parameters-
spec:
  entrypoint: whalesay1
  # Parameters can be passed/overridden via the argo CLI.
  # To override the printed message, run `argo submit` with the -p option:
  # $ argo submit examples/arguments-parameters.yaml -p message="goodbye world"
  arguments:
    parameters:
    - name: message
      value: hello world

  templates:
  - name: whalesay1
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["{{workflow.parameters.message}}"]
全局参数
  • argo 通过 -p key=value 方式传递参数给 Workflow 的容器,Workflow 中使用 args: ["{{inputs.parameters.message}}"] 接收参数;
  • --parameter-file params.yaml 参数构建可以指定 YAML 或 JSON 格式参数文件;
  • {{workflow.parameters.message}} 这种方式 workflow 全局参数 message

3.2.3 Steps

多步骤 workflow

argo submit steps.yaml -n argo
argo watch -n argo steps-slbmb
# This template demonstrates a steps template and how to control sequential vs. parallel steps.
# In this example, the hello1 completes before the hello2a, and hello2b steps, which run in parallel.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: hello-hello-hello
  templates:
  - name: hello-hello-hello
    steps:
    - - name: hello1
        template: whalesay
        arguments:
          parameters: [{name: message, value: "hello1"}]
    - - name: hello2a
        template: whalesay
        arguments:
          parameters: [{name: message, value: "hello2a"}]
      - name: hello2b
        template: whalesay
        arguments:
          parameters: [{name: message, value: "hello2b"}]

  - name: whalesay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]
多步骤工作流
最终运行结果
可以看到有 3 个运行完的 POD
串行、并行
argo logs -n argo steps-slbmb
输出日志
  • 可以看到 hello1、hello2a 是串行关系;
  • hello2a、hello2b 是并行关系;
  • argo log 可以看到不同 pod 的输出不同颜色,这点体验不错;

3.2.4 DAG(directed-acyclic graph)

在下面工作流中,步骤 A B 同时运行,因为它们不依赖其它步骤,步骤 C 依赖 A,步骤 D 依赖 AB,它们的依赖步骤运行完成,才会开始。

多根工作流
argo submit -n argo dag-multiroot.yaml
argo watch -n argo dag-multiroot-z4zzz
# The following workflow executes a multi-root workflow
# 
#   A   B
#  / \ /
# C   D
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-multiroot-
spec:
  entrypoint: multiroot
  templates:
  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:3.7
      command: [echo, "{{inputs.parameters.message}}"]
  - name: multiroot
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        template: echo
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        depends: "A"
        template: echo
        arguments:
          parameters: [{name: message, value: C}]
      - name: D
        depends: "A && B"
        template: echo
        arguments:
          parameters: [{name: message, value: D}]
A 和 B 同时开始
依赖步骤运行完成
全部步骤运行完成
argo logs -n argo dag-multiroot-z4zzz
工作流日志

非 FailFast

argo submit -n argo dag-disable-failFast.yaml
argo watch -n argo dag-primay-branch-jdg6l
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-primay-branch-
spec:
  entrypoint: statis
  templates:
  - name: a
    container:
      image:  docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
  - name: b
    retryStrategy:
      limit: "2"
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["sleep 30; echo haha"]
  - name: c
    retryStrategy:
      limit: "3"
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo intentional failure; exit 2"]
  - name: d
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
  - name: statis
    dag:
      failFast: false
      tasks:
      - name: A
        template: a
      - name: B
        depends: "A"
        template: b
      - name: C
        depends: "A"
        template: c
      - name: D
        depends: "B"
        template: d
      - name: E
        depends: "D"
        template: d

FailFast false 示例:dag-disable-failFast.yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-primay-branch-
spec:
  entrypoint: statis
  templates:
  - name: a
    container:
      image:  docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
  - name: b
    retryStrategy:
      limit: "2"
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["sleep 30; echo haha"]
  - name: c
    retryStrategy:
      limit: "3"
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo intentional failure; exit 2"]
  - name: d
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
  - name: statis
    dag:
      failFast: false
      tasks:
      - name: A
        template: a
      - name: B
        depends: "A"
        template: b
      - name: C
        depends: "A"
        template: c
      - name: D
        depends: "B"
        template: d
      - name: E
        depends: "D"
        template: d
运行 dag-disable-failFast.yaml
非 FailFast
A运行完
C运行出错
C后面的步骤继续运行
所有步骤运行完成
argo logs -n argo dag-primay-branch-jdg6l
查看日志输出
  • DAG 默认 FailFast 设置为 true,即一旦有步骤失败,它将停止调度后面的步骤,只等待正在运行的步骤完成;
  • 如果将 FailFast 设置为 false,它将不管步骤运行结果,继续调度后面步骤,直至所有步骤运行完成;

3.2.5 Artifacts

配置制品库参考:https://argoproj.github.io/argo-workflows/configure-artifact-repository/
支持 Minio、AWS s3、GCS、阿里 OSS

argo submit -n argo artifact-repository-ref.yaml

当前官方示例报错,它建议使用 emptyDir

官方示例报错

我们根据官方 emptyDir 示例 继续

argo submit -n argo /tmp/artifactory-emptydir.yaml
argo watch -n argo empty-dir-sqrr9
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: empty-dir-
spec:
  entrypoint: main
  templates:
    - name: main
      container:
        image: argoproj/argosay:v2
        command: [sh, -c]
        args: ["cowsay hello world | tee /mnt/out/hello_world.txt"]
        volumeMounts:
          - name: out
            mountPath: /mnt/out
      volumes:
        - name: out
          emptyDir: { }
      outputs:
        parameters:
          - name: message
            valueFrom:
              path: /mnt/out/hello_world.txt
使用emptydir输出制品
正常输出
  • 当前 k8s 不允许 workflow 直接输出制品在目录或文件中,须使用 emptyDir 或 pvc 等;
  • 制品默认被打包为 Tarballs,默认情况下是 gzipped。可以通过使用归档字段指定归档策略来自定义此行为;

3.2.6 The Structure of Workflow Specs

Workflow 基本结构:

  • 包含元数据的 kubernetes 头部
  • Spec 主体
    • 带有可选参数的入口点调用
    • 模板定义列表
  • 对于每个模板定义
    • 模板名称
    • 可选的输入列表
    • 可选的输出列表
  • 容器调用或步骤列表
    • 对于每个步骤,都有一个模板调用

发表评论

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据