Network plugin custom (#9819)

* network_plugin/custom_cni: add CNI to apply provided manifests

Add a new simple custom_cni to install provided Kubernetes manifests.
This could be useful to use manifests directly provided by a CNI when
there are not support by Kubespray (i.e.: helm chart or any other manifests
generation method).

Co-authored-by: James Landrein <james.landrein@proton.ch>
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>

* network_plugin/custom_cni: add test with cilium

Co-authored-by: James Landrein <james.landrein@proton.ch>
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>

---------

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>
Co-authored-by: James Landrein <james.landrein@proton.ch>

.ansible-lint

@@ -35,3 +35,6 @@ skip_list:
   # Roles in kubespray don't need fully qualified collection names
   # (Disabled in Feb 2023)
   - 'fqcn-builtins'
+exclude_paths:
+  # Generated files
+  - tests/files/custom_cni/cilium.yaml

.gitlab-ci/packet.yml

@@ -268,6 +268,11 @@ packet_fedora36-kube-ovn:
   extends: .packet_periodic
   when: on_success
 
+packet_debian11-custom-cni:
+  stage: deploy-part2
+  extends: .packet_pr
+  when: manual
+
 # ### PR JOBS PART3
 # Long jobs (45min+)
 

.yamllint

@@ -3,6 +3,8 @@ extends: default
 
 ignore: |
   .git/
+  # Generated file
+  tests/files/custom_cni/cilium.yaml
 
 rules:
   braces:

docs/ci.md

@@ -4,60 +4,60 @@ To generate this Matrix run `./tests/scripts/md-table/main.py`
 
 ## containerd
 
-| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
-|---| --- | --- | --- | --- | --- | --- | --- | --- |
-almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
-amazon |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-centos7 |  :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
-debian10 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
-debian11 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
-fedora35 |  :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: |
-fedora36 |  :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
-opensuse |  :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
-rockylinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-rockylinux9 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
-ubuntu16 |  :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: |
-ubuntu18 |  :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: |
-ubuntu20 |  :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: |
-ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+| OS / CNI | calico | canal | cilium | custom_cni | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
+amazon |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
+debian10 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian11 |  :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
+debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
+fedora35 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: |
+fedora36 |  :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
+opensuse |  :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux9 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 |  :x: | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: |
+ubuntu18 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: |
+ubuntu20 |  :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: |
+ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 
 ## crio
 
-| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
-|---| --- | --- | --- | --- | --- | --- | --- | --- |
-almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-centos7 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian10 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian11 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-fedora35 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-fedora36 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-opensuse |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu16 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu18 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu20 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu22 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+| OS / CNI | calico | canal | cilium | custom_cni | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian10 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian11 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora35 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora36 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+opensuse |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu18 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu20 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu22 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 
 ## docker
 
-| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
-|---| --- | --- | --- | --- | --- | --- | --- | --- |
-almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-centos7 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian10 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian11 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-fedora35 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-fedora36 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
-opensuse |  :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
-rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu16 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
-ubuntu18 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu20 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+| OS / CNI | calico | canal | cilium | custom_cni | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian10 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian11 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora35 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora36 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
+opensuse |  :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
+ubuntu18 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu20 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |

roles/kubernetes/preinstall/tasks/0020-verify-settings.yml

@@ -30,7 +30,7 @@
 
 - name: Stop if unknown network plugin
   assert:
-    that: kube_network_plugin in ['calico', 'canal', 'flannel', 'weave', 'cloud', 'cilium', 'cni','kube-ovn', 'kube-router', 'macvlan']
+    that: kube_network_plugin in ['calico', 'canal', 'flannel', 'weave', 'cloud', 'cilium', 'cni', 'kube-ovn', 'kube-router', 'macvlan', 'custom_cni']
     msg: "{{ kube_network_plugin }} is not supported"
   when:
     - kube_network_plugin is defined

roles/network_plugin/custom_cni/defaults/main.yml

+---
+
+custom_cni_manifests: []

roles/network_plugin/custom_cni/tasks/main.yml

+---
+
+- name: Cilium | Check Cilium encryption `cilium_ipsec_key` for ipsec
+  assert:
+    that:
+      - "custom_cni_manifests | length > 0"
+    msg: "custom_cni_manifests should not be empty"
+
+- name: Custom CNI | Copy Custom manifests
+  template:
+    src: "{{ item }}"
+    dest: "{{ kube_config_dir }}/{{ item | basename | replace('.j2', '') }}"
+    mode: 0644
+  loop: "{{ custom_cni_manifests }}"
+  delegate_to: "{{ groups['kube_control_plane'] | first }}"
+  run_once: true
+
+- name: Custom CNI | Start Resources
+  kube:
+    namespace: "kube-system"
+    kubectl: "{{ bin_dir }}/kubectl"
+    filename: "{{ kube_config_dir }}/{{ item | basename | replace('.j2', '') }}"
+    state: "latest"
+    wait: true
+  loop: "{{ custom_cni_manifests }}"
+  delegate_to: "{{ groups['kube_control_plane'] | first }}"
+  run_once: true

roles/network_plugin/meta/main.yml

@@ -42,6 +42,11 @@ dependencies:
     tags:
       - kube-router
 
+  - role: network_plugin/custom_cni
+    when: kube_network_plugin == 'custom_cni'
+    tags:
+      - custom_cni
+
   - role: network_plugin/multus
     when: kube_network_plugin_multus
     tags:

tests/files/custom_cni/README.md

+# Custom CNI manifest generation
+
+As an example we are using Cilium for testing the network_plugins/custom_cni.
+
+To update the generated manifests to the latest version do the following:
+
+```sh
+helm repo add cilium https://helm.cilium.io/
+helm repo update
+helm template cilium/cilium -n kube-system -f values.yaml > cilium.yaml
+```

tests/files/custom_cni/values.yaml

+---
+
+# We disable hubble so that helm doesn't try to generate any certificate.
+# This is not needed to test network_plugin/custom_cni anyway.
+hubble:
+  enabled: false
+
+ipam:
+  operator:
+    # Set the appropriate pods subnet
+    clusterPoolIPv4PodCIDR: "{{ kube_pods_subnet }}"

tests/files/packet_debian11-custom-cni.yml

+---
+# Instance settings
+cloud_image: debian-11
+mode: default
+
+# Kubespray settings
+kube_network_plugin: custom_cni
+custom_cni_manifests:
+  - tests/files/custom_cni/cilium.yaml