Skip to content
Snippets Groups Projects
Select Git revision
  • ca66a96d0ac7934b6af66ab2aad5c2ce975c6dca
  • master default protected
  • v2.27.1
  • v2.28.0
  • v2.27.0
  • v2.25.1
  • v2.24.3
  • v2.26.0
  • v2.24.2
  • v2.25.0
  • v2.24.1
  • v2.22.2
  • v2.23.3
  • v2.24.0
  • v2.23.2
  • v2.23.1
  • v2.23.0
  • v2.22.1
  • v2.22.0
  • v2.21.0
  • v2.20.0
  • v2.19.1
22 results

nodes.md

Blame
  • user avatar
    Alexander Petermann authored and GitHub committed
    75571ed3
    History

    Adding/replacing a node

    Modified from comments in #3471

    Limitation: Removal of first kube-master and etcd-master

    Currently you can't remove the first node in your kube-master and etcd-master list. If you still want to remove this node you have to:

    1) Change order of current masters

    Modify the order of your master list by pushing your first entry to any other position. E.g. if you want to remove node-1 of the following example:

      children:
        kube-master:
          hosts:
            node-1:
            node-2:
            node-3:
        kube-node:
          hosts:
            node-1:
            node-2:
            node-3:
        etcd:
          hosts:
            node-1:
            node-2:
            node-3:

    change your inventory to:

      children:
        kube-master:
          hosts:
            node-2:
            node-3:
            node-1:
        kube-node:
          hosts:
            node-2:
            node-3:
            node-1:
        etcd:
          hosts:
            node-2:
            node-3:
            node-1:

    2) Upgrade the cluster

    run cluster-upgrade.yml or cluster.yml. Now you are good to go on with the removal.

    Adding/replacing a worker node

    This should be the easiest.

    1) Add new node to the inventory

    2) Run scale.yml

    You can use --limit=NODE_NAME to limit Kubespray to avoid disturbing other nodes in the cluster.

    Before using --limit run playbook facts.yml without the limit to refresh facts cache for all nodes.

    3) Remove an old node with remove-node.yml

    With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed.

    If the node you want to remove is not online, you should add reset_nodes=false to your extra-vars: -e node=NODE_NAME reset_nodes=false. Use this flag even when you remove other types of nodes like a master or etcd nodes.

    5) Remove the node from the inventory

    That's it.

    Adding/replacing a master node

    1) Run cluster.yml

    Append the new host to the inventory and run cluster.yml. You can NOT use scale.yml for that.

    3) Restart kube-system/nginx-proxy

    In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.

    # run in every host
    docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart

    4) Remove old master nodes

    With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed. If the node you want to remove is not online, you should add reset_nodes=false to your extra-vars.

    Adding an etcd node

    You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replace or scale up operation. Either add two new nodes or remove an old one.

    1) Add the new node running cluster.yml

    Update the inventory and run cluster.yml passing --limit=etcd,kube-master -e ignore_assert_errors=yes. If the node you want to add as an etcd node is already a worker or master node in your cluster, you have to remove him first using remove-node.yml.

    Run upgrade-cluster.yml also passing --limit=etcd,kube-master -e ignore_assert_errors=yes. This is necessary to update all etcd configuration in the cluster.

    At this point, you will have an even number of nodes. Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node. Even so, running applications should continue to be available.

    If you add multiple ectd nodes with one run, you might want to append -e etcd_retries=10 to increase the amount of retries between each ectd node join. Otherwise the etcd cluster might still be processing the first join and fail on subsequent nodes. etcd_retries=10 might work to join 3 new nodes.

    Removing an etcd node

    1) Remove an old etcd node

    With the node still in the inventory, run remove-node.yml passing -e node=NODE_NAME as the name of the node that should be removed. If the node you want to remove is not online, you should add reset_nodes=false to your extra-vars.

    2) Make sure only remaining nodes are in your inventory

    Remove NODE_NAME from your inventory file.

    3) Update kubernetes and network configuration files with the valid list of etcd members

    Run cluster.yml to regenerate the configuration files on all remaining nodes.

    4) Shutdown the old instance

    That's it.