* Fix etcd certificates reference to support etcd_kubeadm_enabled:true

* Add retries to ETCD Join Member task

* Fix etcd certificates reference when etcd_kubeadm_enabled:true

* Fix conflicts

* Add kubectl variable

* Replace kubectl usage by kubectl variable in roles

* Remove redundant --kubeconfig on kubectl usage

* Replace unecessary shell usage with command

- Use builtin task scheduling of ansible (same task on each host)
  instead of manual looping on master

Benefits:
- One less play in remove-node.yml playbook
- Parralel node drain
- Drain parameters (timeout, grace period, retries,
  allow_ungraceful_removal) can be adjusted separately for each node
  with ansible variables

Revert "remove no longer present etcd nodes from  APIEndpoints list in kubeadm-config configmap (#8244)" (#8287)

This reverts commit dc767c14.

* Limit kubectl delete node to k8s nodes

This avoids the use of `kubectl delete node` when removing etcd nodes
which are not part of the cluser (separate etcd)

* Take errors into account when deleting node

There should not be error now that we're limiting the deletion to nodes
actually in the cluster

* Retrying on error

The path of kubeconfig should be configurable, and its default value
is /etc/kubernetes/admin.conf. Most paths of the file are configurable
but some were not. This make those configurable.

* Ansible: move to Ansible 3.4.0 which uses ansible-base 2.10.10

* Docs: add a note about ansible upgrade post 2.9.x

* CI: ensure ansible is removed before ansible 3.x is installed to avoid pip failures

* Ansible: use newer ansible-lint

* Fix ansible-lint 5.0.11 found issues

* syntax issues
* risky-file-permissions
* var-naming
* role-name
* molecule tests

* Mitogen: use 0.3.0rc1 which adds support for ansible 2.10+

* Pin ansible-base to 2.10.11 to get package fix on RHEL8

* Improve scale flow

* Add confirmation prompt again

* rename ansible groups to use _ instead of -

k8s-cluster -> k8s_cluster
k8s-node -> k8s_node
calico-rr -> calico_rr
no-floating -> no_floating

Note: kube-node,k8s-cluster groups in upgrade CI
      need clean-up after v2.16 is tagged

* ensure old groups are mapped to the new ones

* kubelet absolute path

* kubelet absolute path

Signed-off-by: Etienne Champetier <e.champetier@ateme.com>

Signed-off-by: Etienne Champetier <e.champetier@ateme.com>

This replaces kube-master with kube_control_plane because of [1]:

  The Kubernetes project is moving away from wording that is
  considered offensive. A new working group WG Naming was created
  to track this work, and the word "master" was declared as offensive.
  A proposal was formalized for replacing the word "master" with
  "control plane". This means it should be removed from source code,
  documentation, and user-facing configuration from Kubernetes and
  its sub-projects.

NOTE: The reason why this changes it to kube_control_plane not
      kube-control-plane is for valid group names on ansible.

[1]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md#motivation

* Remove ignore_errors from drain tasks and enable retires

* Fix lint error by checking if stdout length is not 0, ie string is not empty.

and add configuration to allow ungraceful removal

* respect kube_override_hostname during removal/upgrade

* Use hostvars in loop

* add remove_node_ip

* move remove_node_ip to remove etcd part

* fix: remove tail space

* fix: handle ubuntu: focal

Change-Id: Ib27d051ff111f813097a9b33a86465a2a30a6db0

Change-Id: I1c59249f08f16d0f6fd60df6ab61f17a0a7df189

Kubespray waits exit of every drain before run other one.
Running drain every after each other seems better than parallel, because we should check resources availability every time.
But, this way, we have one additional problem: possible restart pods on the nodes that are killed little bit later.
Fast cordon before heavy drain seems like an easy solution.

* Remove non-kubeadm deployment

* More cleanup

* More cleanup

* More cleanup

* More cleanup

* Fix gitlab

* Try stop gce first before absent to make the delete process work

* More cleanup

* Fix bug with checking if kubeadm has already run

* Fix bug with checking if kubeadm has already run

* More fixes

* Fix test

* fix

* Fix gitlab checkout untill kubespray 2.8 is on quay

* Fixed

* Add upgrade path from non-kubeadm to kubeadm. Revert ssl path

* Readd secret checking

* Do gitlab checks from v2.7.0 test upgrade path to 2.8.0

* fix typo

* Fix CI jobs to kubeadm again. Fix broken hyperkube path

* Fix gitlab

* Fix rotate tokens

* More fixes

* More fixes

* Fix tokens

The post-remove action fails during the kubectl delete node action because with rc: 2, command not found. The kubectl is not in the system PATH and the full path to the binary is required

Drain node except daemonsets resource
Use reset cluser for delete deploy data
Then delete node