This was already approved in #4106 but there are CI issues
with that PR due to references to kubernetes incubator.

After upgrading to Kubespray 2.8.1 with Kubeadm enabled Rook
Ceph volume provision failed due to the flexvolume plugin dir not
being correct. Adding the var fixed the issue

because of etc-kubernetes-manifests not empty.

* Adding ability to maintain existing Encryption Secrets at Rest.

If secrets_encryption.yaml is present it will not be overriten with a new kube_encrypt_token.

This should allow for it to be set ahead of a playbook running or maintain it if cluster.yml is ran on the same cluster and the ansible host does not have access to the secrets.

* Setting existing kube_encrypt_token across all master nodes in case it was missing in one or more nodes.

Fix issue where `kubeadm join` could wait forever for joining.

Fix issue where `kubeadm join` were not reaching the user, making
impossible to find the cause of the failure.

New behaviour is to first attempt to join without bypassing the
verifications checks and to display them if needed.

If this fails it still attempts to join by ignoring the check in
order to make previous behavior.

A timeout of 60 seconds is allocated for a joining.

Related-bug: #3973

check kube_pods_subnet and kube_service_addresses to valid ip network range, not single ip address (#4188)

This PR ensures that the e2fsprogs and xfsprogs packages are
installed on all Kubernetes nodes and that the packages are
the latest versions. It also ensures that the nodes can
create XFS filesystems when necessary, since not all distros
install xfsprogs by default.

e2fsprogs - ext2/ext3/ext4 file system utilities
xfsprogs - Utilities for managing the XFS filesystem

Avoid waiting forever for this task that should be very quick.

Fixes: #4148

* use not deprecated ansible_play_hosts variable

* Using tests as filters is deprecated

* Fix deprecation warning about pkg list

* Set cluster DNS correctly in case of nodelocal dns cache

* Pass in cluster_ip based on dns mode

* Disable nodelocaldns by default

* Fix syntax error

* Fix syntax issue

* Add nodelocadns ip to vars of node installation

* Change location of nodelocaldns_ip

* Try to remove newlines from jinja template

* Add debug for config file

* Move parameter logic outside of template

* Adapt templates after feedback

* Remove debugging

* Suport calico 3.4.0

Signed-off-by: wangxf1987 <xiaofeix.wang@gmail.com>

* Remove symlink + cni conflist template when 3.3.0+, handle Canal, addition of install-cni: sidecar(3.3.0) or initontainer(3.4.0), KUBECONFIG_FILEPATH, calico_cert_dir, advertise cluster ips

* scheduler.alpha.kubernetes.io/critical-pod deprecated since 1.12

Addressing the discussion started in #4064, this PR moves kubeadm and
hyperkube binaries to /usr/local/bin before running them on the master
nodes.

It is to address the case where local_release_dir points to /tmp
(kubespray default) and /tmp is mounted with noexec mode, preventing
any binaries to be run in that partition.

In role "node", we still move kubeadm to bin_dir only on the worker
nodes.

I know this is a bit hack.
If you use cloud LB, you can use kubeadm's controlPlaneEndpoint to configure kube-proxy's server field.
But for nginx-proxy, it didn't start when kubeadm init.

* Use external LB IP for external api endpoint

Use loadbalancer_apiserver.address instead of apiserver_loadbalancer_domain_name for kudadm init --apiserver-advertise-address argument

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#options states apiserver-advertise-address needs to be a IPv4 or IPv6 address

* only use loadbalancer IP if it is defined

* Generate external admin.conf with kubeadm

* Fix apiserver sans

I found a potential use case where `writable` could be null and therfore
not treated like a boolean, so this adds an extra default statement to
avoid negating a non-boolean as boolean which would lead to undefined. refs #4020

--address deprecated

In v1beta1 of `ClusterConfiguration` the extraVolumes `writable` field was changed to `readOnly` and its boolean value must be negated.

Also, the json field for `useHyperKubeImage` was incorrectly capitalized.

Right now we're consistently getting warnings about kubelet not found in
path during `kubeadm init`. We fixed this for `kubeadm join` in #3342, and this brings the change to init
as well.

- Fixed an issue where storage class host directories were looped
through excessive target hosts
- Fixes examples in the LVP `README.md` to use nested dicts instead of a
list of dicts

* Makes local volume provisioner more dynamic

* Correct variable name in local storage provisioner defaults

* Updates external-provisioner readme

* Updates variable naming to be more clear, more documentation, fixes sample inventory

* Variable refactor, untangled some jinja2 loops

* Corrects variable name

* No variable substitution in dict keys, replaced with anchor

* Fixes default storage_classes dict, inline docs

* Fixes spelling in inline docs

* Addresses comments in review

* Updates all the defaults

* Fix failing CI task

* Fixes external provisioner daemonset

* allows to override the bind addresses for controller-manager and scheduler

Useful for Prometheus metrics monitoring

* Add bind addr override support in kubeadm/v1beta1

Adds support for override of bind addresses for controller-manager
and scheduler in kubeadm/v1beta1

* Move location of bind address vars

* Remove double declaration of schedulerExtraArgs

* Use master node taint same as kubeadm configuration v1alpha3 or before.

The change implemented in #3908 remove line breaks for supplementary
addresses in kubeadm SANs, causing errors in the config file and
failure to bring cluster up. This commit reimplement line breaks in
between supplementary addresses.

- Creates and defaults an ansible variable for every configuration option in the `kubeproxy.config.k8s.io/v1alpha1` type spec
  - Fixes vars that were orphaned by removing non-kubeadm
  - Fixes previously harcoded kubeadm values
- Introduces a `main` directory for role default files per component (requires ansible 2.6.0+)
  - Split out just `kube-proxy.yml` in this first effort
- Removes the kube-proxy server field patch task

We should continue to pull out other components from `main.yml` into their own defaults files as I did here for `defaults/main/kube-proxy.yml`. I hope for and will need others to join me in this refactoring across the project until each component config template has a matching role defaults file, with shared defaults in `kubespray-defaults` or `downloads`

* Add scale master features

* Add certificate management with kubeadm

* Add kubeadm kubeconfig

* Fix ymalroles error

* fix upgrade cluster fialed

* force update cert and keys when you reconfigure cluster

Webhook token auth:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication

Fixes #3063.

* controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

Enable load balancer for single-master setups
Fixes an issue where single-master setups are not reachable using the usual admin.conf from outside the cluster. 

controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

* add fix to other api versions

* remove obsolete check completely

* remove check, pass 2

* removes checks in client configuration

* delete 'and'