Merge pull request #22 from ansibl8s/ha_master

- HA (kubernetes and etcd)
- Dockerize kubenertes components (api-server/scheduler/replica-manager/proxy)

README.md

@@ -6,17 +6,52 @@ Based on [CiscoCloud](https://github.com/CiscoCloud/kubernetes-ansible) work.
 
 ### Requirements
 Tested on **Debian Jessie** and **Ubuntu** (14.10, 15.04, 15.10).
-The target servers must have access to the Internet in order to pull docker imaqes.
-The firewalls are not managed, you'll need to implement your own rules the way you used to.
+* The target servers must have access to the Internet in order to pull docker imaqes.
+* The firewalls are not managed, you'll need to implement your own rules the way you used to.
+* the following packages are required: openssl, curl, dnsmasq, python-httplib2 on remote servers and python-ipaddr on deployment machine.
 
 Ansible v1.9.x
 
 ### Components
-* [kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.1.2
+* [kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.1.3
 * [etcd](https://github.com/coreos/etcd/releases) v2.2.2
-* [calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.11.0
+* [calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.12.0
 * [flanneld](https://github.com/coreos/flannel/releases) v0.5.5
-* [docker](https://www.docker.com/) v1.8.3
+* [docker](https://www.docker.com/) v1.9.1
+
+Quickstart
+-------------------------
+The following steps will quickly setup a kubernetes cluster with default configuration.
+These defaults are good for tests purposes.
+
+Edit the inventory according to the number of servers
+```
+[downloader]
+10.115.99.1
+
+[kube-master]
+10.115.99.31
+
+[etcd]
+10.115.99.31
+10.115.99.32
+10.115.99.33
+
+[kube-node]
+10.115.99.32
+10.115.99.33
+
+[k8s-cluster:children]
+kube-node
+kube-master
+```
+
+Run the playbook
+```
+ansible-playbook -i environments/test/inventory cluster.yml -u root
+```
+
+You can jump directly to "*Available apps, installation procedure*"
 
 
 Ansible
@@ -24,7 +59,7 @@ Ansible
 ### Download binaries
 A role allows to download required binaries. They will be stored in a directory defined by the variable
 **'local_release_dir'** (by default /tmp).
-Please ensure that you have enough disk space there (about **1G**).
+Please ensure that you have enough disk space there (about **300M**).
 
 **Note**: Whenever you'll need to change the version of a software, you'll have to erase the content of this directory.
 
@@ -44,11 +79,15 @@ In node-mesh mode the nodes peers with all the nodes in order to exchange routes
 
 [kube-master]
 10.99.0.26
+10.99.0.59
 
 [etcd]
 10.99.0.26
+10.99.0.4
+10.99.0.59
 
 [kube-node]
+10.99.0.59
 10.99.0.4
 10.99.0.5
 10.99.0.36
@@ -60,18 +99,13 @@ In node-mesh mode the nodes peers with all the nodes in order to exchange routes
 10.99.0.5 local_as=xxxxxxxx
 
 [usa]
+10.99.0.59 local_as=xxxxxxxx
 10.99.0.36 local_as=xxxxxxxx
 10.99.0.37 local_as=xxxxxxxx
 
 [k8s-cluster:children]
 kube-node
 kube-master
-
-[paris:vars]
-peers=[{"router_id": "10.99.0.2", "as": "65xxx"}, {"router_id": "10.99.0.3", "as": "65xxx"}]
-
-[usa:vars]
-peers=[{"router_id": "10.99.0.34", "as": "65xxx"}, {"router_id": "10.99.0.35", "as": "65xxx"}]
 ```
 
 ### Playbook
@@ -86,16 +120,17 @@ peers=[{"router_id": "10.99.0.34", "as": "65xxx"}, {"router_id": "10.99.0.35", "
   roles:
     - { role: etcd, tags: etcd }
     - { role: docker, tags: docker }
-    - { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
     - { role: dnsmasq, tags: dnsmasq }
+    - { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
+
+- hosts: kube-node
+  roles:
+    - { role: kubernetes/node, tags: node }
 
 - hosts: kube-master
   roles:
     - { role: kubernetes/master, tags: master }
 
-- hosts: kube-node
-  roles:
-    - { role: kubernetes/node, tags: node }
 ```
 
 ### Run
@@ -107,6 +142,17 @@ ansible-playbook -i environments/dev/inventory cluster.yml -u root
 
 Kubernetes
 -------------------------
+### Multi master notes
+* You can choose where to install the master components. If you want your master node to act both as master (api,scheduler,controller) and node (e.g. accept workloads, create pods ...), 
+the server address has to be present on both groups 'kube-master' and 'kube-node'.
+
+* Almost all kubernetes components are running into pods except *kubelet*. These pods are managed by kubelet which ensure they're always running
+
+* One etcd cluster member per node will be configured. For safety reasons, you should have at least two master nodes.
+
+* Kube-proxy doesn't support multiple apiservers on startup ([#18174]('https://github.com/kubernetes/kubernetes/issues/18174')). An external loadbalancer needs to be configured.
+In order to do so, some variables have to be used '**loadbalancer_apiserver**' and '**apiserver_loadbalancer_domain_name**' 
+ 
 
 ### Network Overlay
 You can choose between 2 network plugins. Only one must be chosen.

apps.yml

@@ -2,7 +2,7 @@
 - hosts: kube-master
   roles:
     # System
-    - { role: apps/k8s-kubedns, tags: 'kubedns' }
+    - { role: apps/k8s-kubedns, tags: ['kubedns', 'kube-system'] }
 
     # Databases
     - { role: apps/k8s-postgres, tags: 'postgres' }
@@ -14,16 +14,16 @@
     - { role: apps/k8s-rabbitmq, tags: 'rabbitmq' }
 
     # Monitoring
-    - { role: apps/k8s-influxdb, tags: 'influxdb'}
-    - { role: apps/k8s-heapster, tags: 'heapster'}
-    - { role: apps/k8s-kubedash, tags: 'kubedash'}
+    - { role: apps/k8s-influxdb, tags: ['influxdb', 'kube-system']}
+    - { role: apps/k8s-heapster, tags: ['heapster', 'kube-system']}
+    - { role: apps/k8s-kubedash, tags: ['kubedash', 'kube-system']}
 
     # logging
     - { role: apps/k8s-kube-logstash, tags: 'kube-logstash'}
 
     # Console
     - { role: apps/k8s-fabric8, tags: 'fabric8' }
-    - { role: apps/k8s-kube-ui, tags: 'kube-ui' }
+    - { role: apps/k8s-kube-ui, tags: ['kube-ui', 'kube-system']}
 
     # ETCD
-    - { role: apps/k8s-etcd, tags: 'etcd'}
\ No newline at end of file
+    - { role: apps/k8s-etcd, tags: 'etcd'}

cluster.yml

@@ -8,13 +8,13 @@
   roles:
     - { role: etcd, tags: etcd }
     - { role: docker, tags: docker }
-    - { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
     - { role: dnsmasq, tags: dnsmasq }
-
-- hosts: kube-master
-  roles:
-    - { role: kubernetes/master, tags: master }
+    - { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
 
 - hosts: kube-node
   roles:
     - { role: kubernetes/node, tags: node }
+
+- hosts: kube-master
+  roles:
+    - { role: kubernetes/master, tags: master }

environments/production/group_vars/all.yml

- # Directory where the binaries will be installed
-bin_dir: /usr/local/bin
-
-# Where the binaries will be downloaded.
-# Note: ensure that you've enough disk space (about 1G)
-local_release_dir: "/tmp/releases"

environments/production/group_vars/k8s-cluster.yml → environments/test/group_vars/all.yml

+ # Directory where the binaries will be installed
+bin_dir: /usr/local/bin
+
+# Where the binaries will be downloaded.
+# Note: ensure that you've enough disk space (about 1G)
+local_release_dir: "/tmp/releases"
+
+# Cluster Loglevel configuration
+kube_log_level: 2
+
 # Users to create for basic auth in Kubernetes API via HTTP
-# kube_users:
-#   kube:
-#     pass: changeme
-#     role: admin
+kube_users:
+  kube:
+    pass: changeme
+    role: admin
 #   root:
 #     pass: changeme
 #     role: admin
 
 # Kubernetes cluster name, also will be used as DNS domain
-# cluster_name: cluster.local
+cluster_name: cluster.local
 
 # set this variable to calico if needed. keep it empty if flannel is used
-# kube_network_plugin: calico
+kube_network_plugin: calico
 
 # Kubernetes internal network for services, unused block of space.
-# kube_service_addresses: 10.233.0.0/18
+kube_service_addresses: 10.233.0.0/18
 
 # internal network. When used, it will assign IP
 # addresses from this range to individual pods.
 # This network must be unused in your network infrastructure!
-# kube_pods_subnet: 10.233.64.0/18
+kube_pods_subnet: 10.233.64.0/18
 
 # internal network total size (optional). This is the prefix of the
 # entire network. Must be unused in your environment.
@@ -28,16 +38,17 @@
 # internal network node size allocation (optional). This is the size allocated
 # to each node on your network.  With these defaults you should have
 # room for 4096 nodes with 254 pods per node.
-# kube_network_node_prefix: 24
+kube_network_node_prefix: 24
 
 # With calico it is possible to distributed routes with border routers of the datacenter.
-# peer_with_router: false
+peer_with_router: false
 # Warning : enabling router peering will disable calico's default behavior ('node mesh').
 # The subnets of each nodes will be distributed by the datacenter router
 
 # The port the API Server will be listening on.
-# kube_master_port: 443 # (https)
-# kube_master_insecure_port: 8080 # (http)
+kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
+kube_apiserver_port: 443 # (https)
+kube_apiserver_insecure_port: 8080 # (http)
 
 # Internal DNS configuration.
 # Kubernetes can create and mainatain its own DNS server to resolve service names
@@ -48,13 +59,28 @@
 # Kubernetes won't do this for you (yet).
 
 # Upstream dns servers used by dnsmasq
-# upstream_dns_servers:
-#   - 8.8.8.8
-#   - 4.4.8.8
+upstream_dns_servers:
+  - 8.8.8.8
+  - 4.4.8.8
 #
 # # Use dns server : https://github.com/ansibl8s/k8s-skydns/blob/master/skydns-README.md
-# dns_setup: true
-# dns_domain: "{{ cluster_name }}"
+dns_setup: true
+dns_domain: "{{ cluster_name }}"
 #
 # # Ip address of the kubernetes dns service
-# dns_server: 10.233.0.10
+dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(253)|ipaddr('address') }}"
+
+# For multi masters architecture:
+# kube-proxy doesn't support multiple apiservers for the time being so you'll need to configure your own loadbalancer
+# This domain name will be inserted into the /etc/hosts file of all servers
+# configuration example with haproxy :
+# listen kubernetes-apiserver-https
+#   bind 10.99.0.21:8383
+#    option ssl-hello-chk
+#    mode tcp
+#    timeout client 3h
+#    timeout server 3h
+#    server master1 10.99.0.26:443
+#    server master2 10.99.0.27:443
+#    balance roundrobin
+# apiserver_loadbalancer_domain_name: "lb-apiserver.kubernetes.local"

environments/test/group_vars/new-york.yml

+#---
+#peers:
+#  -router_id: "10.99.0.34"
+#   as: "65xxx"
+#  - router_id: "10.99.0.35"
+#   as: "65xxx"
+#
+#loadbalancer_apiserver:
+#  address: "10.99.0.44"
+#  port: "8383"

environments/test/group_vars/paris.yml

+#---
+#peers:
+#  -router_id: "10.99.0.2"
+#   as: "65xxx"
+#  - router_id: "10.99.0.3"
+#   as: "65xxx"
+#
+#loadbalancer_apiserver:
+#  address: "10.99.0.21"
+#  port: "8383"

environments/test/inventory.example

+[downloader]
+10.99.0.26
+
+[kube-master]
+10.99.0.26
+10.99.0.27
+
+[kube-node]
+10.99.0.27
+10.99.0.4
+10.99.0.5
+10.99.0.36
+10.99.0.37
+
+[paris]
+10.99.0.26
+10.99.0.4 local_as=xxxxxxxx
+10.99.0.5 local_as=xxxxxxxx
+
+[new-york]
+10.99.0.36 local_as=xxxxxxxx
+10.99.0.37 local_as=xxxxxxxx
+
+[k8s-cluster:children]
+kube-node
+kube-master

k8s-common @ c69c5f88

-Subproject commit eaab0692ed375420e183d18392ce79a4c6ed2069
+Subproject commit c69c5f881fe414f6856f811b9bb40cd19bcf83f4

k8s-etcd @ abd61ee9

-Subproject commit e3e574ea25ef4b1db79cc20b6dd31efa8a7d87cb
+Subproject commit abd61ee91ae729e7b79ecd56d6bb4eed0ddbe604

k8s-heapster @ 44a6519b

-Subproject commit dc088e25efcd040e127543b861448aa0d219eac9
+Subproject commit 44a6519bf8957bff316d3e3bc857d554f69c4016

roles/dnsmasq/tasks/main.yml

@@ -8,6 +8,14 @@
   when: hostvars[item].ansible_default_ipv4.address is defined
   with_items: groups['all']
 
+- name: populate kubernetes loadbalancer address into hosts file
+  lineinfile:
+    dest: /etc/hosts
+    regexp: ".*{{ apiserver_loadbalancer_domain_name }}$"
+    line: "{{ loadbalancer_apiserver.address }} lb-apiserver.kubernetes.local"
+    state: present
+  when: loadbalancer_apiserver is defined and apiserver_loadbalancer_domain_name is defined
+
 - name: clean hosts file
   lineinfile:
     dest: /etc/hosts
@@ -21,16 +29,17 @@
   apt:
     name: "{{ item }}"
     state: present
+    update_cache: yes
   with_items:
     - dnsmasq
     - bind9utils
-  when: inventory_hostname in groups['kube-master'][0]
+  when: inventory_hostname in groups['kube-master']
 
 - name: ensure dnsmasq.d directory exists
   file:
     path: /etc/dnsmasq.d
     state: directory
-  when: inventory_hostname in groups['kube-master'][0]
+  when: inventory_hostname in groups['kube-master']
 
 - name: configure dnsmasq
   template:
@@ -39,14 +48,14 @@
     mode: 755
   notify:
     - restart dnsmasq
-  when: inventory_hostname in groups['kube-master'][0]
+  when: inventory_hostname in groups['kube-master']
 
 - name: enable dnsmasq
   service:
     name: dnsmasq
     state: started
     enabled: yes
-  when: inventory_hostname in groups['kube-master'][0]
+  when: inventory_hostname in groups['kube-master']
 
 - name: update resolv.conf with new DNS setup
   template:
@@ -56,3 +65,5 @@
 
 - name: disable resolv.conf modification by dhclient
   copy: src=dhclient_nodnsupdate dest=/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate mode=u+x
+
+- meta: flush_handlers

roles/dnsmasq/templates/resolv.conf.j2

 ; generated by ansible
 search {{ [ 'default.svc.' + dns_domain, 'svc.' + dns_domain, dns_domain ] | join(' ') }}
+{% if inventory_hostname in groups['kube-master'] %}
+nameserver {{ ansible_default_ipv4.address }}
+{% else %}
 {% for host in groups['kube-master'] %}
 nameserver {{ hostvars[host]['ansible_default_ipv4']['address'] }}
 {% endfor %}
+{% endif %}

roles/docker/tasks/install.yml

@@ -13,7 +13,7 @@
   with_items:
     - aufs-tools
     - cgroupfs-mount
-    - docker-engine=1.8.3-0~{{ ansible_distribution_release }}
+    - docker-engine=1.9.1-0~{{ ansible_distribution_release }}
 
 - name: Copy default docker configuration
   template: src=default-docker.j2 dest=/etc/default/docker backup=yes

roles/download/defaults/main.yml

 ---
-etcd_download_url: https://github.com/coreos/etcd/releases/download
-flannel_download_url: https://github.com/coreos/flannel/releases/download
-kube_download_url: https://github.com/GoogleCloudPlatform/kubernetes/releases/download
-calico_download_url: https://github.com/Metaswitch/calico-docker/releases/download
-
 etcd_version: v2.2.2
 flannel_version: 0.5.5
 
-kube_version: v1.1.2
-kube_sha1: 69d110d371752c6492d2f8695aa7a47be5b6ed4e
+kube_version: v1.1.3
+kubectl_checksum: "01b9bea18061a27b1cf30e34fd8ab45cfc096c9a9d57d0ed21072abb40dd3d1d"
+kubelet_checksum: "62191c66f2d670dd52ddf1d88ef81048977abf1ffaa95ee6333299447eb6a482"
+
+calico_version: v0.12.0
+
+etcd_download_url: "https://github.com/coreos/etcd/releases/download"
+flannel_download_url: "https://github.com/coreos/flannel/releases/download"
+kube_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/amd64"
+calico_download_url: "https://github.com/Metaswitch/calico-docker/releases/download"
 
-calico_version: v0.11.0

roles/download/tasks/kubernetes.yml

 ---
-- name: Create kubernetes release directory
+- name: Create kubernetes binary directory
   local_action: file
-     path={{ local_release_dir }}/kubernetes
+     path="{{ local_release_dir }}/kubernetes/bin"
      state=directory
+     recurse=yes
 
-- name: Check if kubernetes release archive has been downloaded
-  local_action: stat
-     path={{ local_release_dir }}/kubernetes/kubernetes.tar.gz
-  register: k_tar
-
-# issues with get_url module and redirects, to be tested again in the near future
-- name: Download kubernetes
-  local_action: shell
-    curl -o {{ local_release_dir }}/kubernetes/kubernetes.tar.gz -Ls {{ kube_download_url }}/{{ kube_version }}/kubernetes.tar.gz
-  when: not k_tar.stat.exists or k_tar.stat.checksum != "{{ kube_sha1 }}"
-  register: dl_kube
-
-- name: Compare kubernetes archive checksum
-  local_action: stat
-     path={{ local_release_dir }}/kubernetes/kubernetes.tar.gz
-  register: k_tar
-  failed_when: k_tar.stat.checksum != "{{ kube_sha1 }}"
-  when: dl_kube|changed
-
-- name: Extract kubernetes archive
-  local_action: unarchive
-     src={{ local_release_dir }}/kubernetes/kubernetes.tar.gz
-     dest={{ local_release_dir }}/kubernetes copy=no
-  when: dl_kube|changed
-
-- name: Extract kubernetes binaries archive
-  local_action: unarchive
-     src={{ local_release_dir }}/kubernetes/kubernetes/server/kubernetes-server-linux-amd64.tar.gz
-     dest={{ local_release_dir }}/kubernetes copy=no
-  when: dl_kube|changed
-
-- name: Pick up only kubernetes binaries
-  local_action: synchronize
-     src={{ local_release_dir }}/kubernetes/kubernetes/server/bin
-     dest={{ local_release_dir }}/kubernetes
-  when: dl_kube|changed
-
-- name: Delete unused kubernetes files
-  local_action: file
-     path={{ local_release_dir }}/kubernetes/kubernetes state=absent
-  when: dl_kube|changed
+- name: Download kubelet and kubectl
+  local_action: get_url
+    url="{{ kube_download_url }}/{{ item.name }}"
+    dest="{{ local_release_dir }}/kubernetes/bin"
+    sha256sum="{{ item.checksum }}"
+  with_items:
+    - name: kubelet
+      checksum: "{{ kubelet_checksum }}"
+    - name: kubectl
+      checksum: "{{ kubectl_checksum }}"

roles/etcd/handlers/main.yml

 ---
-- name: restart daemons
-  command: /bin/true
-  notify:
-    - reload systemd
-    - restart etcd2
-
 - name: reload systemd
   command: systemctl daemon-reload
 
-- name: restart etcd2
-  service: name=etcd2 state=restarted
+- name: restart reloaded-etcd2
+  service:
+    name: etcd2
+    state: restarted
 
-- name: Save iptables rules
-  command: service iptables save
+- name: restart etcd2
+  command: /bin/true
+  notify:
+    - reload systemd
+    - restart reloaded-etcd2

roles/etcd/tasks/configure.yml

 ---
-- name: Disable ferm
-  service: name=ferm state=stopped enabled=no
+- name: Copy etcd2.service systemd file
+  template:
+    src: systemd-etcd2.service.j2
+    dest: /lib/systemd/system/etcd2.service
+    backup: yes
+  notify:
+    - restart etcd2
 
 - name: Create etcd2 environment vars dir
   file: path=/etc/systemd/system/etcd2.service.d state=directory
 
 - name: Write etcd2 config file
-  template: src=etcd2.j2 dest=/etc/systemd/system/etcd2.service.d/10-etcd2-cluster.conf backup=yes
+  template: src=etcd2.j2 dest=/etc/systemd/system/etcd2.service.d/10-etcd2.conf backup=yes
   notify:
-    - reload systemd
     - restart etcd2
 
 - name: Ensure etcd2 is running

roles/etcd/tasks/install.yml

@@ -11,15 +11,7 @@
   with_items:
     - etcdctl
     - etcd
-  notify:
-    - restart daemons
+  notify: restart etcd2
 
 - name: Create etcd2 binary symlink
   file: src=/usr/local/bin/etcd dest=/usr/local/bin/etcd2 state=link
-
-- name: Copy etcd2.service systemd file
-  template:
-    src: systemd-etcd2.service.j2
-    dest: /lib/systemd/system/etcd2.service
-    backup: yes
-  notify: restart daemons

roles/etcd/templates/etcd2.j2

 # etcd2.0
 [Service]
-{% if inventory_hostname in groups['kube-master'] %}
-Environment="ETCD_ADVERTISE_CLIENT_URLS=http://{{ ansible_default_ipv4.address }}:2379,http://{{ ansible_default_ipv4.address }}:4001"
-Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{ ansible_default_ipv4.address }}:2380"
-Environment="ETCD_INITIAL_CLUSTER=master=http://{{ ansible_default_ipv4.address }}:2380"
+{% if inventory_hostname in groups['etcd'] %}
+{% set etcd = {} %}
+{% for srv in groups['etcd'] %}
+{% if inventory_hostname == srv %}
+{% set _dummy = etcd.update({'name':"master"+loop.index|string}) %}
+{% endif %}
+{% endfor %}
+Environment="ETCD_ADVERTISE_CLIENT_URLS=http://{{ hostvars[inventory_hostname]['ip'] | default( ansible_default_ipv4.address) }}:2379"
+Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{ hostvars[inventory_hostname]['ip'] | default( ansible_default_ipv4.address)  }}:2380"
+Environment="ETCD_INITIAL_CLUSTER={% for srv in groups['etcd'] %}master{{ loop.index|string }}=http://{{ srv }}:2380{% if not loop.last %},{% endif %}{% endfor %}"
 Environment="ETCD_INITIAL_CLUSTER_STATE=new"
 Environment="ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd"
-Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001"
-Environment="ETCD_LISTEN_PEER_URLS=http://:2380,http://{{ ansible_default_ipv4.address }}:7001"
-Environment="ETCD_NAME=master"
-{% else %}
-Environment="ETCD_ADVERTISE_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001"
-Environment="ETCD_INITIAL_CLUSTER=master=http://{{ groups['kube-master'][0] }}:2380"
-Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001"
-Environment="ETCD_PROXY=on"
+Environment="ETCD_LISTEN_CLIENT_URLS=http://{{ hostvars[inventory_hostname]['ip'] | default( ansible_default_ipv4.address)  }}:2379,http://127.0.0.1:2379"
+Environment="ETCD_LISTEN_PEER_URLS=http://{{ hostvars[inventory_hostname]['ip'] | default( ansible_default_ipv4.address)  }}:2380"
+Environment="ETCD_NAME={{ etcd.name }}"
+{% else  %}
+Environment="ETCD_INITIAL_CLUSTER={% for srv in groups['etcd'] %}master{{ loop.index|string }}=http://{{ srv }}:2380{% if not loop.last %},{% endif %}{% endfor %}"
+Environment="ETCD_LISTEN_CLIENT_URLS=http://127.0.0.1:23799"
 {% endif %}