kubesprayでkubernetesクラスタを構築
kubesprayとは
ansibleベースでkubernetesを構築するツール。ProductionReady環境を構築できるということで、使ってみる。
以前kubeadmつかったが、細かいところまでいじれないので、ansibleのフルスクラッチ構築が可能なkubesprayに移行したいというのがモチベ。
環境準備
| ホスト | OS | 目的 |
|---|---|---|
| kubespray Client | CentOS7 | kubespray(ansible) 実行用クライアント |
| k8s-master1 | CentOS7 | k8sのマスターノード |
| k8s-master2 | CentOS7 | k8sのマスターノード |
| k8s-master3 | CentOS7 | k8sのマスターノード |
kubespray Client
Kubesprayを実行するホスト。ansible等が動作すれば良いため、クライアントPCで実行しても良いが、環境差異があると動かなかったりするので、手順をまとめておく。 kubespray用実行用のDockerコンテナとかあれば便利そう。
必要パッケージのインストール
sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm yum install -y python36u python36u-pip python36u-devel ansible git gcc python-netaddr libffi-devel openssl-devel pip3.6 install pip --upgrade pip install jinja2
kubespray
git clone https://github.com/kubernetes-incubator/kubespray.git cd kubespray/ pip install -r requirements.txt cp -r inventory/ my_inventory
kubesprayで構築するノード情報を設定(IPとかは自分の環境に合わせて) vi my_inventory/sample/inventory.ini
[all] k8s-master1 ansible_host=192.168.0.11 k8s-master2 ansible_host=192.168.0.12 k8s-master3 ansible_host=192.168.0.13 [kube-master] k8s-master1 k8s-master2 k8s-master3 [etcd] k8s-master1 k8s-master2 k8s-master3 [kube-node] k8s-master1 k8s-master2 k8s-master3 [k8s-cluster:children] kube-master kube-node [all:vars] ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_pass=<Password> ansible_sudo_pass=<Password>
各k8s-masterノードの下処理
- firewalld の無効
- swapの無効
- ansibleが無効にしてくれるっぽい
- SElinuxの無効
- ansibleが無効にしてくれるっぽい(permissive)
構築実行
[root@localhost kubespray]# ansible-playbook -i my_inventory/inventory.ini cluster.yml -v ... ... PLAY RECAP ****************************************************************************************************************** k8s-master1 : ok=425 changed=69 unreachable=0 failed=0 k8s-master2 : ok=350 changed=54 unreachable=0 failed=0 k8s-master3 : ok=352 changed=55 unreachable=0 failed=0 localhost : ok=1 changed=0 unreachable=0 failed=0 Sunday 09 June 2019 06:26:08 -0400 (0:00:00.072) 0:05:58.662 *********** =============================================================================== kubernetes/master : kubeadm | Init other uninitialized masters ------------------------------------------------------ 23.64s kubernetes/master : kubeadm | Initialize first master --------------------------------------------------------------- 21.12s download : container_download | download images for kubeadm config images ------------------------------------------- 10.61s etcd : reload etcd -------------------------------------------------------------------------------------------------- 10.57s etcd : wait for etcd up ---------------------------------------------------------------------------------------------- 9.25s kubernetes-apps/ansible : Kubernetes Apps | Start Resources ---------------------------------------------------------- 6.26s kubernetes/master : kubeadm | write out kubeadm certs ---------------------------------------------------------------- 5.53s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 3.75s kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS Template ------------------------------------------------ 3.52s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 3.52s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 3.50s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 3.43s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 3.37s network_plugin/calico : Calico | Create calico manifests ------------------------------------------------------------- 2.84s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 2.81s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 2.74s download : container_download | Download containers if pull is required or told to always pull (all nodes) ----------- 2.67s policy_controller/calico : Start of Calico kube controllers ---------------------------------------------------------- 2.32s kubernetes/node : Persist ip_vs modules ------------------------------------------------------------------------------ 2.19s policy_controller/calico : Create calico-kube-controllers manifests -------------------------------------------------- 2.00s
躓きポイント
以下のようなエラーが出たが、原因はipv6も含めた登録するDNSが多すぎたことが原因。
Too many nameservers. You can relax this check by set docker_dns_servers_strict=false in all.yml and we will only use the first 3
roles/container-engine/docker/tasks/set_facts_dns.yml の 56 行目 のlengthを 3⇒5にして対処した。
56 when: docker_dns_servers|length > 5 and docker_dns_servers_strict|bool
