Jump to content

Portal:Cloud VPS/Admin/Devstack magnum/Stable xena

From Wikitech
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Overview

These are magnum install instructions for installing on devstack stable/xena

Notes

You will need a user with sudo access. Often this is the stack user, but any will do.

This was tested on Ubuntu 20.04.

To install a CoreOS beyond the listed version, you'll need a newer hyperkube as cgroups updated to v2 and it does weird things that k8s >= 1.19 handles, but the last official hyperkube was 1.18. Though rancher provides an unofficial one that seems to work: 

wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20211203.3.0/x86_64/fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2.xz
unxz fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2.xz
openstack image create Fedora-CoreOS --file=fedora-coreos-35.20211203.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public
openstack coe cluster template create my-template --image Fedora-CoreOS --external-network public --fixed-network private --fixed-subnet private-subnet --dns-nameserver 8.8.8.8 --network-driver flannel --docker-storage-driver overlay2 --docker-volume-size 30 --master-flavor m1.small --flavor m1.medium --coe kubernetes --labels kube_tag=v1.20.14-rancher1-linux-amd64,hyperkube_prefix=docker.io/rancher/

Additionally the following patch appears necessary. It is seen at opendev, but isn't merged yet. It would go in before building magnum with pip. 

diff --git a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
index 05053115..9d669e78 100644
--- a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
+++ b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh
@@ -195,8 +195,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /run:/run \\
     --volume /dev:/dev \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     --volume /var/lib/calico:/var/lib/calico \\
     --volume /var/lib/docker:/var/lib/docker \\
@@ -236,8 +235,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
     --volume /usr/lib/os-release:/etc/os-release:ro \\
     --volume /etc/ssl/certs:/etc/ssl/certs:ro \\
     --volume /run:/run \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     \${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\
diff --git a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
index b74e856b..ef869272 100644
--- a/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
+++ b/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh
@@ -93,8 +93,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /run:/run \\
     --volume /dev:/dev \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     --volume /var/lib/calico:/var/lib/calico \\
     --volume /var/lib/docker:/var/lib/docker \\
@@ -134,8 +133,7 @@ ExecStart=/bin/bash -c '/usr/bin/podman run --name kube-proxy \\
     --volume /usr/lib/os-release:/etc/os-release:ro \\
     --volume /etc/ssl/certs:/etc/ssl/certs:ro \\
     --volume /run:/run \\
-    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \\
-    --volume /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd \\
+    --volume /sys/fs/cgroup:/sys/fs/cgroup \\
     --volume /lib/modules:/lib/modules:ro \\
     --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\
     \${CONTAINER_INFRA_PREFIX:-\${HYPERKUBE_PREFIX}}hyperkube:\${KUBE_TAG} \\

git fetch https://review.opendev.org/openstack/magnum refs/changes/28/800428/2 
git cherry-pick fe75ca3459a5b44b835b4157149d062526953915


xena was having a strange issue where it refused to load more than one CoreOS version at a time.

dns doesn't seem to be working from pods until flannel is restarted: 

kubectl -n kube-system rollout restart daemonset kube-flannel-ds

Installation

git clone https://github.com/openstack/devstack.git --branch stable/xena
cd devstack

cat <<EOF > local.conf
[[local|localrc]]
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=\$ADMIN_PASSWORD
RABBIT_PASSWORD=\$ADMIN_PASSWORD
SERVICE_PASSWORD=\$ADMIN_PASSWORD

enable_plugin barbican https://opendev.org/openstack/barbican stable/xena
enable_plugin heat https://github.com/openstack/heat stable/xena

enable_service h-eng h-api h-api-cfn h-api-cw
EOF

./stack.sh
source openrc
export OS_USERNAME='admin'
export OS_PASSWORD='secret'

cd ~
wget https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/34.20210518.3.0/x86_64/fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2.xz
unxz fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2.xz 
openstack image create Fedora-CoreOS --file=fedora-coreos-34.20210518.3.0-openstack.x86_64.qcow2 --disk-format=qcow2 --container-format=bare --property os_distro='fedora-coreos' --public


ssh-keygen -f admin -P ""
openstack keypair create --public-key admin.pub admin


sudo mkdir /etc/magnum
cd ~
git clone https://git.openstack.org/openstack/magnum --branch stable/xena
cd magnum
sudo pip install -e .
sudo tox -egenconfig
sudo cp etc/magnum/api-paste.ini /etc/magnum/api-paste.ini


sudo sh -c 'cat <<EOF > /etc/magnum/magnum.conf
[DEFAULT]
pybasedir = /usr/lib/python3/dist-packages/magnum
bindir = /usr/bin
state_path = /var/lib/magnum
transport_url = rabbit://magnum:secret@localhost:5672
[api]
host = <host ip>
[barbican_client]
[certificates]
cert_manager_type = barbican
[cinder]
[cinder_client]
region_name = RegionOne
api_version = 3
[cluster]
[cluster_heat]
[cluster_template]
[conductor]
[cors]
[database]
connection = mysql+pymysql://magnum:secret@localhost/magnum
[docker]
[docker_registry]
[drivers]
[glance_client]
[heat_client]
[keystone_auth]
[keystone_authtoken]
memcached_servers = localhost:11211
auth_version = v3
www_authenticate_uri = http://localhost/identity
project_domain_id = default
user_domain_id = default
password = secret
auth_url = http://localhost/identity
auth_type = password
admin_user = magnum
admin_password = secret
admin_tenant_name = service
project_name = service
project_domain_name = default
username = magnum
user_domain_name = default
region_name = RegionOne
auth_type = password
[kubernetes]
[magnum_client]
[neutron_client]
[nova_client]
[octavia_client]
[oslo_concurrency]
lock_path = /var/lock/magnum
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver = log
[oslo_messaging_rabbit]
[oslo_policy]
[profiler]
[quotas]
[trust]
trustee_domain_name = magnum
trustee_domain_admin_name = magnum_domain_admin
trustee_domain_admin_password = secret
trustee_keystone_interface = public
[x509]
EOF'


cd ~
git clone https://git.openstack.org/openstack/python-magnumclient --branch stable/xena
cd python-magnumclient
sudo pip install -e .


sudo vim /etc/magnum/magnum.conf # update host = <current ip>
mysql -h 127.0.0.1 -u root -psecret mysql <<EOF
CREATE DATABASE magnum;
CREATE USER 'magnum'@'%' IDENTIFIED BY 'secret';
GRANT ALL PRIVILEGES ON magnum.* TO 'magnum'@'%';
EOF


openstack user create --domain default --password secret magnum
openstack role add --project service --user magnum admin
openstack service create --name magnum --description "OpenStack Container Infrastructure Management Service" container-infra
openstack domain create --description "Owns users and projects created by magnum" magnum
openstack user create --domain magnum --password secret magnum_domain_admin
openstack role add --domain magnum --user-domain magnum --user magnum_domain_admin admin
openstack subnet set --dns-nameserver 8.8.8.8 private-subnet
# note replace <host ip> below
openstack endpoint create --region RegionOne container-infra public http://<host ip>:9511/v1
openstack endpoint create --region RegionOne keystone internal http://<host ip>/identity


sudo rabbitmqctl add_user magnum secret
sudo rabbitmqctl set_permissions -p / magnum ".*" ".*" "."
sudo magnum-db-manage upgrade
sudo magnum-api # leave this running
sudo magnum-conductor # leave this running


openstack coe cluster template create my-template --image Fedora-CoreOS --external-network public --fixed-network private --fixed-subnet private-subnet --dns-nameserver 8.8.8.8 --network-driver flannel --docker-storage-driver overlay2 --docker-volume-size 10 --master-flavor m1.small --flavor m1.small --coe kubernetes
openstack coe cluster create my-cluster --cluster-template my-template --master-count 1 --node-count 1 --keypair admin
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Devstack_magnum/Stable_xena&oldid=2097242"