Design and planning
is currently a category 2 design role, ceph and nova. As long as ceph-node cluster, you are nova, computing services need to take control node and network node currently served by the ceph {01..03} The ground wire.
| vlan | name | Segment (CIDR mark) | use | device | Remark |
|---|---|---|---|---|---|
| 1031-1060 | os-taking | Custom | Private network project | Computing and network nodes located switcher | There are 31 individual and private networks, should be enough, or in the future be extended to 900-1030 bar. |
| 1031 | the-wuhan31 | 100.100.31.0/24 | Business District (wuhan31) host network | Computing and network nodes located switcher | This cluster does not need. In order to avoid that says is wrong. |
| 33 | the-extnet | 192.168.33.0/24 | Floating IP network. Private network NAT. | Switcher for all nodes, three switches. | Allow private network access to the outside world, or from the outside into the (IP Bind float) |
| 34-37 | the-pubnet | 192.168.34.0/24 - 192.168.37.0/24 | Straight-through network | Switcher for all nodes, three switches | As the general public export network. |
IP and hostname planning
gateway 100.100.31.1
127.0.0.1 localhost
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
100.100.31.254 cloud-wuhan31.***.org
100.100.31.201 wuhan31-ceph01.v3.os wuhan31-ceph01
100.100.31.202 wuhan31-ceph02.v3.os wuhan31-ceph02
100.100.31.203 wuhan31-ceph03.v3.os wuhan31-ceph03
100.100.31.102 wuhan31-nova01.v3.os wuhan31-nova01
100.100.31.103 wuhan31-nova02.v3.os wuhan31-nova02Virtual Machine Specification
cpu 1 2 4 8
Memory 124 816
Disk 2050
And defining memory / cpu values between 1 and 4 need to script as follows: 22 eventually generates the flavors.
#!/bin/bash
desc="create flavors for openstack."
log_file="/dev/shm/create-flavor.log"
# config cpu, ram, and disk. seperated value with space.
cpu_count_list="1 2 4 8"
ram_gb_list="1 2 4 8 16"
disk_gb_list="20 50"
# accept ram/cpu ratio.
ram_cpu_factor_min=1
ram_cpu_factor_max=4
tip(){ echo >&2 "$*"; }
die(){ tip "$*"; exit 1; }
#openstack flavor create [-h] [-f {json,shell,table,value,yaml}]
# [-c COLUMN] [--max-width <integer>]
# [--fit-width] [--print-empty] [--noindent]
# [--prefix PREFIX] [--id <id>] [--ram <size-mb>]
# [--disk <size-gb>] [--ephemeral <size-gb>]
# [--swap <size-mb>] [--vcpus <vcpus>]
# [--rxtx-factor <factor>] [--public | --private]
# [--property <key=value>] [--project <project>]
# [--description <description>]
# [--project-domain <project-domain>]
# <flavor-name>
OSC="openstack flavor create"
if [ "$1" != "run" ]; then
tip "Usage: $0 [run] -- $desc"
tip " add argument 'run' to execute these command really, otherwise show it on screen only."
tip ""
OSC="echo $OSC"
else
# check openrc env.
[ -z "$OS_USERNAME" ] && die "to run openstack command, you need source openrc file first."
fi
for cpu in $cpu_count_list; do
for ram in $ram_gb_list; do
ram_cpu_factor=$((ram/cpu))
[ $ram_cpu_factor -lt $ram_cpu_factor_min ] && \
{ tip "INFO: ignore flavor beacuse ram_cpu_factor is less \
than ram_cpu_factor_min: $ram/$cpu < $ram_cpu_factor_min"
continue; }
[ $ram_cpu_factor -gt $ram_cpu_factor_max ] && \
{ tip "INFO: ignore flavor beacuse ram_cpu_factor is more \
than ram_cpu_factor_max: $ram/$cpu > $ram_cpu_factor_max"
continue; }
for disk in $disk_gb_list; do
name="c$cpu-m${ram}G-d${disk}G"
$OSC --id "$name" \
--vcpus "$cpu" \
--ram $((ram*1024)) \
--disk "$disk" "$name"
sleep 0.01
done
done
doneThis is the view of the installation is complete
[root@wuhan31-ceph01 ~]# openstack flavor list
+--------------+--------------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------+--------------+-------+------+-----------+-------+-----------+
| c1-m1G-d20G | c1-m1G-d20G | 1024 | 20 | 0 | 1 | True |
| c1-m1G-d50G | c1-m1G-d50G | 1024 | 50 | 0 | 1 | True |
| c1-m2G-d20G | c1-m2G-d20G | 2048 | 20 | 0 | 1 | True |
| c1-m2G-d50G | c1-m2G-d50G | 2048 | 50 | 0 | 1 | True |
| c1-m4G-d20G | c1-m4G-d20G | 4096 | 20 | 0 | 1 | True |
| c1-m4G-d50G | c1-m4G-d50G | 4096 | 50 | 0 | 1 | True |
| c2-m2G-d20G | c2-m2G-d20G | 2048 | 20 | 0 | 2 | True |
| c2-m2G-d50G | c2-m2G-d50G | 2048 | 50 | 0 | 2 | True |
| c2-m4G-d20G | c2-m4G-d20G | 4096 | 20 | 0 | 2 | True |
| c2-m4G-d50G | c2-m4G-d50G | 4096 | 50 | 0 | 2 | True |
| c2-m8G-d20G | c2-m8G-d20G | 8192 | 20 | 0 | 2 | True |
| c2-m8G-d50G | c2-m8G-d50G | 8192 | 50 | 0 | 2 | True |
| c4-m16G-d20G | c4-m16G-d20G | 16384 | 20 | 0 | 4 | True |
| c4-m16G-d50G | c4-m16G-d50G | 16384 | 50 | 0 | 4 | True |
| c4-m4G-d20G | c4-m4G-d20G | 4096 | 20 | 0 | 4 | True |
| c4-m4G-d50G | c4-m4G-d50G | 4096 | 50 | 0 | 4 | True |
| c4-m8G-d20G | c4-m8G-d20G | 8192 | 20 | 0 | 4 | True |
| c4-m8G-d50G | c4-m8G-d50G | 8192 | 50 | 0 | 4 | True |
| c8-m16G-d20G | c8-m16G-d20G | 16384 | 20 | 0 | 8 | True |
| c8-m16G-d50G | c8-m16G-d50G | 16384 | 50 | 0 | 8 | True |
| c8-m8G-d20G | c8-m8G-d20G | 8192 | 20 | 0 | 8 | True |
| c8-m8G-d50G | c8-m8G-d50G | 8192 | 50 | 0 | 8 | True |
+--------------+--------------+-------+------+-----------+-------+-----------+
[root@wuhan31-ceph01 ~]#Virtual machine network
offers two kinds of networking. Direct access to the network and a private network for the virtual machine.
vlan planning please refer to the corresponding sections.
Through the network
to provide 4/24 segments can access up to four devices 251 *. (254 ip-1 gateway host -2dhcp), late if needs, self expansion.
# 创建可以直通内网的私有网络. 因为vlan id不是上述定义的范围, 所以需要使用管理员权限创建.
for net in {34..37}; do
openstack network create --provider-network-type vlan --provider-physical-network physnet0 --provider-segment "$net" --share --project admin net-lan$net
openstack subnet create --network net-lan$net --gateway 192.168.$net.1 --subnet-range 192.168.$net.0/24 --dns-nameserver 100.100.31.254 subnet-lan$net
donePrivate Network
Preferences 30 vlan, can be deployed in 30 separate networks, each subnet number and size with no restrictions.
Private network can freely create subnets and routing. Recommended for use only within the network to create a cluster.
If you need the outside world communication, can access the floating IP network. If you need to access from the outside world, need to bind floating IP, or use load balancing? (this section to be confirmed)
floating IP network currently has 250 IP, network memory if needed in a large number of virtual machines access from the outside, it is recommended to select "within the straight-through" approach to access the network.
# 创建外部网络, 管理员权限.
for net in {33..33}; do
openstack network create --external --provider-network-type vlan --provider-physical-network physnet1 --provider-segment "$net" --share --project antiy net-ext-lan$net
openstack subnet create --network net-ext-lan$net --gateway 192.168.$net.1 --subnet-range 192.168.$net.0/24 --dns-nameserver 100.100.31.254 subnet-floating$net
doneThe following operations can be completed as normal user:
# 创建私有网络. 用户权限亦可.
openstack network create --project antiy net-private-antiy01
# 创建路由.
openstack router create --ha --project antiy router-antiy
# 把路由接入网络, 我还没找到配置external network的命令, 建议这段在web界面配置.
#openstack router add subnet router-antiy subnet-private-antiy01
#openstack router add subnet router-antiy subnet-floating43Physical network configuration
arranged slightly, core x0 / 0/1 connected C8-41 x0 / 0/1
Above for the design and planning of the network, following the formal deployment
First, the basic environment ready
1, ready environment
| system | ip | CPU name | Character |
|---|---|---|---|
| centos7.4 | 100.100.31.201 | wuhan31-ceph01.v3.os | ceph01、kolla-ansible |
| centos7.4 | 100.100.31.202 | wuhan31-ceph02.v3.os | ceph02 |
| centos7.4 | 100.100.31.203 | wuhan31-ceph03.v3.os | ceph03 |
| centos7.4 | 100.100.31.101 | wuhan31-nova01.v3.os | nova01 |
| centos7.4 | 100.100.31.102 | wuhan31-nova02.v3.os | nova01 |
ip and host name is written to / etc / hosts Lane
2, modify the hostname
hostnamectl set-hostname wuhan31-ceph01.v3.os
hostnamectl set-hostname wuhan31-ceph02.v3.os
hostnamectl set-hostname wuhan31-ceph03.v3.os3, turn off the firewall, selinux
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 04, the configuration yum source:
修改yum源为公司内部源.
包括centos的cloud和ceph的mimic源:
curl -v http://mirrors.***.org/repo/centos7.repo > /etc/yum.repos.d/CentOS-Base.repo
curl -v http://mirrors.***.org/repo/cloud.repo > /etc/yum.repos.d/cloud.repo
yum makecache5, unified network card name
[root@localhost network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
TYPE=bond
ONBOOT=yes
IPADDR=100.100.31.203
NETMASK=255.255.255.0
GATEWAY=100.100.31.1
DNS1=192.168.55.55
USERCTL=no
BONDING_MASTER=yes
BONDING_OPTS="miimon=200 mode=1"
[root@localhost network-scripts]# cat ifcfg-em1
TYPE=Ethernet
BOOTPROTO=none
DEVICE=em1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
[root@localhost network-scripts]# cat ifcfg-em2
TYPE=Ethernet
BOOTPROTO=none
DEVICE=em2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
[root@localhost network-scripts]#All devices use the card name bond0
6, installation docker
Configuring docker yum source
cat > /etc/yum.repos.d/docker.repo <<EOF
[docker]
name=docker
baseurl=https://download.docker.com/linux/centos/7/x86_64/stable
enabled=1
gpgcheck=0
EOFThen install the docker-ce
curl http://mirrors.***.org/repo/docker.repo > /etc/yum.repos.d/docker.repo
yum install docker-ceConfigure private warehouse
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["http://mirrors.***.org:5000"]
}
EOFStart Service
systemctl enable docker
systemctl start docker7, install the required software
to all nodes need to be installed:
yum install ceph python-pip -y
Debugging aids, in order to facilitate debugging, the proposed installation completion script.
yum install bash-completion-extras libvirt-bash-completion net-tools bind-utils sysstat iftop nload tcpdump htop -y
8, mounting kolla-ansible
mounted pip.
yum install python-pip -y
Installation kolla-ansible dependent software required:
yum install ansible python2-setuptools python-cryptography python-openstackclient -y
Installation using pip kolla-ansible:
pip install kolla-ansible
note:
如果出现`requests 2.20.0 has requirement idna<2.8,>=2.5, but you'll have idna 2.4 which is incompatible.`错误，则强制更新requets库
pip install --ignore-installed requests
同样，出现Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.错误，强制更新
sudo pip install --ignore-installed PyYAMLNote: all the nodes operating step 1-7, 9, Step 8 deployment node operation (used here wuhan32-ceph01)
Second, the deployment ceph cluster
1, remote login user configuration ceph
all ceph node operation. (According to the public under their own machine to fill the actual situation, purpose of this step is to make ceph by key user login system)
ssh-keygen -t rsa //一路回车
usermod -s /bin/bash ceph
mkdir ~ceph/.ssh/
cat >> ~ceph/.ssh/authorized_keys << EOF
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDW6VghEC1cUrTZ6TfI9XcOEJZShkoL5YqtHBMtm2iZUnw8Pj6S3S1TCwKfdY0m+kInKlfZhoFCw3Xyee9XY7ZwPX6IEnixZMqO9EpC58LfxH841lw6xC0HesfF0QwWs+EVs5I1RwCN+Zoz2NPfu8RH30LHhBoSQpm75vRkF2trEbdtEI/kuzysO+73oF7R42lGJtgJtFbzLQSO2Vp/Xo7jdD/tdD/gcEsPniSPP3vFQg4EuSafdwxnJFuAxLAMCK+K1SQg7eNqboWYGhSWjOy39bTCZjieXOyNehPTVoqn3/qyC88c7D0PEbvTYxbNkuFU2MM7x9/k+ZGyvYnpex4t [email protected]
EOF
cat >> ~/.ssh/authorized_keys << EOF
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDW6VghEC1cUrTZ6TfI9XcOEJZShkoL5YqtHBMtm2iZUnw8Pj6S3S1TCwKfdY0m+kInKlfZhoFCw3Xyee9XY7ZwPX6IEnixZMqO9EpC58LfxH841lw6xC0HesfF0QwWs+EVs5I1RwCN+Zoz2NPfu8RH30LHhBoSQpm75vRkF2trEbdtEI/kuzysO+73oF7R42lGJtgJtFbzLQSO2Vp/Xo7jdD/tdD/gcEsPniSPP3vFQg4EuSafdwxnJFuAxLAMCK+K1SQg7eNqboWYGhSWjOy39bTCZjieXOyNehPTVoqn3/qyC88c7D0PEbvTYxbNkuFU2MM7x9/k+ZGyvYnpex4t [email protected]
EOF
cat > /etc/sudoers.d/ceph <<EOF
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
EOF
chown -R ceph:ceph ~ceph/.ssh/
chmod -R o-rwx ~ceph/.ssh/2. Create ceph cluster
deployment node operation
Installation deployment tools ceph-deploy
yum install ceph-deploy -y
mkdir ~ceph/ceph-deploy
cd ~ceph/ceph-deploy
ceph-deploy new wuhan31-ceph{01..03}.osEdit the configuration file ceph.conf
vim ceph.conf
[global]
fsid = 567be343-d631-4348-8f9d-2f18be36ce74
mon_initial_members = wuhan31t-ceph01, wuhan31-ceph02,wuhan31-ceph03
mon_host = wuhan31-ceph01,wuhan31-ceph02,wuhan31-ceph03
mon_addr = 100.100.31.201:6789,00.100.31.202:6789,00.100.31.203:6789
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
mon_allow_pool_delete = 1
[osd]
osd_client_message_size_cap = 524288000
osd_deep_scrub_stride = 131072
osd_op_threads = 2
osd_disk_threads = 1
osd_mount_options_xfs = "rw,noexec,nodev,noatime,nodiratime,nobarrier"
osd_recovery_op_priority = 1
osd_recovery_max_active = 1
osd_max_backfills = 1
osd-recovery-threads=1
[client]
rbd_cache = true
rbd_cache_size = 1073741824
rbd_cache_max_dirty = 134217728
rbd_cache_max_dirty_age = 5
rbd_cache_writethrough_until_flush = true
rbd_concurrent_management_ops = 50
rgw frontends = civetweb port=7480Then create the initial node:
ceph-deploy mon create-initial
ceph-deploy admin wuhan31-ceph01 wuhan31-ceph02,wuhan31-ceph03
# 可选: 允许ceph用户使用admin keyring.
sudo setfacl -m u:ceph:r /etc/ceph/ceph.client.admin.keyringCreating mgr:ceph-deploy mgr create wuhan31-ceph01 wuhan31-ceph02,wuhan31-ceph03
Add osd
Here is the reuse of the hard disk, so you need to zap disk :( my machine disk is sdb to sdk)
ceph-deploy disk zap wuhan31-ceph01 /dev/sd{b..k}
ceph-deploy disk zap wuhan31-ceph02 /dev/sd{b..k}
ceph-deploy disk zap wuhan31-ceph03 /dev/sd{b..k}You can add osd use the following batch script:
for dev in /dev/sd{b..k}; do ceph-deploy osd create --data "$dev"wuhan31-ceph01 || break; done
for dev in /dev/sd{b..k}; do ceph-deploy osd create --data "$dev" wuhan31-ceph02 || break; done
for dev in /dev/sd{b..k}; do ceph-deploy osd create --data "$dev" wuhan31-ceph03 || break; doneIf you encounter an error during the execution can continue alone.
3, create pools
Deploying node operation
to create the desired openstack pools:
calculated: https://ceph.com/pgcalc/
Since currently only 3 * 10 pg osd it follows the initial predetermined number disposed according to the number of different sizes pg:.. Press late need to expand.
images 32
volumes 256
vms 64
backups 128Ceph admin has privileges at any node performs created:
ceph osd pool create images 32
ceph osd pool create volumes 256
ceph osd pool create vms 64
ceph osd pool create backups 1284, create ceph client
deployment node operation
to create a client, and to give permission, the following information is written script execution or direct execution
# 定义客户端
clients="client.cinder client.nova client.glance client.cinder-backup"
# 创建客户端.
for client in $clients; do
ceph auth get-or-create "$client"
done
# 配置权限
ceph auth caps client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=cinder-ssd, allow rwx pool=vms, allow rwx pool=images'
ceph auth caps client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=cinder-ssd, allow rwx pool=vms, allow rwx pool=images'
ceph auth caps client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth caps client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
# 导出
for client in $clients; do
ceph auth export "$client" -o /etc/ceph/ceph."$client".keyring
doneDefinitions Create Client:
ceph auth get-or-create client.cinder
ceph auth get-or-create client.nova
ceph auth get-or-create client.glance
ceph auth get-or-create client.cinder-backupConfiguring Permissions
ceph auth caps client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=cinder-ssd, allow rwx pool=vms, allow rwx pool=images'
ceph auth caps client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=cinder-ssd, allow rwx pool=vms, allow rwx pool=images'
ceph auth caps client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth caps client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'Export keyting
ceph auth export client.cinder -o /etc/ceph/ceph.client.cinder.keyring
ceph auth export client.nova -o /etc/ceph/ceph.client.nova.keyring
ceph auth export client.glance -o /etc/ceph/ceph.client.glance.keyring
ceph auth export client.cinder-backup -o /etc/ceph/ceph.client.cinder-backup.keyring5, the configuration plug ceph dashboard
deployment operation node
ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/ssl false
ceph config set mgr mgr/dashboard/server_address ::
ceph config set mgr mgr/dashboard/server_port 7000
ceph dashboard set-login-credentials 用户名 密码Three, kolla deployment openstack
following deployment node operation
1, write a configuration
copy the template
to copy the template kolla-ansible, here is the use pip install:
Required: Copy configuration templatescp -ar /usr/share/kolla-ansible/etc_examples/* /etc/
2, generate passwords
be sure to complete the "Copy template" link. Unable to generate password otherwise
execute the following command tokolla-genpwd
glolals.yml
editorial changes /etc/kolla/globals.yml
# 这里是openstack的版本信息. 这里选择rocky版本,source即源码安装, 因为这种方式的软件包最全. 如果为binary且为CentOS系统, 那么只有红帽提供的包, 有些不全.
kolla_install_type: "source"
openstack_release: "rocky"
# 如果有多个控制节点, 则启用高可用, 注意, vip(虚拟IP)必须为目前未用到的IP. 且和节点IP位于同一网段.
enable_haproxy: "yes"
kolla_internal_vip_address: "100.100.31.254"
# 这些fqdn需要在内网DNS和hosts文件同时做好解析.
kolla_internal_fqdn: "xiaoxuantest.***.org"
kolla_external_fqdn: "xiaoxuantest.***.org"
# 这里就是自定义配置的路径. 只在部署节点上.
node_custom_config: "/etc/kolla/config"
# 虚拟化类型, 如果是在虚拟机里做实验, 这里的类型需要改为qemu. 慢点就慢点.
# kvm类型需要CPU,主板和BIOS支持, 且BIOS启用了硬件虚拟化. 如果在计算节点无法安装kvm内核模块, 请根据dmesg报错排查.
nova_compute_virt_type: "kvm"
# 网络接口. 注意external必须为独立接口, 不然会导致节点断网.
neutron_external_interface: "eth1"
network_interface: "bond0"
api_interface: "bond0"
storage_interface: "bond0"
cluster_interface: "bond0"
tunnel_interface: "bond0"
# dns_interface: "eth" # dns功能未集成, 后期自行研究吧.
# 网络虚拟化技术. 我们这里不使用openvswitch, 直接使用linuxbridge
neutron_plugin_agent: "linuxbridge"
enable_openvswitch: "no"
# 网络高可用, 就是创建多个agent: dhcp和l3(路由)
enable_neutron_agent_ha: "yes"
# 网络封装, 目前都是vlan, flat留着备用, 用于直接使用物理网卡.
neutron_type_drivers: "flat,vlan"
# 租户网络的隔离方式, 这里是vlan, 但是kolla不支持, 所以我们需要自己在node_custom_config这项对应的目录里加自定义配置.
neutron_tenant_network_types: "vlan"
# 网络插件
enable_neutron_lbaas: "yes"
enable_neutron_***aas: "yes"
enable_neutron_fwaas: "yes"
# elk集中日志管理
enable_central_logging: "yes"
# 启用debug模式, 日志很详细. 按需临时开启.
#openstack_logging_debug: "True"
# 忘了这里的用途... 可以关了试试, 如果其他组件有依赖会自动开的.
enable_kafka: "yes"
enable_fluentd: "yes"
# 这里是我们使用了外部的ceph, 不让kolla部署, 因为kolla部署时部分osd可能会出问题, 导致osd id顺序错位, 看着不方便. 而且后期从主机管理存储集群也别捏.
enable_ceph: "no"
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"
gnocchi_backend_storage: "ceph"
enable_manila_backend_cephfs_native: "yes"
# 启用的功能.
#enable_ceilometer: "yes"
enable_cinder: "yes"
#enable_designate: "yes"
enable_destroy_images: "yes"
#enable_gnocchi: "yes"
enable_grafana: "yes"
enable_heat: "yes"
enable_horizon: "yes"
#enable_ironic: "yes"
#enable_ironic_ipxe: "yes"
#enable_ironic_neutron_agent: "yes"
#enable_kuryr: "yes"
#enable_magnum: "yes"
# enable_neutron_dvr
# enable_ovs_dpdk
#enable_nova_serialconsole_proxy: "yes"
#enable_octavia: "yes"
enable_redis: "yes"
#enable_trove: "yes"
# 其他配置
glance_backend_file: "no"
#designate_ns_record: "nova."
#ironic_dnsmasq_dhcp_range: "11.0.0.10,11.0.0.111"
openstack_region_name: "xiaoxuantest"3, inventory write files
to create a directory for writing inventory file:
mkdir kolla-ansible
cp /usr/share/kolla-ansible/ansible/inventory/multinode kolla-ansible/inventory-xiaoxuantestInventory file key elements of the edited elsewhere unchanged:
key elements
[control]
wuhan31-ceph01
wuhan31-ceph02
wuhan31-ceph03
[network]
wuhan31-ceph01
wuhan31-ceph02
wuhan31-ceph03
[external-compute]
wuhan31-ceph01
wuhan31-ceph02
wuhan31-ceph03
[monitoring:children]
control
[storage:children]
control4, ceph integrated
network
because we use the vlan, so the need to manually configure:
mkdir /etc/kolla/config/neutron
cat > /etc/kolla/config/neutron/ml2_conf.ini <<EOF
[ml2_type_vlan]
network_vlan_ranges = physnet0:1031:1060,physnet1
[linux_bridge]
physical_interface_mappings = physnet0:eth0,physnet1:eth1
EOFDashboard
to create a virtual machine interface prohibited by default create a new volume.
mkdir /etc/kolla/config/horizon/
cat > /etc/kolla/config/horizon/custom_local_settings <<EOF
LAUNCH_INSTANCE_DEFAULTS = {
'create_volume': False,
}
EOFDirect paste / etc / kolla / config / directory of all files
[root@wuhan32-ceph01 config]# ls -lR
.:
total 4
lrwxrwxrwx. 1 kolla kolla 19 Mar 11 17:06 ceph.conf -> /etc/ceph/ceph.conf
drwxr-xr-x. 4 kolla kolla 117 Mar 28 14:43 cinder
-rw-r--r--. 1 root root 39 Mar 28 14:39 cinder.conf
drwxr-xr-x. 2 kolla kolla 80 Mar 11 17:18 glance
drwxr-xr-x. 2 root root 35 Mar 19 11:21 horizon
drwxr-xr-x. 2 root root 26 Mar 14 15:49 neutron
drwxr-xr-x. 2 kolla kolla 141 Mar 11 17:18 nova
./cinder:
total 8
lrwxrwxrwx. 1 kolla kolla 19 Mar 11 17:10 ceph.conf -> /etc/ceph/ceph.conf
drwxr-xr-x. 2 kolla kolla 81 Mar 11 17:18 cinder-backup
-rwxr-xr-x. 1 kolla kolla 274 Feb 26 16:47 cinder-backup.conf
drwxr-xr-x. 2 kolla kolla 40 Mar 11 17:18 cinder-volume
-rwxr-xr-x. 1 kolla kolla 534 Mar 28 14:38 cinder-volume.conf
./cinder/cinder-backup:
total 0
lrwxrwxrwx. 1 kolla kolla 43 Mar 11 17:18 ceph.client.cinder-backup.keyring -> /etc/ceph/ceph.client.cinder-backup.keyring
lrwxrwxrwx. 1 kolla kolla 36 Mar 11 17:18 ceph.client.cinder.keyring -> /etc/ceph/ceph.client.cinder.keyring
./cinder/cinder-volume:
total 0
lrwxrwxrwx. 1 kolla kolla 36 Mar 11 17:18 ceph.client.cinder.keyring -> /etc/ceph/ceph.client.cinder.keyring
./glance:
total 4
lrwxrwxrwx. 1 kolla kolla 36 Mar 11 17:18 ceph.client.glance.keyring -> /etc/ceph/ceph.client.glance.keyring
lrwxrwxrwx. 1 kolla kolla 19 Mar 11 17:07 ceph.conf -> /etc/ceph/ceph.conf
-rwxr-xr-x. 1 kolla kolla 138 Feb 27 11:55 glance-api.conf
./horizon:
total 4
-rw-r--r--. 1 root root 59 Mar 19 11:21 custom_local_settings
./neutron:
total 4
-rw-r--r--. 1 root root 141 Mar 14 15:49 ml2_conf.ini
./nova:
total 8
lrwxrwxrwx. 1 kolla kolla 36 Mar 11 17:18 ceph.client.cinder.keyring -> /etc/ceph/ceph.client.cinder.keyring
lrwxrwxrwx. 1 kolla kolla 34 Mar 11 17:18 ceph.client.nova.keyring -> /etc/ceph/ceph.client.nova.keyring
lrwxrwxrwx. 1 kolla kolla 19 Mar 11 17:07 ceph.conf -> /etc/ceph/ceph.conf
-rwxr-xr-x. 1 kolla kolla 101 Feb 27 17:28 nova-compute.conf
-rwxr-xr-x. 1 kolla kolla 39 Dec 24 16:52 nova-scheduler.confExcerpts config directory configuration file as follows:
# find -type f -printf "=== FILE: %p ===\n" -exec cat {} \;
=== FILE: ./cinder/cinder-backup.conf ===
[DEFAULT]
backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user=cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_pool=backups
backup_driver = cinder.backup.drivers.ceph
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true
=== FILE: ./cinder/cinder-volume.conf ===
[DEFAULT]
enabled_backends=cinder-sas,cinder-ssd
[cinder-sas]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=volumes
volume_backend_name=cinder-sas
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid=5b3ec4eb-c276-4cf2-a042-8ec906d05f69
[cinder-ssd]
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
backend_host=rbd:volumes
rbd_pool=cinder-ssd
volume_backend_name=cinder-ssd
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_secret_uuid=5b3ec4eb-c276-4cf2-a042-8ec906d05f69
=== FILE: ./glance/glance-api.conf ===
[glance_store]
default_store = rbd
stores = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
=== FILE: ./nova/nova-compute.conf ===
[libvirt]
images_rbd_pool=vms
images_type=rbd
images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=nova
=== FILE: ./nova/nova-scheduler.conf ===
[DEFAULT]
scheduler_max_attempts = 100
=== FILE: ./neutron/ml2_conf.ini ===
[ml2_type_vlan]
network_vlan_ranges = physnet0:1000:1030,physnet1
[linux_bridge]
physical_interface_mappings = physnet0:eth0,physnet1:eth1
=== FILE: ./horizon/custom_local_settings ===
LAUNCH_INSTANCE_DEFAULTS = {
'create_volume': False,
}
=== FILE: ./cinder.conf ===
[DEFAULT]
default_volume_type=standard* Note,. / Cinder / cinder-volume.conf rbd_secret_uuid = fill in the below this value
[root@wuhan31-ceph01 ~]# grep cinder_rbd_secret_uuid /etc/kolla/passwords.yml
cinder_rbd_secret_uuid: 1ae2156b-7c33-4fbb-a26a-c770fadc54b6Create a connection as follows
ceph.conf:
ln -s /etc/ceph/ceph.conf /etc/kolla/config/nova/
ln -s /etc/ceph/ceph.conf /etc/kolla/config/glance/
ln -s /etc/ceph/ceph.conf /etc/kolla/config/cinder/
keyring:
ln -s /etc/ceph/ceph.client.cinder-backup.keyring /etc/kolla/config/cinder/cinder-backup/
ln -s /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/
ln -s /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-volume/
ln -s /etc/ceph/ceph.client.glance.keyring /etc/kolla/config/glance/
ln -s /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/nova/
ln -s /etc/ceph/ceph.client.nova.keyring /etc/kolla/config/nova/5, deployment openstack
Provisioning node environment
bootstrap will also install a lot of things, the latter may consider installing them in advance.kolla-ansible -i kolla-ansible/inventory-xiaoxuantest bootstrap-servers
Pre-inspectionkolla-ansible -i kolla-ansible/inventory-xiaoxuantest prechecks
Formal deploymentkolla-ansible -i kolla-ansible/inventory-xiaoxuantest deploy
Adjust the configuration and re-deploy
if needed to adjust the configuration. Then after editing globals.yml, and then run the reconfigure. Use -t parameters can only be adjusted to change the module.
kolla-ansible -i kolla-ansible/inventory-xiaoxuantest reconfigure -t neutron
kolla-ansible -i kolla-ansible/inventory-xiaoxuantest deploy -t neutronComplete the deployment
of this step is mainly generated admin-openrc.sh.
kolla-ansible post-deploy
. /etc/kolla/admin-openrc.shThe initial demo
After performing cirros will automatically download images, create networks, and create a number of virtual test machine.
/usr/share/kolla-ansible/init-runonceQueries Password:
grep admin /etc/kolla/passwords.ymlIf there is no internal source to deploy a long time, foreign resources to download very slowly.
The official Troubleshooting Guide:https://docs.openstack.org/kolla-ansible/latest/user/troubleshooting.html
Run after the results of the deployment container as follows:
[root@wuhan31-ceph01 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d141ac504ec6 kolla/centos-source-grafana:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks grafana
41e12c24ba7f kolla/centos-source-horizon:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks horizon
6989a4aeb33a kolla/centos-source-heat-engine:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks heat_engine
35665589b4a4 kolla/centos-source-heat-api-cfn:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks heat_api_cfn
f18c98468796 kolla/centos-source-heat-api:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks heat_api
bc77f4d3c957 kolla/centos-source-neutron-metadata-agent:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_metadata_agent
7334b93c6564 kolla/centos-source-neutron-lbaas-agent:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_lbaas_agent
0dd7a55245c4 kolla/centos-source-neutron-l3-agent:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_l3_agent
beec0f19ec7f kolla/centos-source-neutron-dhcp-agent:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_dhcp_agent
af6841ebc21e kolla/centos-source-neutron-linuxbridge-agent:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_linuxbridge_agent
49dc0457445d kolla/centos-source-neutron-server:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks neutron_server
677c0be4ab6b kolla/centos-source-nova-compute:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_compute
402b1e673777 kolla/centos-source-nova-novncproxy:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_novncproxy
e35729b76996 kolla/centos-source-nova-consoleauth:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_consoleauth
8b193f562e47 kolla/centos-source-nova-conductor:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_conductor
885581445be0 kolla/centos-source-nova-scheduler:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_scheduler
171128b7bcb7 kolla/centos-source-nova-api:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_api
8d7f3de2ad63 kolla/centos-source-nova-placement-api:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks placement_api
ab763320f268 kolla/centos-source-nova-libvirt:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_libvirt
bbd4c3e2c961 kolla/centos-source-nova-ssh:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks nova_ssh
80e7098f0bfb kolla/centos-source-cinder-backup:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks cinder_backup
20e2ff43d0e1 kolla/centos-source-cinder-volume:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks cinder_volume
6caba29f7ce2 kolla/centos-source-cinder-scheduler:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks cinder_scheduler
3111622e4e83 kolla/centos-source-cinder-api:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks cinder_api
2c011cfae829 kolla/centos-source-glance-api:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks glance_api
be84e405afdd kolla/centos-source-kafka:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks kafka
09aef04ad59e kolla/centos-source-keystone-fernet:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks keystone_fernet
2ba9e19844fd kolla/centos-source-keystone-ssh:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks keystone_ssh
8eebe226b065 kolla/centos-source-keystone:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks keystone
662d85c00a64 kolla/centos-source-rabbitmq:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks rabbitmq
7d373ef0fdee kolla/centos-source-mariadb:rocky "dumb-init kolla_sta…" 8 weeks ago Up 8 weeks mariadb
ab9f5d612925 kolla/centos-source-memcached:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks memcached
a728298938f7 kolla/centos-source-kibana:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks kibana
7d22d71cc31b kolla/centos-source-keepalived:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks keepalived
dae774ca7e33 kolla/centos-source-haproxy:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks haproxy
14b340bb8139 kolla/centos-source-redis-sentinel:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks redis_sentinel
3023e95f465f kolla/centos-source-redis:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks redis
a3ed7e8fe8ff kolla/centos-source-elasticsearch:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks elasticsearch
06b28cd0f7c7 kolla/centos-source-zookeeper:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks zookeeper
630219f5fb29 kolla/centos-source-chrony:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks chrony
6f6189a4dfda kolla/centos-source-cron:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks cron
039f08ec1bbf kolla/centos-source-kolla-toolbox:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks kolla_toolbox
f839d23859cc kolla/centos-source-fluentd:rocky "dumb-init --single-…" 8 weeks ago Up 8 weeks fluentd
[root@wuhan31-ceph01 ~]# 
This is added to the network, all using straight-through network
Fourth, the common faults
mariadb, this situation is encountered after the shutdown of all nodes.
VIP status monitor 3306, node 3306 non-listening state, a restart of the container
may be disconnected at the same time because the nodes, MariaDB service is not available, the need to restore service kolla-ansible -i kolla-ansible/inventory-xiaoxuantest mariadb_recovery
execution confirmation after each node to listen shape 3306
Problems encountered:
ironic: Checking Files exist for Agent-ironic Ironic Inspector
TASK [ironic : Checking ironic-agent files exist for Ironic Inspector] ********************
failed: [localhost -> localhost] (item=ironic-agent.kernel) => {"changed": false, "failed_when_result": true, "item": "ironic-agent.kernel", "stat": {"exists": false}}
failed: [localhost -> localhost] (item=ironic-agent.initramfs) => {"changed": false, "failed_when_result": true, "item": "ironic-agent.initramfs", "stat": {"exists": false}} Temporarily shut down at the beginning of enable_ironic configuration solution.
neutron : Checking if 'MountFlags' for docker service is set to 'shared'
TASK [neutron : Checking if 'MountFlags' for docker service is set to 'shared'] ***********
fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["systemctl", "show", "docker"], "delta": "0:00:00.010391", "end": "2018-12-24 20:44:46.791156", "failed_when_result": true, "rc": 0, "start": "2018-12-24 20:44:46.780765",...See section Environmental ready --docker--
ceilometer : Checking gnocchi backend for ceilometer
TASK [ceilometer : Checking gnocchi backend for ceilometer] *******************************
fatal: [localhost -> localhost]: FAILED! => {"changed": false, "msg": "gnocchi is required but not enabled"}Enable gnocchi
octavia : Checking certificate files exist for octavia
TASK [octavia : Checking certificate files exist for octavia] *****************************
failed: [localhost -> localhost] (item=cakey.pem) => {"changed": false, "failed_when_result": true, "item": "cakey.pem", "stat": {"exists": false}}
failed: [localhost -> localhost] (item=ca_01.pem) => {"changed": false, "failed_when_result": true, "item": "ca_01.pem", "stat": {"exists": false}}
failed: [localhost -> localhost] (item=client.pem) => {"changed": false, "failed_when_result": true, "item": "client.pem", "stat": {"exists": false}}Run kolla-ansible certificates still not generated, the next check, the official did not fix: https://bugs.launchpad.net/kolla-ansible/+bug/1668377
disable octavia latter part of the investigation artificially generated.
common : Restart fluentd container
RUNNING HANDLER [common : Restart fluentd container] **************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unknown error message: Get https://192.168.55.201:4000/v1/_ping: dial tcp 100.100.31.201:4000: getsockopt: connection refused"}Under looked really did not start the 4000 port. According to the official documents [Reference 1] deployed registry.