Ceph集群部署实战

其他 2018-12-31 17:19:03 阅读次数: 0

建议系统内核升级到4.x，不然容易出各种问题，不要给自己找不必要的麻烦。 参考文章:Centos7 内核编译升级到指定的版本
https://blog.csdn.net/happyfreeangel/article/details/85088706

[root@ceph-admin ceph-ansible-3.1.7]# more hosts

[admins]
10.20.4.10

[osds]
10.20.4.21
10.20.4.22
10.20.4.23

[rgws]
10.20.4.11
10.20.4.12
10.20.4.13

[mds]
10.20.4.31

[clients]
10.20.4.51

[rbdmirrors]
10.20.4.51

[mons]
10.20.4.1
10.20.4.2
10.20.4.3

[mgrs]
10.20.4.1
10.20.4.2
10.20.4.3

[agents]
10.20.4.51

1.自动创建虚拟机 （OK）
我是用vmware Esxi 做实验的，全场采用自动化创建虚拟机。

在这里插入图片描述
1.1 确保虚拟机有2个磁盘 一个是系统本身的,来自虚拟机模版, 默认是/dev/sda,
另外是/dev/sdb
/dev/sdc
为了实验，我创建了/dev/sdb 我是都在osd 主机上创建的，每个创建500GB 也就是 /dev/sdb 容量500GB .
这个可以通过ansible 自动化完成.

2.设置免密码登陆. 详细请参考文章:

   #下面这个操作会创建{{ceph_username}}帐号,下面的group, owner 等创建文件夹时会用到.
- name: "安装不同主机间用户{{hostconfig['hadoop_config']['the_ceph_username']}}之间免密码登陆 passwordless-ssh-login"
  include_role:
    name: passwordless-ssh-login
  vars:
    user_host_list: "{{passwordless_host_list}}"
    username: "root"
    password: "{{ceph_salt_password}}"
    sudo_privilege: False
    auto_generate_etc_host_list: True
    debug_mode: False

 #下面这个操作会创建{{ceph_username}}帐号,下面的group, owner 等创建文件夹时会用到.
- name: "安装不同主机间用户{{hostconfig['hadoop_config']['the_ceph_username']}}之间免密码登陆 passwordless-ssh-login"
  include_role:
    name: passwordless-ssh-login
  vars:
    user_host_list: "{{passwordless_host_list}}"
    username: "{{ceph_username}}"
    password: "{{ceph_salt_password}}"
    sudo_privilege: True
    debug_mode: False

3.关闭所有yum repo 源头,设置yum 源头,请把docker-ce.repo 重命名一下,这个连接到国外,老卡住,容易导致安装失败.

  1. ceph-admin 安装ansible 安装虚拟环境
    4.1 确保你的虚拟机已经有了python
    4.2 安装虚拟机环境 virtual-env
    pip install virtualenv
    virtualenv -p python2 venv #可以指定python版本 python3
    #python3 -m venv venv #如果你有安装python3的话，也可以这样操作.

5.切换到虚拟环境
source venv/bin/activate

  1. 在ceph-admin 虚拟机上 安装指定版本的ansible
    6.1 切换目录 cd /home/ceph

6.2 用pip执行安装ansible #务必严格按照官方文档的要求来，ansible版本过新或旧都会有各种报错。 这里注意不要用yum install ansible
pip install ansible==2.4.2

或者用git 下载
#6.2 下载ceph-ansible 部署脚本
wget -c https://github.com/ceph/ceph-ansible/archive/v3.1.7.tar.gz
tar xf v3.1.7.tar.gz
cd ceph-ansible-3.1.7
或 git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible
git checkout remotes/origin/stable-3.2 # 这里刚好没有3.1.7 版本。

#在虚拟环境里安装依赖包
pip install -r requirements.txt

  1. 准备好配置文件

cp group_vars/all.yml.sample group_vars/all.yml
cp group_vars/osds.yml.sample group_vars/osds.yml
cp site.yml.sample site.yml

vim group_vars/all.yml

[root@ceph-admin group_vars]# more all.yml
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://mirrors.163.com/ceph
ceph_stable_key: http://mirrors.163.com/ceph/keys/release.asc
ceph_stable_release: luminous
ceph_stable_repo: “{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}”

#fsid: 82D6CE06-6E92-4C2A-AB26-11FF63B7E67D ##通过uuidgen生成
fsid: 82d6ce06-6e92-4c2a-ab26-11ff63b7e67d ##通过uuidgen生成 建议改成小写的,大写的之前运行脚本有问题会报错.
generate_fsid: false

cephx: true

public_network: 10.20.0.0/16
cluster_network: 10.20.0.0/16
monitor_interface: ens160

ceph_conf_overrides:
global:
rbd_default_features: 7
auth cluster required: cephx
auth service required: cephx
auth client required: cephx
osd journal size: 2048
osd pool default size: 3
osd pool default min size: 1
mon_pg_warn_max_per_osd: 1024
osd pool default pg num: 128
osd pool default pgp num: 128
max open files: 131072
osd_deep_scrub_randomize_ratio: 0.01

mgr:
mgr modules: dashboard

mon:
mon_allow_pool_delete: true

client:
rbd_cache: true
rbd_cache_size: 335544320
rbd_cache_max_dirty: 134217728
rbd_cache_max_dirty_age: 10

osd:
osd mkfs type: xfs
# osd mount options xfs: “rw,noexec,nodev,noatime,nodiratime,nobarrier”
ms_bind_port_max: 7100
osd_client_message_size_cap: 2147483648
osd_crush_update_on_start: true
osd_deep_scrub_stride: 131072
osd_disk_threads: 4
osd_map_cache_bl_size: 128
osd_max_object_name_len: 256
osd_max_object_namespace_len: 64
osd_max_write_size: 1024
osd_op_threads: 8

osd_recovery_op_priority: 1
osd_recovery_max_active: 1
osd_recovery_max_single_start: 1
osd_recovery_max_chunk: 1048576
osd_recovery_threads: 1
osd_max_backfills: 4
osd_scrub_begin_hour: 23
osd_scrub_end_hour: 7

# bluestore block create: true
# bluestore block db size: 73014444032
# bluestore block db create: true
# bluestore block wal size: 107374182400
# bluestore block wal create: true

vim group_vars/osds.yml
devices:

  • /dev/sdb

- /dev/vdd

- /dev/vde

osd_scenario: collocated
osd_objectstore: bluestore

#osd_scenario: non-collocated
#osd_objectstore: bluestore
#devices:

- /dev/sdc

- /dev/sdd

- /dev/sde

#dedicated_devices:

- /dev/sdf

- /dev/sdf

- /dev/sdf

#bluestore_wal_devices:

- /dev/sdg

- /dev/sdg

- /dev/sdg

#monitor_address: 192.168.66.125

注释不需要的组件

vim site.yml

Defines deployment design and assigns role to server groups

  • hosts:
    • mons

- agents

  • osds
  • mdss

- rgws

- nfss

- restapis

- rbdmirrors

  • clients
  • mgrs

- iscsigws

- iscsi-gws # for backward compatibility only!

[root@ceph-admin group_vars]# more osds.yml
devices:

  • /dev/sdb
    osd_scenario: collocated
    osd_objectstore: bluestore

#osd_scenario: non-collocated
#osd_objectstore: bluestore
#devices:

- /dev/sdc

- /dev/sdd

- /dev/sde

#dedicated_devices:

- /dev/sdf

- /dev/sdf

- /dev/sdf

#bluestore_wal_devices:

- /dev/sdg

- /dev/sdg

- /dev/sdg

#monitor_address: 192.168.66.125

注释不需要的组件

vim site.yml

Defines deployment design and assigns role to server groups

  • hosts:
    • mons

- agents

  • osds
  • mdss

- rgws

- nfss

- restapis

- rbdmirrors

  • clients
  • mgrs

- iscsigws

- iscsi-gws # for backward compatibility only!

客户端卸载pip中安装的urllib3，不然会失败

pip freeze|grep urllib3
pip uninstall urllib3

ansible-playbook -i hosts site.yml

至此ceph部署完成，登陆ceph节点检查状态。

清空集群
如果部署过程中出现报错，建议先清空集群 再进行部署操作

cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必须copy到项目根目录下
ansible-playbook -i hosts purge-cluster.yml

创建客户端访问环境:
[root@mon1 ~]# ceph auth get-or-create client.rbd mon ‘allow r’ osd ‘allow class-read object_prefix rbd_children,allow rwx pool=rbd’
[client.rbd]
key = AQAMyxxc1zSRBRAAhp8HTqmXXky8azntPF7gdQ==
[root@mon1 ~]#

[root@mon1 ~]# ceph auth get-or-create client.rbd | ssh ceph@client1 sudo tee /etc/ceph/ceph.client.rbd.keyring
The authenticity of host ‘client1 (10.20.4.51)’ can’t be established.
ECDSA key fingerprint is SHA256:1Li21Z7NsbSmDsHubXT2R7tYFNM003JYfEuSuV8mSJ4.
ECDSA key fingerprint is MD5:b9:d0:23:5d:12:26:84:b9:76:a6:d6:7a:e9:9a:b0:46.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘client1,10.20.4.51’ (ECDSA) to the list of known hosts.
ceph@client1’s password:
[client.rbd]
key = AQAMyxxc1zSRBRAAhp8HTqmXXky8azntPF7gdQ==
[root@mon1 ~]#

#在client1 上操作
cat /etc/ceph/ceph.client.rbd.keyring >> /etc/ceph/keyring

Since we are not using the default user client.admin we

need to supply username that will connect to the Ceph cluster
ceph -s --name client.rbd

猜你喜欢

转载自blog.csdn.net/happyfreeangel/article/details/85171736
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
周排行
Python环境安装与基础语法（1）——计算机基础知识
IMU预积分
ADAS中的LDW、FCW、BSD、LCA、ACC、AEB、APA、DMS代表的含义
B站笔试两道题
skyeye arm 硬件虚拟机环境的搭建
Web前端静态页面示例
数组-合并排序数组 II-简单
springcloud之版本问题启动报错
面向对象-------------匿名对象(六)
输入URL到页面呈现中间发生了什么？
每日归档
更多
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022