Ceph-Nautilus之Ansible配置手册

1.1 拓扑

+--------------------+ +--------------------+ | [node1.1000cc.net] | | [node5.1000cc.net] | | Ceph-Ansible | | client | | 192.168.10.11 | | 192.168.10.15 | +--------------------+ +--------------------+ |--------------|-----------------| | +----------------------------+----------------------------+ | | | |192.168.10.12 |192.168.10.13 |192.168.10.14 +-----------+-----------+ +-----------+-----------+ +-----------+-----------+ | [node02.1000cc.net] | | [node03.1000cc.net] | | [node04.1000cc.net] | | Object Storage +----+ Object Storage +----+ Object Storage | | Monitor Daemon | | | | | | Manager Daemon | | | | | +-----------------------+ +-----------------------+ +-----------------------+

1.2 配置Ceph Nautilus

1) 为所有OSD主机,增加一块不小于5G的硬盘.位置位于各主机的sdb 2) 关闭EPEL源，否则ansible版本将出现冲突 3) 创建管理账户于所有节点,并赋予sudo权限及ssh免密登录 [root@node1 ~]# echo -e 'Defaults:snow !requiretty\nsnow ALL = (root) NOPASSWD:ALL' > /etc/sudoers.d/ceph [root@node1 ~]# pscp.pssh -h host-list.txt /etc/sudoers.d/ceph /etc/sudoers.d/ [1] 16:25:39 [SUCCESS] root@node4.1000cc.net [2] 16:25:39 [SUCCESS] root@node2.1000cc.net [3] 16:25:39 [SUCCESS] root@node3.1000cc.net [4] 16:25:39 [SUCCESS] root@node5.1000cc.net [root@node1 ~]# pssh -h host-list.txt -i 'ls -l /etc/sudoers.d/ceph' [1] 16:35:58 [SUCCESS] root@node4.1000cc.net -rw-r--r-- 1 root root 57 Jan 26 16:35 /etc/sudoers.d/ceph [2] 16:35:58 [SUCCESS] root@node3.1000cc.net -rw-r--r-- 1 root root 57 Jan 26 16:35 /etc/sudoers.d/ceph [3] 16:35:58 [SUCCESS] root@node2.1000cc.net -rw-r--r-- 1 root root 57 Jan 26 16:35 /etc/sudoers.d/cep [4] 16:35:58 [SUCCESS] root@node5.1000cc.net -rw-r--r-- 1 root root 57 Jan 26 16:35 /etc/sudoers.d/cephh [root@node1 ~]# pssh -h host-list.txt -i 'useradd snow' [1] 16:39:11 [SUCCESS] root@node1.1000cc.net [2] 16:39:11 [SUCCESS] root@node2.1000cc.net [3] 16:39:11 [SUCCESS] root@node4.1000cc.net [4] 16:39:11 [SUCCESS] root@node3.1000cc.net [5] 16:39:11 [SUCCESS] root@node5.1000cc.net [root@host1 ~]# pssh -h host-list.txt -i 'echo 123456 | passwd --stdin snow' [1] 16:39:37 [SUCCESS] root@node2.1000cc.net Changing password for user snow. passwd: all authentication tokens updated successfully. [2] 16:39:37 [SUCCESS] root@node1.1000cc.net Changing password for user snow. passwd: all authentication tokens updated successfully. [3] 16:39:37 [SUCCESS] root@node4.1000cc.net Changing password for user snow. passwd: all authentication tokens updated successfully. [4] 16:39:37 [SUCCESS] root@node3.1000cc.net Changing password for user snow. passwd: all authentication tokens updated successfully. [5] 16:39:37 [SUCCESS] root@node5.1000cc.net Changing password for user snow. passwd: all authentication tokens updated successfully. [root@node1 ~]# su - snow [snow@node1 ~]$ ssh-keygen -N '' Generating public/private rsa key pair. Enter file in which to save the key (/home/snow/.ssh/id_rsa): Created directory '/home/snow/.ssh'. Your identification has been saved in /home/snow/.ssh/id_rsa. Your public key has been saved in /home/snow/.ssh/id_rsa.pub. The key fingerprint is: SHA256:2nrNBsv6c4ZBFtBKQk691gTS6BsoPGvAGRWDF0FF6ho snow@node1.1000cc.net The key's randomart image is: +---[RSA 2048]----+ | +OX*o+ | | o +=.+ + | |o ++.o = . | |o=o o + + | |.Eo. + oS | | oo . oo | |.. ...B | | .* * | | o+.= | +----[SHA256]-----+ [snow@node1 ~]$ vim ~/.ssh/config Host node1 Hostname node1.1000cc.net User snow Host node2 Hostname node2.1000cc.net User snow Host node3 Hostname node3.1000cc.net User snow Host node4 Hostname node4.1000cc.net User snow Host node5 Hostname node5.1000cc.net User snow [snow@node1 ~]$ chmod 600 ~/.ssh/config [snow@node1 ~]$ ssh-copy-id node2 [snow@node1 ~]$ ssh-copy-id node3 [snow@node1 ~]$ ssh-copy-id node4 [snow@node1 ~]$ ssh-copy-id node5 4) 安装Ceph Ansible软件包 [snow@node1 ~]$ sudo pssh -h host-list.txt -i 'yum install epel-release centos-release-ceph-nautilus centos-release-openstack-stein -y' [1] 16:43:29 [SUCCESS] root@node4.1000cc.net [2] 16:43:29 [SUCCESS] root@node2.1000cc.net [3] 16:43:29 [SUCCESS] root@node3.1000cc.net [4] 16:43:29 [SUCCESS] root@node5.1000cc.net [snow@node1 ~]$ sudo yum install ceph-ansible -y 5) 配置Ceph Cluster [snow@node1 ~]$ sudo vim /usr/share/ceph-ansible/group_vars/all.yml ceph_origin: repository ceph_repository: community ceph_repository_type: cdn ceph_stable_release: nautilus fetch_directory: ~/ceph-ansible-keys # 设置监控nic monitor_interface: enp0s3 # 设置监控的网络段 public_network: 192.168.10.0/24 # 设置集群网络 cluster_network: "{{ public_network }}" [snow@node1 ~]$ sudo vim /usr/share/ceph-ansible/group_vars/osds.yml # 指定至存储节点时，设备名称 devices: - /dev/sdb [snow@node1 ~]$ sudo vim /etc/ansible/hosts ...... ...... ...... ...... ...... ...... # 于最后追加如下内容 [all:vars] ansible_ssh_user=snow ansible_become=true ansible_become_method=sudo ansible_become_user=root # 设置MonitorNode [mons] node2.1000cc.net # 设置Manager节点 [mgrs] node2.1000cc.net # 设置OSD节点 [osds] node2.1000cc.net node3.1000cc.net node4.1000cc.net [grafana-server] node2.1000cc.net node3.1000cc.net node4.1000cc.net 6) 运行Playbook来设置Ceph Cluster [snow@node1 ~]$ cd /usr/share/ceph-ansible [snow@node1 ceph-ansible]$ sudo cp -p site.yml.sample site.yml [snow@node1 ceph-ansible]$ ansible-playbook site.yml ...... ...... ...... PLAY RECAP *************************************************************************************************** node2.1000cc.net : ok=351 changed=12 unreachable=0 failed=0 skipped=431 rescued=0 ignored=0 node3.1000cc.net : ok=206 changed=7 unreachable=0 failed=0 skipped=273 rescued=0 ignored=0 node4.1000cc.net : ok=208 changed=7 unreachable=0 failed=0 skipped=271 rescued=0 ignored=0 INSTALLER STATUS *************************************************************************************************** Install Ceph Monitor : Complete (0:00:32) Install Ceph OSD : Complete (0:01:25) Install Ceph Dashboard : Complete (0:01:02) Install Ceph Grafana : Complete (0:02:16) Install Ceph Node Exporter : Complete (0:02:21) Monday 27 January 2020 11:02:32 +0800 (0:00:00.121) 0:09:23.027 ******** =============================================================================== ceph-container-engine : install container and lvm2 packages -------------------------------- 92.96s ceph-grafana : wait for grafana to start --------------------------------------------------- 33.01s gather and delegate facts ------------------------------------------------------------------ 22.53s ceph-common : configure red hat ceph community repository stable key ----------------------- 10.74s ceph-dashboard : set or update dashboard admin username and password ------------------------ 9.30s ceph-prometheus : ship systemd services ----------------------------------------------------- 5.45s ceph-grafana : make sure grafana configuration directories exist ---------------------------- 5.12s ceph-osd : apply operating system tuning ---------------------------------------------------- 4.62s ceph-grafana : download ceph grafana dashboards --------------------------------------------- 4.36s ceph-prometheus : start prometheus services ------------------------------------------------- 4.19s ceph-config : look up for ceph-volume rejected devices -------------------------------------- 3.62s ceph-node-exporter : ship systemd service --------------------------------------------------- 3.35s ceph-infra : open monitor and manager ports ------------------------------------------------- 3.11s ceph-dashboard : disable mgr dashboard module (restart) ------------------------------------- 3.08s ceph-grafana : write grafana.ini ------------------------------------------------------------ 3.00s ceph-osd : systemd start osd ---------------------------------------------------------------- 2.96s ceph-container-engine : start container service --------------------------------------------- 2.96s ceph-grafana : install ceph-grafana-dashboards package on RedHat or SUSE -------------------- 2.86s ceph-grafana : make sure grafana is down ---------------------------------------------------- 2.84s ceph-grafana : write datasources provisioning config file ----------------------------------- 2.83s # 问题1:因ceph-ansible的配置原因。默认其ceph源指向http://download.ceph.com,key指向http://mirrors.ustc.edu.cn/ceph。download.ceph.com有时候极慢无比。可直接更换快速的源。如果打算更改。可修改以下文件 [snow@node1 ~]$ sudo vim /usr/share/ceph-ansible/roles/ceph-defaults/defaults/main.yml # 修改144-145行，改为以下内容: ceph_mirror: http://mirrors.ustc.edu.cn/ceph ceph_stable_key: http://mirrors.ustc.edu.cn/ceph/keys/release.asc # 问题2:因网络问题。如果到install redhat ceph packages处，被重试N次。请使用-vvvv执行。如果报错找不到”liboath”请安装。如果还是存在问题。可将以下软件包安装到所有节点上。 $ sudo yum -y install epel-release centos-release-ceph-nautilus centos-release-openstack-stein 7) 测试完整情况 [snow@node1 ~]$ ssh node2 "ceph --version" ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) [snow@node1 ~]$ ssh node2 "sudo ceph -s" cluster: id: 23adeee8-aa37-4a7d-8286-33ec0fa44281 health: HEALTH_OK> services: mon: 1 daemons, quorum node2 (age 7h) mgr: node2(active, since 28m) osd: 3 osds: 3 up (since 30m), 3 in (since 30m) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 3.0 GiB used, 24 GiB / 27 GiB avail pgs:

1.3 配置客户端

1) 给Client添加snow账户并赋予sudo权限 [root@node5 ~]# echo -e 'Defaults:snow !requiretty\nsnow ALL = (root) NOPASSWD:ALL' > /etc/sudoers.d/ceph 2) 在Node1上为Ansible配置client并初始化 [snow@node1 ~]$ ssh-copy-id node5 [snow@node1 ~]$ sudo vim /usr/share/ceph-ansible/group_vars/clients.yml copy_admin_key: true [snow@node1 ~]$ sudo vim /etc/ansible/hosts ...... ...... ...... ...... ...... ...... # 在文件最底部追加客户端信息 [clients] node5.1000cc.net # 初始化客户端 [snow@node1 ~]$ cd /usr/share/ceph-ansible [snow@node1 ceph-ansible]$ ansible-playbook site.yml --limit=clients ...... ...... PLAY RECAP *************************************************************************************************** node5.1000cc.net : ok=125 changed=15 unreachable=0 failed=0 skipped=257 rescued=0 ignored=0 INSTALLER STATUS *************************************************************************************************** Install Ceph Client : Complete (0:00:23) Install Ceph Node Exporter : Complete (0:01:37)

2.1 块存储实现

1) 创建RBD存储池 [snow@node5 ~]$ sudo ceph osd pool create rbd 0 pool 'rbd' created [snow@node5 ~]$ sudo rbd pool init rbd [snow@node5 ~]$ sudo rbd create rbd1 --size 2G --image-feature layering [snow@node5 ~]$ sudo rbd ls -l NAME SIZE PARENT FMT PROT LOCK rbd1 2 GiB 2 2) 映射 [snow@node5 ~]$ sudo rbd map rbd1 /dev/rbd0 [snow@node5 ~]$ rbd showmapped id pool namespace image snap device 0 rbd rbd1 - /dev/rbd0 3) 使用 [snow@node5 ~]$ sudo mkfs.ext4 /dev/rbd0 [snow@node5 ~]$ sudo mount /dev/rbd0 /mnt [snow@node5 ~]$ df -Th | grep /mnt /dev/rbd0 ext4 2.0G 6.0M 1.8G 1% /mnt

2.2 使用Ceph文件系统

1) 创建MDS [snow@node1 ~]$ sudo vim /etc/ansible/hosts ...... ...... ...... ...... ...... ...... # 于最后追加 [mdss] node2.1000cc.net [snow@node1 ~]$ cd /usr/share/ceph-ansible [snow@node1 ceph-ansible]$ ansible-playbook site.yml --limit=mdss ...... ...... INSTALLER STATUS *********************************************************************************************** Install Ceph Monitor : Complete (0:00:36) Install Ceph Manager : Complete (0:00:43) Install Ceph OSD : Complete (0:00:49) Install Ceph MDS : Complete (0:01:17) Install Ceph Dashboard : Complete (0:00:56) Install Ceph Grafana : Complete (0:01:05) Install Ceph Node Exporter : Complete (0:00:32) ...... ...... [snow@node2 ~]$ sudo ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] [snow@node2 ~]$ sudo ceph mds stat cephfs:1 {0=node2=up:active} [snow@node2 ~]$ sudo ceph fs status cephfs cephfs - 1 clients ====== +------+--------+-------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-------+---------------+-------+-------+ | 0 | active | node2 | Reqs: 0 /s | 10 | 13 | +------+--------+-------+---------------+-------+-------+ +-----------------+----------+-------+-------+ | Pool | type | used | avail | +-----------------+----------+-------+-------+ | cephfs_metadata | metadata | 1536k | 16.9G | | cephfs_data | data | 0 | 16.9G | +-----------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) 2) 客户端使用Ceph FS [snow@node5 ~]$ sudo yum install ceph-fuse -y # 获取管理秘钥 [snow@node5 ~]$ ssh snow@node2.1000cc.net "sudo ceph-authtool -p /etc/ceph/ceph.client.admin.keyring" > admin.key [snow@node5 ~]$ chmod 600 admin.key [snow@node5 ~]$ sudo mount.ceph node2.1000cc.net:/ /mnt -o name=admin,secretfile=admin.key [snow@node5 ~]$ df -Th | grep /mnt 192.168.10.12:/ ceph 17G 0 17G 0% /mnt

2.3 开启Object Gateway

1) 配置 [snow@node1 ~]$ sudo vim /usr/share/ceph-ansible/group_vars/rgws.yml radosgw_interface: eth0 [snow@node1 ~]$ sudo vim /etc/ansible/hosts ...... ...... ...... ...... ...... ...... # 于最底部追加 [rgws] node5.1000cc.net [snow@node1 ~]$ cd /usr/share/ceph-ansible [snow@node1 ceph-ansible]$ ansible-playbook site.yml --limit=rgws ...... ...... PLAY RECAP *************************************************************************************************** node5.1000cc.net : ok=194 changed=13 unreachable=0 failed=0 skipped=329 rescued=0 ignored=0 INSTALLER STATUS *************************************************************************************************** Install Ceph RGW : Complete (0:00:38) Install Ceph Client : Complete (0:00:27) Install Ceph Node Exporter : Complete (0:00:33) ...... ...... [snow@node1 ~]$ curl node5.1000cc.net:8080 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> 2) 客户端调用 [snow@node5 ~]$ sudo radosgw-admin user create --uid=gzliu --display-name="Gz Lau" --email=admin@1000cc.net { "user_id": "gzliu", "display_name": "Gz Lau", "email": "admin@1000cc.net", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "gzliu", "access_key": "KCJ634DW8NXLEKZ84B6D", "secret_key": "RpMj9yaqaKvMVyRNO3Iw0TkU07K820jH2TJG1BFv" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } [snow@node5 ~]$ sudo radosgw-admin user list [ "gzliu" ] [snow@node5 ~]$ sudo radosgw-admin user info --uid=gzliu { "user_id": "gzliu", "display_name": "Gz Lau", "email": "admin@1000cc.net", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "gzliu", "access_key": "KCJ634DW8NXLEKZ84B6D", "secret_key": "RpMj9yaqaKvMVyRNO3Iw0TkU07K820jH2TJG1BFv" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } 3) 验证 [root@node1 ~]# yum install python-boto -y [root@node1 ~]# vim s3.py import sys import boto import boto.s3.connection # 加入user的key ACCESS_KEY = 'KCJ634DW8NXLEKZ84B6D' SECRET_KEY = 'RpMj9yaqaKvMVyRNO3Iw0TkU07K820jH2TJG1BFv' # 设置Object Gateway的FQDN及所监听的端口 HOST = 'node5.1000cc.net' PORT = 8080 conn = boto.connect_s3( aws_access_key_id = ACCESS_KEY, aws_secret_access_key = SECRET_KEY, port = PORT, host = HOST, is_secure = False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), ) # 创建[snow-test]的bucket bucket = conn.create_bucket('snow-test') # 显示bucket列表 for bucket in conn.get_all_buckets(): print "{name}\t{created}".format( name = bucket.name, created = bucket.creation_date, ) [root@node1 ~]# python s3.py snow-test 2020-01-29T17:54:42.371Z