Source： https://www.cnblogs.com/bfmq/p/9990467.html
- 1. 扩容操作
1 [root@g1 ~]# gluster peer probe g3 # 将新节点添加到集群里，如果是原本集群内的机器操作则省略 2 peer probe: success. Host g3 port 24007 already in peer list # 这台机器添加过了 3 [root@g1 ~]# gluster volume info test # 此时该卷块设备为4个 4 5 Volume Name: test 6 Type: Distributed-Replicate 7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 2 x 2 = 4 11 Transport-type: tcp 12 Bricks: 13 Brick1: g1:/glusterfs/data1 14 Brick2: g2:/glusterfs/data1 15 Brick3: g3:/glusterfs/data1 16 Brick4: g1:/glusterfs/data2 17 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 g3:/glusterfs/data3 18 volume add-brick: failed: Incorrect number of bricks supplied 5 with count 2 # 很明显又是之前的块设备与复制数备份问题。因此注意服务器上的磁盘数量要与卷复制数匹配问题，比如一个复制数为3的卷，买了10块盘其中一块是加不进来的 19 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 20 volume add-brick: failed: The brick g1:/glusterfs/data3 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior. # 我还是用的/所以要强制 21 [root@g1 ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 force 22 volume add-brick: success 23 [root@g1 ~]# gluster volume info test # 卷的块设备变多了 24 25 Volume Name: test 26 Type: Distributed-Replicate 27 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 28 Status: Started 29 Snapshot Count: 0 30 Number of Bricks: 4 x 2 = 8 31 Transport-type: tcp 32 Bricks: 33 Brick1: g1:/glusterfs/data1 34 Brick2: g2:/glusterfs/data1 35 Brick3: g3:/glusterfs/data1 36 Brick4: g1:/glusterfs/data2 37 Brick5: g2:/glusterfs/data2 38 Brick6: g3:/glusterfs/data2 39 Brick7: g1:/glusterfs/data3 40 Brick8: g2:/glusterfs/data3 41 [root@g1 ~]# gluster volume rebalance test start # 让以前的数据再次均匀分布 42 volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebalance process. 43 ID: a2f4b603-283a-4303-8ad0-84db00adb5a5 44 [root@g1 ~]# gluster volume rebalance test status # 查看任务状态，要均衡文件较大时需要一段时间 45 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 46 --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------47 localhost 2 0Bytes 10 0 0 completed 0:00:00 48 g2 1 0Bytes 9 0 0 completed 0:00:00 49 g3 3 0Bytes 6 0 0 completed 0:00:00 50 volume rebalance: test: success 51 [root@g1 ~]# gluster volume rebalance test stop # 等所有状态completed就可以停了 52 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 53 --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------54 localhost 2 0Bytes 10 0 0 completed 0:00:00 55 g2 1 0Bytes 9 0 0 completed 0:00:00 56 g3 3 0Bytes 6 0 0 completed 0:00:00 57 volume rebalance: test: success: rebalance process may be in the middle of a file migration. 58 The process will be fully stopped once the migration of the file is complete. 59 Please check rebalance process for completion before doing any further brick related tasks on the volume. 6061 [root@g1 ~]# gluster volume rebalance test status # 现在该卷上已经没有在均衡的任务了 62 volume rebalance: test: failed: Rebalance not started for volume test.
- 2.缩容操作
1 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 start # 去掉复制倍数的块设备，此时去除的设备上的数据会开始迁移 2 Running remove-brick with cluster.force-migration enabled can result in data corruption. It is safer to disable this option so that files that receive writes during migration are not migrated. 3 Files that are not migrated can then be manually copied after the remove-brick commit operation. 4 Do you want to continue with your current cluster.force-migration settings? (y/n) y 5 volume remove-brick start: success 6 ID: 9b4657c0-ed29-4c75-8bb6-7b8f277f02ec 7 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status # 查看迁移状态 8 Node Rebalanced-files size scanned failures skipped status run time in h:m:s 9 --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------10 localhost 0 0Bytes 10 0 0 completed 0:00:00 11 g2 0 0Bytes 5 0 0 completed 0:00:00 12 g3 0 0Bytes 5 0 0 completed 0:00:00 13 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 commit # completed后从卷中将块设备去除掉 14 volume remove-brick commit: success 15 Check the removed bricks to ensure all files are migrated. 16 If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. 17 [root@g1 ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status # 没有任务 18 volume remove-brick status: failed: remove-brick not started for volume test. 19 [root@g1 ~]# gluster volume info test # 又变回了以前的4块盘 20 21 Volume Name: test 22 Type: Distributed-Replicate 23 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 24 Status: Started 25 Snapshot Count: 0 26 Number of Bricks: 2 x 2 = 4 27 Transport-type: tcp 28 Bricks: 29 Brick1: g1:/glusterfs/data1 30 Brick2: g2:/glusterfs/data1 31 Brick3: g3:/glusterfs/data1 32 Brick4: g1:/glusterfs/data2 33 [root@g1 ~]# ls -a /glusterfs/data3 # 将刚才所有被去除的块设备下隐藏目录删除，否则可能会影响到以后再次使用此块设备加入其它卷 34 . .. .glusterfs 35 [root@g1 ~]# rm -fr /glusterfs/data3/.glusterfs/
- 3.更换操作
1 [root@g1 ~]# gluster volume replace-brick test g3:/glusterfs/data2 g3:/glusterfs/data3 commit force # 使用g3:/glusterfs/data3替换掉g3:/glusterfs/data2 2 volume replace-brick: success: replace-brick commit force operation successful 3 [root@g1 ~]# gluster volume info test 4 5 Volume Name: test 6 Type: Distributed-Replicate 7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4 8 Status: Started 9 Snapshot Count: 0 10 Number of Bricks: 4 x 2 = 8 11 Transport-type: tcp 12 Bricks: 13 Brick1: g1:/glusterfs/data1 14 Brick2: g2:/glusterfs/data1 15 Brick3: g3:/glusterfs/data1 16 Brick4: g1:/glusterfs/data2 17 Brick5: g2:/glusterfs/data2 18 Brick6: g3:/glusterfs/data3 # 被换掉了 19 Brick7: g1:/glusterfs/data3 20 Brick8: g2:/glusterfs/data3
其他的维护命令： https://www.cnblogs.com/jicki/p/5801712.html 1. 查看GlusterFS中所有的volume: [root@swarm-manager ~]#gluster volume list
2. 删除GlusterFS磁盘： [root@swarm-manager ~]#gluster volume stop models #停止名字为 models 的磁盘 [root@swarm-manager ~]#gluster volume delete models #删除名字为 models 的磁盘
注： 删除 磁盘 以后，必须删除 磁盘( /opt/gluster/data ) 中的 （ .glusterfs/ .trashcan/ ）目录。
否则创建新 volume 相同的 磁盘 会出现文件 不分布，或者 类型 错乱 的问题。
3. 卸载某个节点GlusterFS磁盘 [root@swarm-manager ~]#gluster peer detach swarm-node-2
4. 设置访问限制,按照每个volume 来限制 [root@swarm-manager ~]#gluster volume set models auth.allow 10.6.0.*,10.7.0.*
5. 添加GlusterFS节点： [root@swarm-manager ~]#gluster peer probe swarm-node-3 [root@swarm-manager ~]#gluster volume add-brick models swarm-node-3:/opt/gluster/data 注：如果是复制卷或者条带卷，则每次添加的Brick数必须是replica或者stripe的整数倍
6. 配置卷 [root@swarm-manager ~]# gluster volume set
7. 缩容volume: 先将数据迁移到其它可用的Brick，迁移结束后才将该Brick移除： [root@swarm-manager ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data start
在执行了start之后，可以使用status命令查看移除进度： [root@swarm-manager ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data status
不进行数据迁移，直接删除该Brick： [root@swarm-manager ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit
注意，如果是复制卷或者条带卷，则每次移除的Brick数必须是replica或者stripe的整数倍。
扩容： gluster volume add-brick models swarm-node-2:/opt/gluster/data
8. 修复命令: [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit -force
9. 迁移volume: [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data start
pause 为暂停迁移 [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data pause
abort 为终止迁移 [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data abort
status 查看迁移状态 [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data status
迁移结束后使用commit 来生效 [root@swarm-manager ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit
10. 均衡volume:
[root@swarm-manager ~]#gluster volume models lay-outstart [root@swarm-manager ~]#gluster volume models start [root@swarm-manager ~]#gluster volume models startforce [root@swarm-manager ~]#gluster volume models status [root@swarm-manager ~]#gluster volume models stop