glusterfs 的缩容、扩容和替换(转载)

Source: https://www.cnblogs.com/bfmq/p/9990467.html

  • 1. 扩容操作
1 [[email protected] ~]# gluster peer probe g3                # 将新节点添加到集群里,如果是原本集群内的机器操作则省略
2 peer probe: success. Host g3 port 24007 already in peer list        # 这台机器添加过了
3 [[email protected] ~]# gluster volume info test                             # 此时该卷块设备为4个
4 5 Volume Name: test
6 Type: Distributed-Replicate
7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4
8 Status: Started
9 Snapshot Count: 0
10 Number of Bricks: 2 x 2 = 4
11 Transport-type: tcp
12 Bricks:
13 Brick1: g1:/glusterfs/data1
14 Brick2: g2:/glusterfs/data1
15 Brick3: g3:/glusterfs/data1
16 Brick4: g1:/glusterfs/data2
17 [[email protected] ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 g3:/glusterfs/data3
18 volume add-brick: failed: Incorrect number of bricks supplied 5 with count 2        # 很明显又是之前的块设备与复制数备份问题。因此注意服务器上的磁盘数量要与卷复制数匹配问题,比如一个复制数为3的卷,买了10块盘其中一块是加不进来的
19 [[email protected] ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3
20 volume add-brick: failed: The brick g1:/glusterfs/data3 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior.        # 我还是用的/所以要强制
21 [[email protected] ~]# gluster volume add-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 force
22 volume add-brick: success
23 [[email protected] ~]# gluster volume info test                 # 卷的块设备变多了
24 25 Volume Name: test
26 Type: Distributed-Replicate
27 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4
28 Status: Started
29 Snapshot Count: 0
30 Number of Bricks: 4 x 2 = 8
31 Transport-type: tcp
32 Bricks:
33 Brick1: g1:/glusterfs/data1
34 Brick2: g2:/glusterfs/data1
35 Brick3: g3:/glusterfs/data1
36 Brick4: g1:/glusterfs/data2
37 Brick5: g2:/glusterfs/data2
38 Brick6: g3:/glusterfs/data2
39 Brick7: g1:/glusterfs/data3
40 Brick8: g2:/glusterfs/data3
41 [[email protected] ~]# gluster volume rebalance test start                # 让以前的数据再次均匀分布
42 volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebalance process.
43 ID: a2f4b603-283a-4303-8ad0-84db00adb5a5
44 [[email protected] ~]# gluster volume rebalance test status            # 查看任务状态,要均衡文件较大时需要一段时间
45                                     Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
46                                ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------47                                localhost                2        0Bytes            10             0             0            completed        0:00:00
48                                       g2                1        0Bytes             9             0             0            completed        0:00:00
49                                       g3                3        0Bytes             6             0             0            completed        0:00:00
50 volume rebalance: test: success
51 [[email protected] ~]# gluster volume rebalance test stop                # 等所有状态completed就可以停了
52                                     Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
53                                ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------54                                localhost                2        0Bytes            10             0             0            completed        0:00:00
55                                       g2                1        0Bytes             9             0             0            completed        0:00:00
56                                       g3                3        0Bytes             6             0             0            completed        0:00:00
57 volume rebalance: test: success: rebalance process may be in the middle of a file migration.
58 The process will be fully stopped once the migration of the file is complete.
59 Please check rebalance process for completion before doing any further brick related tasks on the volume.
6061 [[email protected] ~]# gluster volume rebalance test status        # 现在该卷上已经没有在均衡的任务了
62 volume rebalance: test: failed: Rebalance not started for volume test.
  • 2.缩容操作
1 [[email protected] ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 start                # 去掉复制倍数的块设备,此时去除的设备上的数据会开始迁移
2 Running remove-brick with cluster.force-migration enabled can result in data corruption. It is safer to disable this option so that files that receive writes during migration are not migrated.
3 Files that are not migrated can then be manually copied after the remove-brick commit operation.
4 Do you want to continue with your current cluster.force-migration settings? (y/n) y
5 volume remove-brick start: success
6 ID: 9b4657c0-ed29-4c75-8bb6-7b8f277f02ec
7 [[email protected] ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status            # 查看迁移状态
8                                     Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
9                                ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------10                                localhost                0        0Bytes            10             0             0            completed        0:00:00
11                                       g2                0        0Bytes             5             0             0            completed        0:00:00
12                                       g3                0        0Bytes             5             0             0            completed        0:00:00
13 [[email protected] ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 commit            # completed后从卷中将块设备去除掉
14 volume remove-brick commit: success
15 Check the removed bricks to ensure all files are migrated.
16 If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick.
17 [[email protected] ~]# gluster volume remove-brick test g2:/glusterfs/data2 g3:/glusterfs/data2 g1:/glusterfs/data3 g2:/glusterfs/data3 status            # 没有任务
18 volume remove-brick status: failed: remove-brick not started for volume test.
19 [[email protected] ~]# gluster volume info test                 # 又变回了以前的4块盘
20 21 Volume Name: test
22 Type: Distributed-Replicate
23 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4
24 Status: Started
25 Snapshot Count: 0
26 Number of Bricks: 2 x 2 = 4
27 Transport-type: tcp
28 Bricks:
29 Brick1: g1:/glusterfs/data1
30 Brick2: g2:/glusterfs/data1
31 Brick3: g3:/glusterfs/data1
32 Brick4: g1:/glusterfs/data2
33 [[email protected] ~]# ls -a /glusterfs/data3        # 将刚才所有被去除的块设备下隐藏目录删除,否则可能会影响到以后再次使用此块设备加入其它卷
34 .  ..  .glusterfs
35 [[email protected] ~]# rm -fr /glusterfs/data3/.glusterfs/
  • 3.更换操作
1 [[email protected] ~]# gluster volume replace-brick test g3:/glusterfs/data2 g3:/glusterfs/data3 commit force                    # 使用g3:/glusterfs/data3替换掉g3:/glusterfs/data2
2 volume replace-brick: success: replace-brick commit force operation successful
3 [[email protected] ~]# gluster volume info test
4 5 Volume Name: test
6 Type: Distributed-Replicate
7 Volume ID: 92ffe586-ea14-4b7b-9b89-5dfd626cb6d4
8 Status: Started
9 Snapshot Count: 0
10 Number of Bricks: 4 x 2 = 8
11 Transport-type: tcp
12 Bricks:
13 Brick1: g1:/glusterfs/data1
14 Brick2: g2:/glusterfs/data1
15 Brick3: g3:/glusterfs/data1
16 Brick4: g1:/glusterfs/data2
17 Brick5: g2:/glusterfs/data2
18 Brick6: g3:/glusterfs/data3                    # 被换掉了
19 Brick7: g1:/glusterfs/data3
20 Brick8: g2:/glusterfs/data3
其他的维护命令:
https://www.cnblogs.com/jicki/p/5801712.html

1. 查看GlusterFS中所有的volume:
[[email protected] ~]#gluster volume list
2. 删除GlusterFS磁盘: [[email protected] ~]#gluster volume stop models #停止名字为 models 的磁盘 [[email protected] ~]#gluster volume delete models #删除名字为 models 的磁盘 
注: 删除 磁盘 以后,必须删除 磁盘( /opt/gluster/data ) 中的 ( .glusterfs/ .trashcan/ )目录。 
否则创建新 volume 相同的 磁盘 会出现文件 不分布,或者 类型 错乱 的问题。
3. 卸载某个节点GlusterFS磁盘 [[email protected] ~]#gluster peer detach swarm-node-2 
4. 设置访问限制,按照每个volume 来限制
[[email protected] ~]#gluster volume set models auth.allow 10.6.0.*,10.7.0.*
5. 添加GlusterFS节点: [[email protected] ~]#gluster peer probe swarm-node-3 [[email protected] ~]#gluster volume add-brick models swarm-node-3:/opt/gluster/data 注:如果是复制卷或者条带卷,则每次添加的Brick数必须是replica或者stripe的整数倍 
6. 配置卷 [[email protected] ~]# gluster volume set 
7. 缩容volume: 先将数据迁移到其它可用的Brick,迁移结束后才将该Brick移除: [[email protected] ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data start 
在执行了start之后,可以使用status命令查看移除进度: [[email protected] ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data status 
不进行数据迁移,直接删除该Brick:
[[email protected] ~]#gluster volume remove-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit
注意,如果是复制卷或者条带卷,则每次移除的Brick数必须是replica或者stripe的整数倍。
扩容:
gluster volume add-brick models swarm-node-2:/opt/gluster/data

8. 修复命令:
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit -force

9. 迁移volume:
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data start
pause 为暂停迁移
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data pause
abort 为终止迁移
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data abort
status 查看迁移状态
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data status
迁移结束后使用commit 来生效
[[email protected] ~]#gluster volume replace-brick models swarm-node-2:/opt/gluster/data swarm-node-3:/opt/gluster/data commit
10. 均衡volume:

[[email protected] ~]#gluster volume models lay-outstart [[email protected] ~]#gluster volume models start [[email protected] ~]#gluster volume models startforce [[email protected] ~]#gluster volume models status [[email protected] ~]#gluster volume models stop
This entry was posted in Glusterfs, Servuce and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *