RocketMQ 笔记4 HA主从同步

编程语言 2018-05-18 00:19:24 阅读次数: 3
broker主从同步包括同步双写 和 异步复制， 其实这两种都是使用相同的方式传输的，同步双写只是在主线程上阻塞了
一段时间，等待传输结果，返回给producer


master针对每一个slave之间会有一个长连接，都会保存已经传输的offset,启动一个线程commitlog最大offset大于已经传输的offset
时候，就会通过这个连接往slave写数据，格式为：thisOffset（8字节）+ size(4字节) + body


当slave收到消息时，根据规则取出消息内容，调用defaultMessageStore.appendToCommitLog写入文件，同时返回一个确认消息
reportSlaveMaxOffset，消息格式为：maxOffset（8字节），master收到后，会写入一个byteBufferRead中，更新slaveAckOffset，
唤醒同步方式阻塞的线程




1.相关的类
org.apache.rocketmq.store.ha.HAService  处理主从同步的类
org.apache.rocketmq.store.ha.HAService.AcceptSocketService    master启动的一个线程，启动一个ServerSocket，等待slave的连接
org.apache.rocketmq.store.ha.HAConnection  master服务收到slave连接请求建立的连接，内部又启动了两个线程处理读写，因为可能多个slave
所以可能有多个(HAService.List<HAConnection>)
org.apache.rocketmq.store.ha.HAService.HAClient  slave启动一个线程连接master服务器，收取消息写入文件，返回确认消息
org.apache.rocketmq.store.ha.HAService.GroupTransferService  同步双写的使用，不停检测是否写成功，唤醒阻塞线程




2.master端启动过程
org.apache.rocketmq.store.ha.HAService.AcceptSocketService.beginAccept()
新建serverSocketChannel  selector，注册OP_ACCEPT事件


org.apache.rocketmq.store.ha.HAService.AcceptSocketService.run() 处理连接事件
SocketChannel sc = ((ServerSocketChannel) k.channel()).accept(); 得到连接
HAConnection conn = new HAConnection(HAService.this, sc); 
新建了writeSocketService  readSocketService 各自新建了一个Selector，分别注册
OP_WRITE  OP_READ 事件


conn.start();
启动writeSocketService  readSocketService线程


HAService.this.addConnection(conn);


3.HAConnection传输过程
HAConnection属性
private volatile long slaveRequestOffset = -1;
private volatile long slaveAckOffset = -1;


ReadSocketService属性
private final ByteBuffer byteBufferRead = ByteBuffer.allocate(READ_MAX_BUFFER_SIZE);//写缓存
private int processPostion = 0;//写缓存当前处理的位置
private volatile long lastReadTimestamp = System.currentTimeMillis();//最后一次读时间戳


WriteSocketService属性
private final ByteBuffer byteBufferHeader = ByteBuffer.allocate(headerSize);//保存消息头
private long nextTransferFromWhere = -1;//下一次传输起始offset
private SelectMappedBufferResult selectMappedBufferResult;//保存需要传输的消息内容
private boolean lastWriteOver = true;//最后一次写是否结束
private long lastWriteTimestamp = System.currentTimeMillis();//最后一次写时间戳


WriteSocketService工作过程

org.apache.rocketmq.store.ha.HAConnection.WriteSocketService.run()

public void run() {
            HAConnection.log.info(this.getServiceName() + " service started");

            while (!this.isStopped()) {
                try {
                    this.selector.select(1000);

                    //slave启动时候会先上报当前salve的commitLog最大偏移
                    //这里应该是先等待上报
                    if (-1 == HAConnection.this.slaveRequestOffset) {
                        Thread.sleep(10);
                        continue;
                    }

                    //这里应该是刚启动时候的计算，假如slave为空，那么上报了0给slaveRequestOffset
                    //这个计算就有点不懂了，此时不是应该从0开始传输吗
                    if (-1 == this.nextTransferFromWhere) {
                        if (0 == HAConnection.this.slaveRequestOffset) {
                            long masterOffset = HAConnection.this.haService.getDefaultMessageStore().getCommitLog().getMaxOffset();
                            masterOffset =
                                masterOffset
                                    - (masterOffset % HAConnection.this.haService.getDefaultMessageStore().getMessageStoreConfig()
                                    .getMapedFileSizeCommitLog());

                            if (masterOffset < 0) {
                                masterOffset = 0;
                            }

                            this.nextTransferFromWhere = masterOffset;
                        } else {
                            this.nextTransferFromWhere = HAConnection.this.slaveRequestOffset;
                        }

                        log.info("master transfer data from " + this.nextTransferFromWhere + " to slave[" + HAConnection.this.clientAddr
                            + "], and slave request " + HAConnection.this.slaveRequestOffset);
                    }

                    if (this.lastWriteOver) {//如果最后一次写成功，进行新的一次，否则继续写上一次的

                        long interval =
                            HAConnection.this.haService.getDefaultMessageStore().getSystemClock().now() - this.lastWriteTimestamp;
                        //间隔大于5秒，发送只包含header的消息，作为校验，以及心跳
                        if (interval > HAConnection.this.haService.getDefaultMessageStore().getMessageStoreConfig()
                            .getHaSendHeartbeatInterval()) {

                            // Build Header
                            this.byteBufferHeader.position(0);
                            this.byteBufferHeader.limit(headerSize);
                            this.byteBufferHeader.putLong(this.nextTransferFromWhere);
                            this.byteBufferHeader.putInt(0);
                            this.byteBufferHeader.flip();

                            this.lastWriteOver = this.transferData();
                            if (!this.lastWriteOver)
                                continue;
                        }
                    } else {
                        this.lastWriteOver = this.transferData();
                        if (!this.lastWriteOver)
                            continue;
                    }

                    //从nextTransferFromWhere处读取需要传输的消息内容
                    SelectMappedBufferResult selectResult =
                        HAConnection.this.haService.getDefaultMessageStore().getCommitLogData(this.nextTransferFromWhere);
                    if (selectResult != null) {
                        int size = selectResult.getSize();
                        //一次最大32k
                        if (size > HAConnection.this.haService.getDefaultMessageStore().getMessageStoreConfig().getHaTransferBatchSize()) {
                            size = HAConnection.this.haService.getDefaultMessageStore().getMessageStoreConfig().getHaTransferBatchSize();
                        }

                        long thisOffset = this.nextTransferFromWhere;

                        //增大nextTransferFromWhere用于下一次传输
                        this.nextTransferFromWhere += size;

                        selectResult.getByteBuffer().limit(size);
                        this.selectMappedBufferResult = selectResult;

                        // Build Header
                        this.byteBufferHeader.position(0);
                        this.byteBufferHeader.limit(headerSize);
                        this.byteBufferHeader.putLong(thisOffset);
                        this.byteBufferHeader.putInt(size);
                        this.byteBufferHeader.flip();

                        //往slave写消息
                        this.lastWriteOver = this.transferData();
                    } else {

                        HAConnection.this.haService.getWaitNotifyObject().allWaitForRunning(100);
                    }
                } catch (Exception e) {

                    HAConnection.log.error(this.getServiceName() + " service has exception.", e);
                    break;
                }
            }

            if (this.selectMappedBufferResult != null) {
                this.selectMappedBufferResult.release();
            }

            this.makeStop();

            readSocketService.makeStop();

            haService.removeConnection(HAConnection.this);

            SelectionKey sk = this.socketChannel.keyFor(this.selector);
            if (sk != null) {
                sk.cancel();
            }

            try {
                this.selector.close();
                this.socketChannel.close();
            } catch (IOException e) {
                HAConnection.log.error("", e);
            }

            HAConnection.log.info(this.getServiceName() + " service end");
        }

private boolean transferData() throws Exception {
            int writeSizeZeroTimes = 0;
            // Write Header
            while (this.byteBufferHeader.hasRemaining()) {
                int writeSize = this.socketChannel.write(this.byteBufferHeader);
                if (writeSize > 0) {
                    writeSizeZeroTimes = 0;
                    this.lastWriteTimestamp = HAConnection.this.haService.getDefaultMessageStore().getSystemClock().now();
                } else if (writeSize == 0) {
                    if (++writeSizeZeroTimes >= 3) {
                        break;
                    }
                } else {
                    throw new Exception("ha master write header error < 0");
                }
            }

            if (null == this.selectMappedBufferResult) {
                return !this.byteBufferHeader.hasRemaining();
            }

            writeSizeZeroTimes = 0;

            // Write Body
            if (!this.byteBufferHeader.hasRemaining()) {
                while (this.selectMappedBufferResult.getByteBuffer().hasRemaining()) {
                    int writeSize = this.socketChannel.write(this.selectMappedBufferResult.getByteBuffer());
                    if (writeSize > 0) {
                        writeSizeZeroTimes = 0;
                        this.lastWriteTimestamp = HAConnection.this.haService.getDefaultMessageStore().getSystemClock().now();
                    } else if (writeSize == 0) {
                        if (++writeSizeZeroTimes >= 3) {
                            break;
                        }
                    } else {
                        throw new Exception("ha master write body error < 0");
                    }
                }
            }

            boolean result = !this.byteBufferHeader.hasRemaining() && !this.selectMappedBufferResult.getByteBuffer().hasRemaining();
            //每次写完body后需要清空selectMappedBufferResult，下一次使用
            if (!this.selectMappedBufferResult.getByteBuffer().hasRemaining()) {
                this.selectMappedBufferResult.release();
                this.selectMappedBufferResult = null;
            }

            return result;
        }
ReadSocketService工作过程

org.apache.rocketmq.store.ha.HAConnection.ReadSocketService.processReadEvent()

private boolean processReadEvent() {
            int readSizeZeroTimes = 0;

            //如果byteBufferRead写满了，从头重新写，byteBufferRead初始化1M
            if (!this.byteBufferRead.hasRemaining()) {
                this.byteBufferRead.flip();
                this.processPostion = 0;
            }

            while (this.byteBufferRead.hasRemaining()) {
                try {
                    int readSize = this.socketChannel.read(this.byteBufferRead);
                    if (readSize > 0) {
                        readSizeZeroTimes = 0;
                        this.lastReadTimestamp = HAConnection.this.haService.getDefaultMessageStore().getSystemClock().now();
                        if ((this.byteBufferRead.position() - this.processPostion) >= 8) {
                            //如果读到多等于8个字节，因为HAClient回复消息就是8字节的offset
                            int pos = this.byteBufferRead.position() - (this.byteBufferRead.position() % 8);
                            //取最后一个8字节
                            long readOffset = this.byteBufferRead.getLong(pos - 8);
                            //设置处理位置
                            this.processPostion = pos;

                            //设置上报的slaveAckOffset
                            HAConnection.this.slaveAckOffset = readOffset;

                            //第一次时候设置slaveRequestOffset
                            if (HAConnection.this.slaveRequestOffset < 0) {
                                HAConnection.this.slaveRequestOffset = readOffset;
                                log.info("slave[" + HAConnection.this.clientAddr + "] request offset " + readOffset);
                            }
                            //唤醒同步写的线程
                            HAConnection.this.haService.notifyTransferSome(HAConnection.this.slaveAckOffset);
                        }
                    } else if (readSize == 0) {
                        if (++readSizeZeroTimes >= 3) {
                            break;
                        }
                    } else {
                        log.error("read socket[" + HAConnection.this.clientAddr + "] < 0");
                        return false;
                    }
                } catch (IOException e) {
                    log.error("processReadEvent exception", e);
                    return false;
                }
            }

            return true;
        }
4.HAClient工作过程

org.apache.rocketmq.store.ha.HAService.HAClient.run()

public void run() {
            log.info(this.getServiceName() + " service started");

            while (!this.isStopped()) {
                try {
                    if (this.connectMaster()) {
                    //连接master，取出，初始化currentReportedOffset，lastWriteTimestamp

                        if (this.isTimeToReportOffset()) {
                        //如果间隔大于5秒，上报一次offset
                            boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
                            if (!result) {
                                this.closeMaster();
                            }
                        }

                        this.selector.select(1000);

                        boolean ok = this.processReadEvent();
                        if (!ok) {
                            this.closeMaster();
                        }

                        if (!reportSlaveMaxOffsetPlus()) {
                            continue;
                        }

                        long interval =
                            HAService.this.getDefaultMessageStore().getSystemClock().now()
                                - this.lastWriteTimestamp;
                        if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
                            .getHaHousekeepingInterval()) {
                            log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
                                + "] expired, " + interval);
                            this.closeMaster();
                            log.warn("HAClient, master not response some time, so close connection");
                        }
                    } else {
                        this.waitForRunning(1000 * 5);
                    }
                } catch (Exception e) {
                    log.warn(this.getServiceName() + " service has exception. ", e);
                    this.waitForRunning(1000 * 5);
                }
            }

            log.info(this.getServiceName() + " service end");
        }

private boolean dispatchReadRequest() {
            final int msgHeaderSize = 8 + 4; // phyoffset + size
            int readSocketPos = this.byteBufferRead.position();

            while (true) {
                int diff = this.byteBufferRead.position() - this.dispatchPostion;
                if (diff >= msgHeaderSize) {
                //如果读到的长度大于header长度，表示可以处理
                    long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
                    int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);

                    long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

                    if (slavePhyOffset != 0) {
                        //对比这一次是否是接着上一次的
                        if (slavePhyOffset != masterPhyOffset) {
                            log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
                                + slavePhyOffset + " MASTER: " + masterPhyOffset);
                            return false;
                        }
                    }

                    //如果包含body
                    if (diff >= (msgHeaderSize + bodySize)) {
                        byte[] bodyData = new byte[bodySize];
                        this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
                        this.byteBufferRead.get(bodyData);

                        //写入commitLog文件
                        HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);

                        this.byteBufferRead.position(readSocketPos);
                        //增加dispatchPostion下一次继续使用
                        this.dispatchPostion += msgHeaderSize + bodySize;

                        //上报一次offset
                        if (!reportSlaveMaxOffsetPlus()) {
                            return false;
                        }

                        continue;
                    }
                }

                if (!this.byteBufferRead.hasRemaining()) {
                    //从新初始化
                    this.reallocateByteBuffer();
                }

                break;
            }

            return true;
        }


猜你喜欢

转载自blog.csdn.net/u013038630/article/details/80341286
今日推荐
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
周排行
curl的POST请求，封装方法
8.1.1. Integer Types
Java基础 Day05(个人复习整理)
Python - Django - 中间件 process_exception
小L的试卷
【Shell编程】 （函数）判断用户是否存在
python(css样式)
spring ant path 匹配原则 - 【笔记】
《JavaScript与JScript从入门到精通》(美)James.Jaworski.中译本.扫描版.pdf
Eclipse运行带参数的java程序
每日归档
更多
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022