Reliability and durability of analysis nsq message queue (C) of the message transmission [di] diskqueue

Previous major say how nsq is to ensure the message is successful consumer end consumer , probably put a bit persistent message --mem-queue-sizeis set to 0, all messages will be stored to disk.
Someone always says nsqpersistent problems, the method is to reassure read the original code to do benchmark testing , personal feeling nsqis still very tricky. nsqOwn implements a FIFO queue message file go-diskqueue is to save the message to a local file, it is worth analyzing what he implementation process.

Overall processing logic

go-diskqueueStarts a goroutingread and write data that is the method ioLoop
will be to read and write data according to a parameter set your flowchart follows evernotecid: // D2602A6B-6F53-4199-885D-97DFC21CBA3E / appyinxiangcom / 2479854 / ENResource / p1391

This picture is not particularly accurate ioLoopwith is selectnot if elsewhen there are multiple conditions true, the execution will randomly choose a

nsq Data generated as follows: evernotecid: // D2602A6B-6F53-4199-885D-97DFC21CBA3E / appyinxiangcom / 2479854 / ENResource / p1383

xxxx.diskqueue.meta.dat Metadata stored length of unread messages, the read and stored position data and the read number of
xxxx.diskqueue.编号.dat saved messages file stores each message: the message length + message 4Byte
evernotecid: // D2602A6B-6F53-4199-885D -97DFC21CBA3E / appyinxiangcom / 2479854 / ENResource / p1381

Parameter Description

Some of the major parameters and constraints described use of these parameters will be mentioned later processing logic

// diskQueue implements a filesystem backed FIFO queue
type diskQueue struct {
	// run-time state (also persisted to disk)
	// 读取数据的位置    
	readPos      int64
	// 写入数据的位置
	writePos     int64
	// 读取文件的编号    
	readFileNum  int64
	// 写入文件的编号
	writeFileNum int64
	// 未处理的消息总数    
	depth        int64

	// instantiation time metadata
	// 每个文件的大小限制    
	maxBytesPerFile int64 // currently this cannot change once created
	// 每条消息的最小大小限制    
	minMsgSize      int32
	// 每条消息的最大大小限制    
	maxMsgSize      int32
	// 缓存消息有多少条后进行写入    
	syncEvery       int64         // number of writes per fsync
	// 自动写入消息文件的时间间隔    
	syncTimeout     time.Duration // duration of time per fsync
	exitFlag        int32
	needSync        bool

	// keeps track of the position where we have read
	// (but not yet sent over readChan)
	// 下一条消息的位置    
	nextReadPos     int64
	// 下一条消息的文件编号    
	nextReadFileNum int64

	// 读取的文件
	readFile  *os.File
	// 写入的文件    
	writeFile *os.File
	// 读取的buffer    
	reader    *bufio.Reader
	// 写入的buffer    
	writeBuf  bytes.Buffer

	// exposed via ReadChan()
	// 读取数据的channel    
	readChan chan []byte

	//.....
}
复制代码

data

Metadata

Reading and writing data stored metadata is used mainly as the total number of code fields within xxxxx.diskqueue.meta.data unprocessed message file depthto read the file number of readFileNumthe location of the read data readPos
number written to the file writeFileNumwrite data location of writePos
real data as follows

15
0,22
3,24
复制代码

Save metadata information

func (d *diskQueue) persistMetaData() error {
	// ...
	fileName := d.metaDataFileName()
	tmpFileName := fmt.Sprintf("%s.%d.tmp", fileName, rand.Int())
	// write to tmp file
	f, err = os.OpenFile(tmpFileName, os.O_RDWR|os.O_CREATE, 0600)
	// 元数据信息
	_, err = fmt.Fprintf(f, "%d\n%d,%d\n%d,%d\n",
		atomic.LoadInt64(&d.depth),
		d.readFileNum, d.readPos,
		d.writeFileNum, d.writePos)
	// 保存
	f.Sync()
	f.Close()
	// atomically rename
	return os.Rename(tmpFileName, fileName)
}
复制代码

Get metadata information

func (d *diskQueue) retrieveMetaData() error {
	// ...
	fileName := d.metaDataFileName()
	f, err = os.OpenFile(fileName, os.O_RDONLY, 0600)
	// 读取数据并赋值
	var depth int64
	_, err = fmt.Fscanf(f, "%d\n%d,%d\n%d,%d\n",
		&depth,
		&d.readFileNum, &d.readPos,
		&d.writeFileNum, &d.writePos)
	//...
	atomic.StoreInt64(&d.depth, depth)
	d.nextReadFileNum = d.readFileNum
	d.nextReadPos = d.readPos
	return nil
}
复制代码

Message data

A data write

ioLoopWhen data is written is found, calls the writeOnemethod, the message is saved into the file

		select {
		// ...
		case dataWrite := <-d.writeChan:
			count++
			d.writeResponseChan <- d.writeOne(dataWrite)
		// ...
复制代码

func (d *diskQueue) writeOne(data []byte) error {
	var err error

	if d.writeFile == nil {
		curFileName := d.fileName(d.writeFileNum)
		d.writeFile, err = os.OpenFile(curFileName, os.O_RDWR|os.O_CREATE, 0600)
		// ...
		if d.writePos > 0 {
			_, err = d.writeFile.Seek(d.writePos, 0)
			// ...
		}
	}

	dataLen := int32(len(data))
	// 判断消息的长度是否合法
	if dataLen < d.minMsgSize || dataLen > d.maxMsgSize {
		return fmt.Errorf("invalid message write size (%d) maxMsgSize=%d", dataLen, d.maxMsgSize)
	}
	d.writeBuf.Reset()
	// 写入4字节的消息长度,以大端序保存
	err = binary.Write(&d.writeBuf, binary.BigEndian, dataLen)
	if err != nil {
		return err
	}
	// 写入消息
	_, err = d.writeBuf.Write(data)
	if err != nil {
		return err
	}

	// 写入到文件
	_, err = d.writeFile.Write(d.writeBuf.Bytes())
	// ...
	// 计算写入位置，消息数量加1
	totalBytes := int64(4 + dataLen)
	d.writePos += totalBytes
	atomic.AddInt64(&d.depth, 1)
	// 如果写入位置大于 单个文件的最大限制， 则持久化文件到硬盘
	if d.writePos > d.maxBytesPerFile {
		d.writeFileNum++
		d.writePos = 0

		// sync every time we start writing to a new file
		err = d.sync()
		// ...
	}
	return err
}

复制代码

After writing a message, it will determine whether the size of the current file has been maxBytesPerFileif large, persistent files to the hard disk, and then open a new file numbers again, for writing.

When persistent files to the hard disk

Call the sync()method will be persistent files to the hard disk, and then open a new file numbers again, for writing.
There are several places to call calls this method:

• A number of pieces written to the file reaches syncEverythe time value, which is the largest number of initialization settings. Will callsync()
• syncTimeout Synchronization time interval set at initialization, if the time interval is up, and the number of written documents> 0, the call will besync()
• There is the above-mentioned writeOnemethod, after you write the message, will determine whether the current file size has been maxBytesPerFileif the big, callssync()
• When reading the file, the entire file is read when finished, will delete the file and will needSyncset true, ioLoopwill callsync()
• There is Closetime, callssync()

func (d *diskQueue) sync() error {
	if d.writeFile != nil {
		// 把数据 flash到硬盘，关闭文件并设置为 nil
		err := d.writeFile.Sync()
		if err != nil {
			d.writeFile.Close()
			d.writeFile = nil
			return err
		}
	}
	// 保存元数据信息
	err := d.persistMetaData()
	// ...
	d.needSync = false
	return nil
}
复制代码

Read a piece of data

Holds the read metadata file number readFileNumlocation and reading of data readPos
and diskQueueexposes a method to, by channelreading the data

func (d *diskQueue) ReadChan() chan []byte {
	return d.readChan
}
复制代码

ioLoopWhere, when found in the reading position or writing position is smaller than the file number is less than the read write file number, and the next will read a reading position data is equal to the current position, and then placed in an external global variables dataReadin, and the reading the channelassignment monitor r = d.readChan, when someone reads the external message, perform moveForwardthe operation

func (d *diskQueue) ioLoop() {
	var dataRead []byte
	var err error
	var count int64
	var r chan []byte
	for {
		// ...
		if (d.readFileNum < d.writeFileNum) || (d.readPos < d.writePos) {
			if d.nextReadPos == d.readPos {
				dataRead, err = d.readOne()
				if err != nil {
					d.handleReadError()
					continue
				}
			}
			r = d.readChan
		} else {
			r = nil
		}

		select {
		// ...
		case r <- dataRead:
			count++
			// moveForward sets needSync flag if a file is removed
			d.moveForward()
		// ...
		}
	}

// ...
}

复制代码

readOneA message read from a file, 4 bit size, and then read a specific message. If the reading position is greater than the maximum file size is close. In moveForward Lane will delete operation

func (d *diskQueue) readOne() ([]byte, error) {
	var err error
	var msgSize int32
	// 如果readFile是nil，打开一个新的
	if d.readFile == nil {
		curFileName := d.fileName(d.readFileNum)
		d.readFile, err = os.OpenFile(curFileName, os.O_RDONLY, 0600)
		// ...
		d.reader = bufio.NewReader(d.readFile)
	}
	err = binary.Read(d.reader, binary.BigEndian, &msgSize)
	// ...
	readBuf := make([]byte, msgSize)
	_, err = io.ReadFull(d.reader, readBuf)
	totalBytes := int64(4 + msgSize)
	// ...
	d.nextReadPos = d.readPos + totalBytes
	d.nextReadFileNum = d.readFileNum
	// 如果读取位置大于最大文件限制，则close。在moveForward里会进行删除操作
	if d.nextReadPos > d.maxBytesPerFile {
		if d.readFile != nil {
			d.readFile.Close()
			d.readFile = nil
		}
		d.nextReadFileNum++
		d.nextReadPos = 0
	}
	return readBuf, nil
}

复制代码

moveForwardThe method looks at the numbers read, if the next number and found that the current number is not the same, the old file is deleted.

func (d *diskQueue) moveForward() {
	oldReadFileNum := d.readFileNum
	d.readFileNum = d.nextReadFileNum
	d.readPos = d.nextReadPos
	depth := atomic.AddInt64(&d.depth, -1)

	// see if we need to clean up the old file
	if oldReadFileNum != d.nextReadFileNum {
		// sync every time we start reading from a new file
		d.needSync = true

		fn := d.fileName(oldReadFileNum)
		err := os.Remove(fn)
		// ...
	}
	d.checkTailCorruption(depth)

复制代码