248 lines
11 KiB
Text
248 lines
11 KiB
Text
|
Block replication
|
||
|
|
----------------------------------------
|
||
|
|
Copyright Fujitsu, Corp. 2016
|
||
|
|
Copyright (c) 2016 Intel Corporation
|
||
|
|
Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
|
||
|
|
|
||
|
|
This work is licensed under the terms of the GNU GPL, version 2 or later.
|
||
|
|
See the COPYING file in the top-level directory.
|
||
|
|
|
||
|
|
Block replication is used for continuous checkpoints. It is designed
|
||
|
|
for COLO (COarse-grain LOck-stepping) where the Secondary VM is running.
|
||
|
|
It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario,
|
||
|
|
where the Secondary VM is not running.
|
||
|
|
|
||
|
|
This document gives an overview of block replication's design.
|
||
|
|
|
||
|
|
== Background ==
|
||
|
|
High availability solutions such as micro checkpoint and COLO will do
|
||
|
|
consecutive checkpoints. The VM state of the Primary and Secondary VM is
|
||
|
|
identical right after a VM checkpoint, but becomes different as the VM
|
||
|
|
executes till the next checkpoint. To support disk contents checkpoint,
|
||
|
|
the modified disk contents in the Secondary VM must be buffered, and are
|
||
|
|
only dropped at next checkpoint time. To reduce the network transportation
|
||
|
|
effort during a vmstate checkpoint, the disk modification operations of
|
||
|
|
the Primary disk are asynchronously forwarded to the Secondary node.
|
||
|
|
|
||
|
|
== Workflow ==
|
||
|
|
The following is the image of block replication workflow:
|
||
|
|
|
||
|
|
+----------------------+ +------------------------+
|
||
|
|
|Primary Write Requests| |Secondary Write Requests|
|
||
|
|
+----------------------+ +------------------------+
|
||
|
|
| |
|
||
|
|
| (4)
|
||
|
|
| V
|
||
|
|
| /-------------\
|
||
|
|
| Copy and Forward | |
|
||
|
|
|---------(1)----------+ | Disk Buffer |
|
||
|
|
| | | |
|
||
|
|
| (3) \-------------/
|
||
|
|
| speculative ^
|
||
|
|
| write through (2)
|
||
|
|
| | |
|
||
|
|
V V |
|
||
|
|
+--------------+ +----------------+
|
||
|
|
| Primary Disk | | Secondary Disk |
|
||
|
|
+--------------+ +----------------+
|
||
|
|
|
||
|
|
1) Primary write requests will be copied and forwarded to Secondary
|
||
|
|
QEMU.
|
||
|
|
2) Before Primary write requests are written to Secondary disk, the
|
||
|
|
original sector content will be read from Secondary disk and
|
||
|
|
buffered in the Disk buffer, but it will not overwrite the existing
|
||
|
|
sector content (it could be from either "Secondary Write Requests" or
|
||
|
|
previous COW of "Primary Write Requests") in the Disk buffer.
|
||
|
|
3) Primary write requests will be written to Secondary disk.
|
||
|
|
4) Secondary write requests will be buffered in the Disk buffer and it
|
||
|
|
will overwrite the existing sector content in the buffer.
|
||
|
|
|
||
|
|
== Architecture ==
|
||
|
|
We are going to implement block replication from many basic
|
||
|
|
blocks that are already in QEMU.
|
||
|
|
|
||
|
|
virtio-blk ||
|
||
|
|
^ || .----------
|
||
|
|
| || | Secondary
|
||
|
|
1 Quorum || '----------
|
||
|
|
/ \ || virtio-blk
|
||
|
|
/ \ || ^
|
||
|
|
Primary 2 filter |
|
||
|
|
disk ^ 7 Quorum
|
||
|
|
| /
|
||
|
|
3 NBD -------> 3 NBD /
|
||
|
|
client || server 2 filter
|
||
|
|
|| ^ ^
|
||
|
|
--------. || | |
|
||
|
|
Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4
|
||
|
|
--------' || | backing ^ backing
|
||
|
|
|| | |
|
||
|
|
|| | |
|
||
|
|
|| '-------------------------'
|
||
|
|
|| drive-backup sync=none 6
|
||
|
|
|
||
|
|
1) The disk on the primary is represented by a block device with two
|
||
|
|
children, providing replication between a primary disk and the host that
|
||
|
|
runs the secondary VM. The read pattern (fifo) for quorum can be extended
|
||
|
|
to make the primary always read from the local disk instead of going through
|
||
|
|
NBD.
|
||
|
|
|
||
|
|
2) The new block filter (the name is replication) will control the block
|
||
|
|
replication.
|
||
|
|
|
||
|
|
3) The secondary disk receives writes from the primary VM through QEMU's
|
||
|
|
embedded NBD server (speculative write-through).
|
||
|
|
|
||
|
|
4) The disk on the secondary is represented by a custom block device
|
||
|
|
(called active-disk). It should start as an empty disk, and the format
|
||
|
|
should support bdrv_make_empty() and backing file.
|
||
|
|
|
||
|
|
5) The hidden-disk is created automatically. It buffers the original content
|
||
|
|
that is modified by the primary VM. It should also start as an empty disk,
|
||
|
|
and the driver supports bdrv_make_empty() and backing file.
|
||
|
|
|
||
|
|
6) The drive-backup job (sync=none) is run to allow hidden-disk to buffer
|
||
|
|
any state that would otherwise be lost by the speculative write-through
|
||
|
|
of the NBD server into the secondary disk. So before block replication,
|
||
|
|
the primary disk and secondary disk should contain the same data.
|
||
|
|
|
||
|
|
7) The secondary also has a quorum node, so after secondary failover it
|
||
|
|
can become the new primary and continue replication.
|
||
|
|
|
||
|
|
|
||
|
|
== Failure Handling ==
|
||
|
|
There are 7 internal errors when block replication is running:
|
||
|
|
1. I/O error on primary disk
|
||
|
|
2. Forwarding primary write requests failed
|
||
|
|
3. Backup failed
|
||
|
|
4. I/O error on secondary disk
|
||
|
|
5. I/O error on active disk
|
||
|
|
6. Making active disk or hidden disk empty failed
|
||
|
|
7. Doing failover failed
|
||
|
|
In case 1 and 5, we just report the error to the disk layer. In case 2, 3,
|
||
|
|
4 and 6, we just report block replication's error to FT/HA manager (which
|
||
|
|
decides when to do a new checkpoint, when to do failover).
|
||
|
|
In case 7, if active commit failed, we use replication failover failed state
|
||
|
|
in Secondary's write operation (what decides which target to write).
|
||
|
|
|
||
|
|
== New block driver interface ==
|
||
|
|
We add four block driver interfaces to control block replication:
|
||
|
|
a. replication_start_all()
|
||
|
|
Start block replication, called in migration/checkpoint thread.
|
||
|
|
We must call block_replication_start_all() in secondary QEMU before
|
||
|
|
calling block_replication_start_all() in primary QEMU. The caller
|
||
|
|
must hold the I/O mutex lock if it is in migration/checkpoint
|
||
|
|
thread.
|
||
|
|
b. replication_do_checkpoint_all()
|
||
|
|
This interface is called after all VM state is transferred to
|
||
|
|
Secondary QEMU. The Disk buffer will be dropped in this interface.
|
||
|
|
The caller must hold the I/O mutex lock if it is in migration/checkpoint
|
||
|
|
thread.
|
||
|
|
c. replication_get_error_all()
|
||
|
|
This interface is called to check if error happened in replication.
|
||
|
|
The caller must hold the I/O mutex lock if it is in migration/checkpoint
|
||
|
|
thread.
|
||
|
|
d. replication_stop_all()
|
||
|
|
It is called on failover. We will flush the Disk buffer into
|
||
|
|
Secondary Disk and stop block replication. The vm should be stopped
|
||
|
|
before calling it if you use this API to shutdown the guest, or other
|
||
|
|
things except failover. The caller must hold the I/O mutex lock if it is
|
||
|
|
in migration/checkpoint thread.
|
||
|
|
|
||
|
|
== Usage ==
|
||
|
|
Primary:
|
||
|
|
-drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\
|
||
|
|
children.0.file.filename=1.raw,\
|
||
|
|
children.0.driver=raw
|
||
|
|
|
||
|
|
Run qmp command in primary qemu:
|
||
|
|
{ 'execute': 'human-monitor-command',
|
||
|
|
'arguments': {
|
||
|
|
'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1'
|
||
|
|
}
|
||
|
|
}
|
||
|
|
{ 'execute': 'x-blockdev-change',
|
||
|
|
'arguments': {
|
||
|
|
'parent': 'colo1',
|
||
|
|
'node': 'nbd_client1'
|
||
|
|
}
|
||
|
|
}
|
||
|
|
Note:
|
||
|
|
1. There should be only one NBD Client for each primary disk.
|
||
|
|
2. host is the secondary physical machine's hostname or IP
|
||
|
|
3. Each disk must have its own export name.
|
||
|
|
4. It is all a single argument to -drive and you should ignore the
|
||
|
|
leading whitespace.
|
||
|
|
5. The qmp command line must be run after running qmp command line in
|
||
|
|
secondary qemu.
|
||
|
|
6. After primary failover we need remove children.1 (replication driver).
|
||
|
|
|
||
|
|
Secondary:
|
||
|
|
-drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
|
||
|
|
-drive if=none,id=childs1,driver=replication,mode=secondary,top-id=childs1
|
||
|
|
file.file.filename=active_disk.qcow2,\
|
||
|
|
file.driver=qcow2,\
|
||
|
|
file.backing.file.filename=hidden_disk.qcow2,\
|
||
|
|
file.backing.driver=qcow2,\
|
||
|
|
file.backing.backing=colo1
|
||
|
|
-drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\
|
||
|
|
vote-threshold=1,children.0=childs1
|
||
|
|
|
||
|
|
Then run qmp command in secondary qemu:
|
||
|
|
{ 'execute': 'nbd-server-start',
|
||
|
|
'arguments': {
|
||
|
|
'addr': {
|
||
|
|
'type': 'inet',
|
||
|
|
'data': {
|
||
|
|
'host': 'xxx',
|
||
|
|
'port': 'xxx'
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
{ 'execute': 'nbd-server-add',
|
||
|
|
'arguments': {
|
||
|
|
'device': 'colo1',
|
||
|
|
'writable': true
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
Note:
|
||
|
|
1. The export name in secondary QEMU command line is the secondary
|
||
|
|
disk's id.
|
||
|
|
2. The export name for the same disk must be the same
|
||
|
|
3. The qmp command nbd-server-start and nbd-server-add must be run
|
||
|
|
before running the qmp command migrate on primary QEMU
|
||
|
|
4. Active disk, hidden disk and nbd target's length should be the
|
||
|
|
same.
|
||
|
|
5. It is better to put active disk and hidden disk in ramdisk.
|
||
|
|
6. It is all a single argument to -drive, and you should ignore
|
||
|
|
the leading whitespace.
|
||
|
|
|
||
|
|
After Failover:
|
||
|
|
Primary:
|
||
|
|
The secondary host is down, so we should run the following qmp command
|
||
|
|
to remove the nbd child from the quorum:
|
||
|
|
{ 'execute': 'x-blockdev-change',
|
||
|
|
'arguments': {
|
||
|
|
'parent': 'colo1',
|
||
|
|
'child': 'children.1'
|
||
|
|
}
|
||
|
|
}
|
||
|
|
{ 'execute': 'human-monitor-command',
|
||
|
|
'arguments': {
|
||
|
|
'command-line': 'drive_del xxxx'
|
||
|
|
}
|
||
|
|
}
|
||
|
|
Note: there is no qmp command to remove the blockdev now
|
||
|
|
|
||
|
|
Secondary:
|
||
|
|
The primary host is down, so we should do the following thing:
|
||
|
|
{ 'execute': 'nbd-server-stop' }
|
||
|
|
|
||
|
|
Promote Secondary to Primary:
|
||
|
|
see COLO-FT.txt
|
||
|
|
|
||
|
|
TODO:
|
||
|
|
1. Shared disk
|