@gea, please help.
I'm getting replication errors again between my Omnios servers.
Both servers are running OmniosCE r151030cm and NappIT 21.01b4.
The machines are called ServerA and ServerB.
I had been having problems with replication jobs from ServerA to ServerB. Please see this forum post:
I thought I had solved the problem by limiting the ZFS send stream to 30Mb/s.
However the failed replication jobs have returned.
So I have upgraded the storage pool on ServerB from onboard SATAII to a Broadcom 9300-8i and SATAIII disks.
But the upgraded pool has not stopped the failed replications.
These errors seem to have restarted after I upgraded both machines to r151030cm and NappIT 21.01b4 but I can't be sure.
I've attached the NappIT logs for the most recent failed replication job, please can you help me fix what ever is going wrong.
ServerA is the source.
ServerB is the destination
If I start the failed replication job manually it will complete OK.
ServerA storage pool has 2.04TB available and 604GB in use, there are 337 snapshots.
ServerB storage pool has 2.14TB available and 501GB in use, there are 217 snapshots.
Please note, the source snap tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 exists on the source machine.
ServerA remote log
zfs remote log for 1451725974 newest entry on top
ServerB Job Log
Log: Job 1451725974 (newest first)
ServerB Last run Log
og: Last run of Job 1451725974 (newest first)
many thanks
bob
I'm getting replication errors again between my Omnios servers.
Both servers are running OmniosCE r151030cm and NappIT 21.01b4.
The machines are called ServerA and ServerB.
I had been having problems with replication jobs from ServerA to ServerB. Please see this forum post:
I thought I had solved the problem by limiting the ZFS send stream to 30Mb/s.
However the failed replication jobs have returned.
So I have upgraded the storage pool on ServerB from onboard SATAII to a Broadcom 9300-8i and SATAIII disks.
But the upgraded pool has not stopped the failed replications.
These errors seem to have restarted after I upgraded both machines to r151030cm and NappIT 21.01b4 but I can't be sure.
I've attached the NappIT logs for the most recent failed replication job, please can you help me fix what ever is going wrong.
ServerA is the source.
ServerB is the destination
If I start the failed replication job manually it will complete OK.
ServerA storage pool has 2.04TB available and 604GB in use, there are 337 snapshots.
ServerB storage pool has 2.14TB available and 501GB in use, there are 217 snapshots.
Please note, the source snap tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 exists on the source machine.
ServerA remote log
zfs remote log for 1451725974 newest entry on top
## 20:20 04 grouplib 1878: call job-repli-send.pl for id 1451725974 on interface /usr/bin/nc -w 40 192.168.1.9 56499 |
## 20:20 03 grouplib 1810: zfs snapshot tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 |
## 20:20 03 grouplib 1763: remote call zfs send 1451725974 initiated _________________________________________________________ |
## 20:20 01 grouplib 2018: remote call destroy snap : zfs destroy tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11214 |
ServerB Job Log
Log: Job 1451725974 (newest first)
error | time: 2021.03.01.20.20.43 line: 2258 | my_log end 1551: 887 next destination snap tank/backup/beth@1451725974_repli_zfs_ServerB_nr_11217 from ServerA was not created error check network, systemlog, poolstate, capacity, timeouts on recursive transfers optionally delete newest target snap and retry with the former error check network, systemlog, poolstate, capacity, timeouts on recursive transfers optionally delete newest target snap and retry with the former |
end receive | time: 2021.03.01.20.20.37 line: 1014 | zfs receive 1451725974 running 35 s |
receiver message | ||
time: 2021.03.01.20.20.37 line: 1002 | ||
receive finished with warning 'failed to read from stream', can be a message only: check if target snap is created | time: 2021.03.01.20.20.37 line: 992 | |
receiver message | time: 2021.03.01.20.20.37 line: 985 | cannot receive: failed to read from stream |
error, monitor info | time: 2021.03.01.20.20.37 line: 333 | replication terminated: local receive=1, remote send=0 - check zpool status |
incremental send | time: 2021.03.01.20.20.03 line: 1135 | ServerA: zfs send -i tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11216 tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 | /var/web-gui/data/tools/nc/nc -b 131072 -w 40 192.168.1.9 56499 |
start receiver | time: 2021.03.01.20.20.02 line: 960 | /var/web-gui/data/tools/nc/nc -b 131072 -d -l -p 56499 | /usr/sbin/zfs receive -Fv tank/backup/beth 2>&1 |
zfs destroy | time: 2021.03.01.20.20.02 line: 904 | tank/backup/beth@1451725974_repli_zfs-kh2021022816_ServerB_nr_11210 |
request remote snap destroy | time: 2021.03.01.20.20.01 line: 855 | tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11214 |
next replication started | time: 2021.03.01.20.20.00 line: 136 |
ServerB Last run Log
og: Last run of Job 1451725974 (newest first)
2021.03.01.20.20.43 | &my_log_last line 2259 <- &my_log line 2017 <- &my_end_all line 1986 <- &my_fast_end line 1065 <- &my_remote_delta_replication line 442 | error; |
2021.03.01.20.20.41 | &my_log_last line 1063 <- &my_remote_delta_replication line 442 <- &my_run line 172 | 887 next destination snap tank/backup/beth@1451725974_repli_zfs_ServerB_nr_11217 from ServerA was not created |
2021.03.01.20.20.39 | &my_log_last line 2464 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | end my postreplication |
2021.03.01.20.20.39 | &my_log_last line 2453 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | my postreplication, zfs set readonly=on tank/backup/beth |
2021.03.01.20.20.39 | &my_log_last line 2393 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | zfs set nfs/smb shares=off tank/backup/beth |
2021.03.01.20.20.39 | &my_log_last line 2373 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | zfs set readonly=off tank/backup/beth |
2021.03.01.20.20.39 | &my_log_last line 2345 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | 1522 update id.par: last=01.mar_20_20 |
2021.03.01.20.20.39 | &my_log_last line 2325 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | next_src_snap from host ServerA tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 0 - 1.42G |
2021.03.01.20.20.38 | &my_log_last line 2313 <- &my_postreplication line 1022 <- &my_remote_delta_replication line 442 <- &my_run line 172 | begin my_postreplication |
2021.03.01.20.20.38 | &my_log_last line 2090 <- &my_end_proc line 335 <- &my_monitor line 1183 <- &my_remote_delta_replication line 442 <- &my_run line 172 | kill proc 19410 ? S 00:00 /usr/sbin/zfs receive -Fv tank/backup/beth (19410) |
2021.03.01.20.20.37 | &my_log_last line 1015 <- &my_remote_delta_replication line 442 <- &my_run line 172 | end receive 35 s |
2021.03.01.20.20.37 | &my_log_last line 984 <- &my_remote_delta_replication line 442 <- &my_run line 172 | error;, receiver message cannot receive: failed to read from stream |
2021.03.01.20.20.37 | &my_log_last line 2064 <- &my_end_proc line 335 <- &my_monitor line 1183 <- &my_remote_delta_replication line 442 <- &my_run line 172 | kill proc 19409 ? S 00:00 /var/web-gui/data/tools/nc/nc -b 131072 -d -l -p 56499 (19409) |
2021.03.01.20.20.37 | &my_log_last line 2064 <- &my_end_proc line 335 <- &my_monitor line 1183 <- &my_remote_delta_replication line 442 <- &my_run line 172 | kill proc 19408 ? S 00:00 sh -c /var/web-gui/data/tools/nc/nc -b 131072 -d -l -p 56499 | /usr/sbin/zfs re (19408) |
2021.03.01.20.20.37 | &my_log_last line 334 <- &my_monitor line 1183 <- &my_remote_delta_replication line 442 <- &my_run line 172 | error;, monitor info, replication terminated: local receive=1, remote send=0 - check zpool status |
2021.03.01.20.20.07 | &my_log_last line 313 <- &my_monitor line 1183 <- &my_remote_delta_replication line 442 <- &my_run line 172 | Monitor: Remote proc: remote nc and zfs send not running or finished |
2021.03.01.20.20.04 | &my_log_last line 1154 <- &my_remote_delta_replication line 442 <- &my_run line 172 | source snap1,2: size=0,0 |
2021.03.01.20.20.03 | &my_log_last line 1138 <- &my_remote_delta_replication line 442 <- &my_run line 172 | start sender id=1451725974 src_interface=/usr/bin/nc -w 40 192.168.1.9 56499 src_interface2=/var/web-gui/data/tools/nc/nc -b 131072 -w 40 192.168.1.9 56499 snap_name1=tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11216 snap_name2=tank/storage/beth@1451725974_repli_zfs_ServerB_nr_11217 send_inc=-i snap_rec= send_rec= send_dedup= send_i=-i transfer_zero= send_option= pv_limit=-L 30m |
2021.03.01.20.20.02 | &my_log_last line 963 <- &my_remote_delta_replication line 442 <- &my_run line 172 | start receiver /var/web-gui/data/tools/nc/nc -b 131072 -d -l -p 56499 | /usr/sbin/zfs receive -Fv tank/backup/beth 2>&1 |
2021.03.01.20.20.00 | &my_log_last line 776 <- &my_remote_delta_replication line 442 <- &my_run line 172 | start remote incremental replication |
2021.03.01.20.20.00 | main line 76 | start job-replicate with parameter: id=1451725974, action=run, par='run_1451725974 |
many thanks
bob