ZFS after disk failure: most of data seems to be intact, but how to extract?

Question

I have a zpool after disk failure. Attempt to import it (even zpool import -F -T 12855264 -R /mnt -f rpool ) yields errors

Dec  4 12:56:54 freebsd kernel: Solaris: WARNING: can't open objset 1035, error 5
Dec  4 12:56:54 freebsd ZFS[10155]: pool I/O failure, zpool=rpool error=97
Dec  4 12:56:54 freebsd ZFS[10159]: pool I/O failure, zpool=rpool error=97
Dec  4 12:56:54 freebsd ZFS[10163]: failed to load zpool rpool

zdb shows some data

Dataset mos [META], ID 0, cr_txg 4, 1.02G, 304 objects
Dataset rpool/samba [ZPL], ID 100, cr_txg 135431, 276K, 9 objects
Dataset rpool/ROOT/pve-1 [ZPL], ID 515, cr_txg 10, 4.85G, 74685 objects
Dataset rpool/ROOT [ZPL], ID 259, cr_txg 8, 96K, 7 objects
Dataset rpool/secure/vm-103-state-good [ZVOL], ID 661, cr_txg 2925304, 2.94G, 2 objects
Dataset rpool/secure/vm-101-disk-1 [ZVOL], ID 1441, cr_txg 1445964, 188G, 2 objects
Dataset rpool/secure/vm-104-disk-0@good [ZVOL], ID 1285, cr_txg 2925290, 83.1G, 2 objects
Dataset rpool/secure/vm-104-disk-0 [ZVOL], ID 413, cr_txg 222940, 84.7G, 2 objects
Dataset rpool/secure/vm-104-state-good [ZVOL], ID 173, cr_txg 2925283, 438M, 2 objects
Dataset rpool/secure/subvol-105-disk-0@good [ZPL], ID 521, cr_txg 2925300, 1.36G, 36077 objects
failed to hold dataset 'rpool/secure/subvol-105-disk-0': Input/output error
Dataset rpool/secure/vm-101-disk-2 [ZVOL], ID 942, cr_txg 1445966, 114G, 2 objects
Dataset rpool/secure/vm-103-disk-0@good [ZVOL], ID 1027, cr_txg 2925319, 18.1G, 2 objects
Dataset rpool/secure/vm-103-disk-0 [ZVOL], ID 431, cr_txg 916036, 59.6G, 2 objects
Dataset rpool/secure/vm-101-disk-0 [ZVOL], ID 448, cr_txg 1445962, 160K, 2 objects
Dataset rpool/secure/vm-107-disk-0 [ZVOL], ID 285, cr_txg 219462, 16.9G, 2 objects
Dataset rpool/secure/vm-101-disk-3 [ZVOL], ID 654, cr_txg 2329396, 83.1G, 2 objects
Dataset rpool/secure [ZPL], ID 145, cr_txg 90, 200K, 7 objects
Dataset rpool/data/vm-100-state-good [ZVOL], ID 394, cr_txg 1864406, 114M, 2 objects
Dataset rpool/data/vm-106-disk-0@good [ZVOL], ID 1287, cr_txg 3889962, 3.97G, 2 objects
Dataset rpool/data/vm-106-disk-0 [ZVOL], ID 781, cr_txg 1890481, 27.5G, 2 objects
Dataset rpool/data/vm-100-disk-0@good [ZVOL], ID 518, cr_txg 1864412, 88.5M, 2 objects
Dataset rpool/data/vm-100-disk-0 [ZVOL], ID 266, cr_txg 36631, 88.6M, 2 objects
Dataset rpool/data/vm-106-state-good [ZVOL], ID 668, cr_txg 3889956, 358M, 2 objects
Dataset rpool/data [ZPL], ID 387, cr_txg 9, 96K, 6 objects
Dataset rpool [ZPL], ID 54, cr_txg 1, 10.3G, 26 objects
MOS object 403 (bpobj) leaked
MOS object 522 (DSL deadlist map) leaked
MOS object 1032 (zap) leaked
MOS object 1033 (DSL props) leaked
MOS object 1034 (DSL directory child map) leaked
MOS object 1035 (zap) leaked
MOS object 1036 (DSL dataset snap map) leaked
MOS object 1038 (zap) leaked
Verified large_blocks feature refcount of 0 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
userobj_accounting feature refcount mismatch: 7 consumers != 8 refcount
encryption feature refcount mismatch: 14 consumers != 15 refcount
project_quota feature refcount mismatch: 7 consumers != 8 refcount

and it seems to me that only one dataset is damaged.

So I have two questions:

1. dd reads all the disk fine, what causes the I/O failure?
2. How do I extract intact data? Some datasets are encrypted, so just read blocks is not enough

Answer 1

sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0
zpool import -F -R /mnt -N -f -o readonly=on *yourpool*

And what can be saved will be saved.

(the solution is for FreeBSD. Linux/Solaris uses other ways to set libzfs options, mdb for Solaris, different systl-s for Linux)