OpenZFS on OS X/zfs 6fd8e79tests/zfs-tests/runfiles osx.run.in, tests/zfs-tests/tests/functional/cli_root/zpool_create zpool_create_024_pos.ksh

Minor zfstest fix

OpenZFS on OS X/zfs 668cfc6tests/zfs-tests/runfiles osx.run.in

typo

OpenZFS on OS X/zfs c0f7341tests/zfs-tests/tests/functional/snapshot rollback_001_pos.ksh rollback_002_pos.ksh

zfstester fixes: snapshots

OpenZFS on OS X/zfs d1c84abtests/zfs-tests/tests/functional/cli_root/zpool_import/zpool_import zpool_import.kshlib, tests/zfs-tests/tests/functional/fault auto_offline_001_pos.ksh

Large zfstester fixes

truncate
write_files
sha256sum

OpenZFS on OS X/zfs a89cf05include/sys zfs_ioctl.h

DMU_STREAM_SUPPORTED missing DMU_BACKUP_FEATURE_HOLDS

The zfs send hold feature missed the featureflag addition.

OpenZFS on OS X/zfs 3d95fcbcmd/zfs zfs_main.c

zfs change_key calling parseprop()

A final call to parseprop() in change_key missed the change to boolean for
the return type, causing all zfs change_key commands to fail silently.
DeltaFile
+1-1cmd/zfs/zfs_main.c
+1-11 files

OpenZFS on OS X/zfs d93a701tests/test-runner/stf/contrib/include logapi.shlib, tests/zfs-tests/include libtest.shlib

zfstester fixes

OpenZFS on OS X/zfs 3cd72delib/libspl/crypto/sha2 sha2.c

No #pragma weak in clang

Any linker of libzfs can fail with:
  "_SHA256Final", referenced from:
      _cksummer in libzfs.a(libzfs_sendrecv.o)

Due to lazy-linking in OSX, at any time, in particular certain
zfs recv cases.

For APPLE we make wrapper functions.

OpenZFS on OS X/zfs e6c146dlib/libzfs libzfs_sendrecv.c

Use zfs_ioctl instead of direct ioctl()

The correct error code is handled by zfs_ioctl(), and by using
direct ioctl() many rsend/ tests were failing.

OpenZFS on OS X/zfs dc9fb00include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c dsl_pool.c

Use taskq_dispatch_ent with zp_attach_vnode

Since there can only ever be one taskq per znode (only one attach required)
it makes sense to embed the taskqent into znode.

OpenZFS on OS X/zfs f784ab4module/zfs zvol.c zfs_ioctl.c

Correct zvol_get_stats()

OpenZFS on OS X/zfs f0a9837cmd/zpool zpool_main.c, module/zfs zfeature.c dsl_crypt.c

Better user experience for errata 4

This patch attempts to address some user concerns that have arisen
since errata 4 was introduced.

* The errata warning has been made less scary for users without
  any encrypted datasets.

* The errata warning now clears itself without a pool reimport if
  the bookmark_v2 feature is enabled and no encrypted datasets
  exist.

* It is no longer possible to create new encrypted datasets without
  enabling the bookmark_v2 feature, thus helping to ensure that the
  errata is resolved.

Signed-off-by: Tom Caputi <tcaputi at datto.com>

OpenZFS on OS X/zfs df6e22ecmd/zfs_util zfs_util.c

zfs.util needs to handle devnode pseudo disks

As diskarbitrationd will probe the psuedo disks used for devnode
mounts, zfs.util, called by diskarb, now queries ioreg to find the
"ZFS Dataset" entry to return name and FSUR_RECOGNIZED.

This handles the occasional unmounting error:

"disk5s1 was already unmounted"

related to issue #681
DeltaFile
+312-230cmd/zfs_util/zfs_util.c
+312-2301 files

OpenZFS on OS X/zfs c7d1a1dtests/zfs-tests/cmd/mkfile mkfile.c, tests/zfs-tests/cmd/mkfiles mkfiles.c Makefile.am

Correct zpool initialize zfstesters

OpenZFS on OS X/zfs dd816e9tests/zfs-tests/cmd/libzfs_input_check libzfs_input_check.c, tests/zfs-tests/cmd/nvlist_to_lua Makefile.am

zfstest fixes continued.

OpenZFS on OS X/zfs 9136b90tests/zfs-tests/cmd/nvlist_to_lua Makefile.am, tests/zfs-tests/include libtest.shlib

zfstest fixes continued.

OpenZFS on OS X/zfs 4a37423tests/zfs-tests/runfiles osx.run.in

Remove SCSI and O_TMPFILE tests

Not supported on OSX

OpenZFS on OS X/zfs 1c4200etests/zfs-tests/tests/functional/cli_root/zpool_history cleanup.ksh

test zpool_history does not cleanup properly

OpenZFS on OS X/zfs 2ac5307include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Add zget ASYNC style

The use of zfs_zget_ext( ..., ZGET_WITHOUT_VNODE) has been problematic
as it would leave znodes with NULL vnodes, creating a second cleanup path,
and race waiting on vnode to be attached.

Attempting to follow upstreams solution, which uses VN_RELE_ASYNC() to
avoid the same deadlocks (ZOL deadlock on vput - osx deadlocks on vnode_create)
we now attach vnodes in a separate taskq, so that we will never expect to
have a znode without vnode, for any longer than a short period.

zfs_zget() when vp is NULL will call zfs_znode_asyncwait() to ensure
it is attached if it can. There is a small window where zp/vp is being
torn down, in that case, it does a static wait, and retries.

VN_RELE is also async, just like upstream, and uses condvar to make
sure the attach taskq as completed.

OpenZFS on OS X/zfs afebd42tests/zfs-tests/runfiles osx.run.in, tests/zfs-tests/tests/functional/cli_root/zpool_import import_rewind_config_changed.ksh

The great zfs-test cleanup

We have many tests duplicated in the osx.run.in file, clean up and
synced tests with ZOL.

Change configure.ac to use clang, and compile more of the .C test files.

OpenZFS on OS X/zfs 0707423lib/libzfs libzfs_sendrecv.c, tests/runfiles linux.run

Fix hierarchy misspellings

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reported-by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Richard Laager <rlaager at wiktel.com>
Closes 8563
Closes 8622

OpenZFS on OS X/zfs 9a5a60emodule/zfs dmu_recv.c, tests/runfiles linux.run

Fix issues with truncated files in raw sends

When receiving a raw send stream only reallocated objects
whose contents were not freed by the standard indicators
should call dmu_free_long_range().

Furthermore, if calling dmu_free_long_range() is required
then the objects current block size must be used and not
the new block size.

Two additional test cases were added to provided realistic
test coverage for processing reallocated objects which are
part of a raw receive.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes 8528
Closes 8607

OpenZFS on OS X/zfs 6db6328module/zfs dmu_recv.c, tests/runfiles linux.run

Fix issue in receive_object() during reallocation

When receiving an object to a previously allocated interior slot
the new object should be "allocated" by setting DMU_NEW_OBJECT,
not "reallocated" with dnode_reallocate().  For resilience verify
the slot is free as required in case the stream is malformed.

Add a test case to generate more realistic incremental send streams
that force reallocation to occur during the receive.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8067
Closes 8614

OpenZFS on OS X/zfs 7a5daa6module/zfs dnode.c dsl_dir.c

Fix TXG_MASK cstyle

Fix style issue for 'tx->tx_txg&TXG_MASK'.  There should be white
space around the '&' character.  Split the dnode_reallocate() ASSERT
to make it more readable to clearly separate the checks.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8606

OpenZFS on OS X/zfs 67e63e1module/zfs zap_micro.c

Silence mze warning in zap_micro.c

OpenZFS on OS X/zfs 1aca96dmodule/zfs zfs_vnops_osx.c

More sa_hdl == NULL checks

OpenZFS on OS X/zfs 5c114ecmodule/zfs zfs_vnops_osx.c

Avoid size==0 kmem_alloc in setxattr

OpenZFS on OS X/zfs 15374admodule/zfs dmu_objset.c dsl_dir.c, tests/runfiles linux.run

ZVOLs should not be allowed to have children

zfs create, receive and rename can bypass this hierarchy rule. Update
both userland and kernel module to prevent this issue and use pyzfs
unit tests to exercise the ioctls directly.

Note: this commit slightly changes zfs_ioc_create() ABI. This allow to
differentiate a generic error (EINVAL) from the specific case where we
tried to create a dataset below a ZVOL (ZFS_ERR_WRONG_PARENT).

Reviewed-by: Paul Dagnelie <pcd at delphix.com>
Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: loli10K <ezomori.nozomu at gmail.com>

OpenZFS on OS X/zfs 800fb46cmd/ztest ztest.c, include/sys dmu.h

Fix error handling incallers of dbuf_hold_level()

Currently, the functions dbuf_prefetch_indirect_done() and
dmu_assign_arcbuf_by_dnode() assume that dbuf_hold_level() cannot
fail. In the event of an error the former will cause a NULL pointer
dereference and the later will trigger a VERIFY. This patch adds
error handling to these functions and their callers where necessary.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8291 

OpenZFS on OS X/zfs 8cd38e1module/zfs zfs_vnops_osx_lib.c

zfs_acl memory leak

zfs_obtain_xattr() would fail to call zfs_acl_ids_free() in certain error
conditions.

OpenZFS on OS X/zfs d45fe3binclude/sys dmu.h zap.h, module/zfs dmu.c dmu_object.c

Provide more flexible object allocation interface

Object allocation performance can be improved for complex operations
by providing an interface which returns the newly allocated dnode.
This allows the caller to immediately use the dnode without incurring
the expense of looking up the dnode by object number.

The functions dmu_object_alloc_hold(), zap_create_hold(), and
dmu_bonus_hold_by_dnode() were added for this purpose.

The zap_create_* functions have been updated to take advantage of
this new functionality.  The dmu_bonus_hold_impl() function should
really have never been included in sys/dmu.h and was removed.
It's sole caller was converted to use dmu_bonus_hold_by_dnode().

The new symbols have been exported for use by Lustre.

Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed by: Matt Ahrens <mahrens at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8015

OpenZFS on OS X/zfs 00d2a91module/zfs dmu_recv.c

Make dmu_recv.c identical to zol
DeltaFile
+216-114module/zfs/dmu_recv.c
+216-1141 files

OpenZFS on OS X/zfs 24b5dd3module/zfs zio_crypt.c

Always call rw_init in zio_crypt_key_unwrap

The error path in zio_crypt_key_unwrap would call zio_crypt_key_destroy which
calls rw_destroy(&key->zk_salt_lock); which has not yet been initialized.

We move the rw_init() call to the start of zio_crypt_key_unwrap instead.

Reviewed-by: Tom Caputi <tcaputi at datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Jorgen Lundman <lundman at lundman.net>
Closes 8604
Closes 8605

OpenZFS on OS X/zfs 9e2d0demodule/zfs dmu_recv.c

Fix 0 byte memory leak in zfs receive

Currently, when a DRR_OBJECT record is read into memory in
receive_read_record(), memory is allocated for the bonus buffer.
However, if the object doesn't have a bonus buffer the code will
still "allocate" the zero bytes, but the memory will not be passed
to the processing thread for cleanup later. This causes the spl
kmem tracking code to report a leak. This patch simply changes the
code so that it only allocates this memory if it has a non-zero
length.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8266 

OpenZFS on OS X/zfs e8ea374cmd/zfs zfs_main.c

Fix 'zfs list -t snapshot' depth

Commit df583073 introduced the ability to list the snapshots for a
specified dataset.  This change inadvertently resulted in only the top-
level snapshots being listed when no dataset was specified.  Fix this
issue by adding an additional check to determine if a dataset was
provided to avoid incorrectly restricting the depth.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Reviewed-by: Alek Pinchuk <apinchuk at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8591
Closes 8594
DeltaFile
+20-0cmd/zfs/zfs_main.c
+20-01 files

OpenZFS on OS X/zfs 0d31227module/zfs zfs_vnops_osx.c

Take iocount hold in vnop_pageout

vnop_pageout can be called with a vp with iocount==0 && usecount==0, so
the vp/zp can be relcaimed during the function. We grab an iocount hold
to ensure this does not happen.

OpenZFS on OS X/zfs 6260a77lib/libzpool kernel.c

Don't assume pthread_t is uint_t for portability

POSIX doesn't define pthread_t as uint_t. It could be a pointer.
This code causes below compile error on a platform using pointer
for pthread_t.

--
kernel.c:815:25: error: cast from pointer to integer of different size 
[-Werror=pointer-to-int-cast]
    (void) printf("%u ", (uint_t)pthread_self());

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Igor Kozhukhov <igor at dilos.org>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro at gmail.com>
Closes 8558

OpenZFS on OS X/zfs 0237161

Fix buffer length in strlcpy()

The length used for the strlcpy() used the size of zv_value
when it should have used the size of zc_name.  Correct this
typo.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Jorgen Lundman <lundman at lundman.net>
Reviewed-by: Igor Kozhukhov <igor at dilos.org>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8595
Closes 8596
DeltaFile
+0-00 files

OpenZFS on OS X/zfs 06f8582cmd/zpool zpool_main.c, man/man8 zpool.8

Add 'zpool status -i' option

Only display the full details of the vdev initialization state
in 'zpool status' output when requested with the -i option.
By default display '(initializing)' after vdevs when they are
being actively initialized.  This is consistent with the
established precident of appending '(resilvering), etc' and
fits within the default 80 column terminal width making it
easy to read.

Additionally, updated the 'zpool initialize' documentation to
make it clear the options are mutually exclusive, but allow
duplicate options like all other zfs/zpool commands.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Tim Chase <tim at chase2k.com>
Reviewed-by: George Wilson <george.wilson at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8230

OpenZFS on OS X/zfs 6667400cmd/zfs zfs_main.c, lib/libzfs libzfs_sendrecv.c libzfs_dataset.c

The great ioctl changeover.

ZOL went to SEND_NEW and RECV_NEW a few years back, and we have slowly
drifted apart since. This commit changes us to follow.

The commit changes how

* zfs send
* zfs recv
* zfs rename
* zfs destroy
* zvol (suspend / recv)
* zfs rollback
* zfs crypto (prompt input)

and these commands, and their flags need to be tested.

New flags and abilities contained here-in;

* zfs send --holds
* zfs send -o / -x
* zfs send --backup
* zfs rollback max youngest

This commit also contains the ZOL commits:

    [4 lines not shown]

OpenZFS on OS X/zfs 9255184lib/libzfs libzfs_dataset.c

Add missing zfs_get_recvd_props

OpenZFS on OS X/zfs 93b5f10lib/libzfs_core libzfs_core.c

Add missing lzc_rename lzc_destroy

Lazy linking hides that those two calls were not implemented.

OpenZFS on OS X/zfs 632b10cmodule/zfs zap_micro.c

Silence mze warning in zap_micro.c

OpenZFS on OS X/zfs 26bce3einclude/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Add zget ASYNC style

The use of zfs_zget_ext( ..., ZGET_WITHOUT_VNODE) has been problematic
as it would leave znodes with NULL vnodes, creating a second cleanup path,
and race waiting on vnode to be attached.

Attempting to follow upstreams solution, which uses VN_RELE_ASYNC() to
avoid the same deadlocks (ZOL deadlock on vput - osx deadlocks on vnode_create)
we now attach vnodes in a separate taskq, so that we will never expect to
have a znode without vnode, for any longer than a short period.

zfs_zget() when vp is NULL will call zfs_znode_asyncwait() to ensure
it is attached if it can. There is a small window where zp/vp is being
torn down, in that case, it does a static wait, and retries.

VN_RELE is also async, just like upstream, and uses condvar to make
sure the attach taskq as completed.

OpenZFS on OS X/zfs 4e7937clib/libzfs libzfs_sendrecv.c, lib/libzfs_core libzfs_core.c

Merge branch 'async_zget3'

OpenZFS on OS X/zfs 8991473include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Add zget ASYNC style

The use of zfs_zget_ext( ..., ZGET_WITHOUT_VNODE) has been problematic
as it would leave znodes with NULL vnodes, creating a second cleanup path,
and race waiting on vnode to be attached.

Attempting to follow upstreams solution, which uses VN_RELE_ASYNC() to
avoid the same deadlocks (ZOL deadlock on vput - osx deadlocks on vnode_create)
we now attach vnodes in a separate taskq, so that we will never expect to
have a znode without vnode, for any longer than a short period.

zfs_zget() when vp is NULL will call zfs_znode_asyncwait() to ensure
it is attached if it can. There is a small window where zp/vp is being
torn down, in that case, it does a static wait, and retries.

VN_RELE is also async, just like upstream, and uses condvar to make
sure the attach taskq as completed.

OpenZFS on OS X/zfs cfd28a9module/zfs dmu_recv.c, tests/runfiles linux.run

Fix issues with truncated files in raw sends

When receiving a raw send stream only reallocated objects
whose contents were not freed by the standard indicators
should call dmu_free_long_range().

Furthermore, if calling dmu_free_long_range() is required
then the objects current block size must be used and not
the new block size.

Two additional test cases were added to provided realistic
test coverage for processing reallocated objects which are
part of a raw receive.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes 8528
Closes 8607

OpenZFS on OS X/zfs abd1d72lib/libzfs libzfs_sendrecv.c, tests/runfiles linux.run

Fix hierarchy misspellings

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reported-by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Richard Laager <rlaager at wiktel.com>
Closes 8563
Closes 8622

OpenZFS on OS X/zfs 7414551module/zfs dmu_recv.c, tests/runfiles linux.run

Fix issue in receive_object() during reallocation

When receiving an object to a previously allocated interior slot
the new object should be "allocated" by setting DMU_NEW_OBJECT,
not "reallocated" with dnode_reallocate().  For resilience verify
the slot is free as required in case the stream is malformed.

Add a test case to generate more realistic incremental send streams
that force reallocation to occur during the receive.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8067
Closes 8614

OpenZFS on OS X/zfs 8d77a97module/zfs dnode.c dsl_dir.c

Fix TXG_MASK cstyle

Fix style issue for 'tx->tx_txg&TXG_MASK'.  There should be white
space around the '&' character.  Split the dnode_reallocate() ASSERT
to make it more readable to clearly separate the checks.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes 8606