OpenZFS on OS X/zfs 486e5b1. META

zfs-1.9.3
DeltaFile
+1-1META
+1-11 files

OpenZFS on OS X/zfs efa1fe4lib/libzfs libzfs_mount.c

Temporarily disable the volume custom icon code

OpenZFS on OS X/zfs 9dea79czfs_bundle Makefile.am

zfs_bundle should pick the correct mount_zfs

OpenZFS on OS X/zfs 3741cbcmodule/zfs spa_config.c

Do not spa_write_cachefile() on boot.

If we call spa_write_cachefile() very early in boot process it will
panic. Use similar tests to illumos and wait for rootdir to be set.

OpenZFS on OS X/zfs 9b67175cmd/zfs Makefile.am

Produce static mount_zfs

based off cmd/zfs/ used to mount root on boot.
DeltaFile
+10-4cmd/zfs/Makefile.am
+10-41 files

OpenZFS on OS X/zfs 81e5cf3. configure.ac, cmd Makefile.am

Remove mount_zfs and use cmd/zfs

OsX uses cmd/zfs named "mount_zfs" to do manual mounts. We also need
to make a static compile of cmd/zfs that is not codesigned for
boot support.

OpenZFS on OS X/zfs eb7f648module/zfs ldi_iokit.cpp ldi_vnode.c

Revert "Release mutex before iokit calls"

This reverts commit 5ffe3d6cefe6da0dde7bbc3987be085106aa5845.

One side-effect of this is that if you try to import the same pool
twice at the same time, we can panic in spa_add(avl_add(exists)).

Unfortunately, even though we try to let go of spa_namespace_lock
here, it is still holding spa_config().

This means this commit would try to grab spa_namespace_lock again while
holding spa_config. A different thread may be in B->A case:

spa_open_common(hold spa_namespace_lock -> spa_config_generate( grab spa_config

This unfurtunately means the original ZVOL deadlock problem is probably
back, but we'll have to find a different solution to that issue.

OpenZFS on OS X/zfs 4889d27module/zfs zfs_vnops_osx.c

Revert "Disable decmpfs decompression"

This reverts commit 37ef7e553a3f62e885fa9d8b2c62adf16061168b.

The cause of the issue has been resolved with spl:
https://github.com/openzfsonosx/spl/commit/e1134e3380ed63ebfd38888562751356e2f365c1

OpenZFS on OS X/zfs 37ef7e5module/zfs zfs_vnops_osx.c

Disable decmpfs decompression

Under Catalina, calling _decmpfs_file_is_compressed()
appears to write 0x0000200721000000 into a random memory
location. Disabled until a solution can be found.

OpenZFS on OS X/zfs 28d37abmodule/zfs zfs_ioctl.c

Temporarily remove zfs_ioc_log_history

It appears zfs_ioc_log_history() is the cause of random memory
corruption on (at least) Catalina.

Function stubbed out until cause is known and fixed.

OpenZFS on OS X/zfs d76946emodule/nvpair nvpair.c, module/zfs fm.c zfs_ioctl.c

hardcore events verify

OpenZFS on OS X/zfs 5c37cffmodule/zfs zfs_vnops_osx.c dsl_prop.c

Add _PC_MIN_HOLE_SIZE to pathconf.

Frequently used in Catalina onwards.

OpenZFS on OS X/zfs 41d0e7amodule/zfs zfs_ctldir.c

Check returncode of VN_HOLD()

Don't release vp in error case, caller handles

OpenZFS on OS X/zfs f59d4betests/zfs-tests/runfiles osx.run.in, tests/zfs-tests/tests/functional/cli_root/zpool_import zpool_import.kshlib zpool_import_errata3.ksh

zfs-test fixes

OpenZFS on OS X/zfs 94cfe2ecmd/fsck_zfs Makefile.am, cmd/mount_zfs Makefile.am

Compile fixes for Catalina / Xcode 11

Rename zfs.util -> zfs_util
Add fsck_zfs to zfs_bundle
Add mount_zfs to zfs_bundle
Add newfs_zfs to zfs_bundle

OpenZFS on OS X/zfs 86a28e3module/zfs zfs_vnops_osx.c

Catalina compatibility fixes

OpenZFS on OS X/zfs f7d25d5cmd/zfs zfs_project.c zfs_main.c, module/zfs sa.c zfs_vfsops.c

Add userobj, groupobj and project quotas

Project quota currently does nothing on OsX.

OpenZFS on OS X/zfs 28cef2finclude zfeature_common.h, include/sys dsl_dataset.h

Add types to featureflags in zfs

The boolean featureflags in use thus far in ZFS are extremely useful,
but because they take advantage of the zap layer, more interesting data
than just a true/false value can be stored in a featureflag. In redacted
send/receive, this is used to store the list of redaction snapshots for
a redacted dataset.

This change adds the ability for ZFS to store types other than a boolean
in a featureflag. The only other implemented type is a uint64_t array.
It also modifies the interfaces around dataset features to accomodate
the new capabilities, and adds a few new functions to increase
encapsulation.

This functionality will be used by the Redacted Send/Receive feature.

Reviewed-by: Matthew Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Paul Dagnelie <pcd at delphix.com>
Closes #7981

OpenZFS on OS X/zfs 15ef0eecmd/zfs zfs_main.c, include/sys zfs_vfsops.h

two

OpenZFS on OS X/zfs 31e497acmd/zfs zfs_main.c, include/sys zfs_vfsops.h dnode.h

start

OpenZFS on OS X/zfs ee975c0. META

zfs-1.9.2
DeltaFile
+1-1META
+1-11 files

OpenZFS on OS X/zfs 2c5fb9dmodule/zcommon zfs_fletcher.c, module/zfs vdev_raidz_math.c

Remove AVX512* vectorised functions

Tested to cause panic on modern CPUs.

OpenZFS on OS X/zfs 1a708falib/libzfs libzfs_dataset.c

zfs rename abort

Merging with ZOL missed some cleanup code avoid core dumps.

OpenZFS on OS X/zfs 3a45fe4. META

zfs-1.9.1
DeltaFile
+1-1META
+1-11 files

OpenZFS on OS X/zfs df73a34module/zfs arc.c

Silence debug print
DeltaFile
+1-1module/zfs/arc.c
+1-11 files

OpenZFS on OS X/zfs fbe9d96module/zfs vdev_queue.c

Use ZOL's vdev_queue_max_async_writes with guards

vdev_queue_max_writes was throwing assertions when writing
to raidz/raidz2/raidz3 vdevs because it was miscalculating
writes.

Use the ZOL version of the function edited to add an
ASSERT and a MAX to catch and avoid a division-by-zero KP,
which was the motivation for diverging from the code in
the upstream(s).

All the assertions had stacks like:

                  zfs`vdev_queue_io+0x1b3
                  zfs`zio_vdev_io_start+0x540
                  zfs`zio_nowait+0x2b2
                  zfs`vdev_raidz_io_start+0x658
                  zfs`zio_vdev_io_start+0x755
                  zfs`zio_nowait+0x2b2
                  zfs`vdev_mirror_io_start+0x419
                  zfs`zio_vdev_io_start+0x755
                  zfs`__zio_execute+0x1b2
                  spl`taskq_thread+0x237
                  kernel`call_continuation+0x17


    [7 lines not shown]

OpenZFS on OS X/zfs 1d59f7e. META

zfs-1.9.1-rc1
DeltaFile
+2-2META
+2-21 files

OpenZFS on OS X/zfs 219bb65module/zfs bqueue.c

Temporary work around to mutex starvation

While bqueue_enqueue() is sitting in cv_wait_sig() waiting for the mutex
to be returned; bqueue_dequeue() can spin so fast it never releases the
mutex long enough. We deliberately drop the mutex in each loop.

This is suggesting we have a thread starvation problem, possibly with
thread priority levels being incorrect. Further research required.

OpenZFS on OS X/zfs 6df9bb5module/zfs bqueue.c

Temporary work around to mutex starvation

While bqueue_enqueue() is sitting in cv_wait_sig() waiting for the mutex
to be returned; bqueue_dequeue() can spin so fast it never releases the
mutex long enough. We deliberately drop the mutex in each loop.

This is suggesting we have a thread starvation problem, possibly with
thread priority levels being incorrect. Further research required.

OpenZFS on OS X/zfs 18d41ffmodule/zfs dmu_send.c

hack slash
DeltaFile
+21-7module/zfs/dmu_send.c
+21-71 files

OpenZFS on OS X/zfs 1867705module/zfs abd.c

ABD changes for vectorized RAIDZ

This is a distilled version of ZoL's
https://github.com/zfsonlinux/zfs/commit/a206522c4fd31f03f14ba174d6159b72acfae0a9

* add abd_get_offset_size() interface
* abd_iter_map(): fix calculation of iter_mapsize
* add abd_raidz_gen_iterate() and abd_raidz_rec_iterate()

We had some minor differences in the size calculations causing issues in
certain corner cases.
DeltaFile
+10-6module/zfs/abd.c
+10-61 files

OpenZFS on OS X/zfs 3ca683emodule/zfs dmu_send.c

Pick HAVE_LARGE_STACK path on OsX

OpenZFS on OS X/zfs 1a5cd97module/zfs bqueue.c

Call cv_signal() with mutex held

In bqueue_dequeue(), call cv_signal() with bq_lock held.
Re-enable rsend_009_pos to test the fix.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Boris Protopopov <boris.protopopov at actifio.com>
Closes #5887

OpenZFS on OS X/zfs 3859c2einclude/sys zfs_ioctl.h, module/zfs dmu_send.c dmu_recv.c

Revert "Fix send/recv lost spill block"

This reverts commit 8d7a713e91f7f4594c63e9c851fe926faee3d6d5.

OpenZFS on OS X/zfs d6e954dmodule/zfs zio.c

Enable raw writes to perform dedup with verification

Reviewed by: Dan Kimmel <dan.kimmel at delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: David Quigley <david.quigley at intel.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Issue #5078
DeltaFile
+46-24module/zfs/zio.c
+46-241 files

OpenZFS on OS X/zfs 2be5466module/zfs zio.c

Enable raw writes to perform dedup with verification

Reviewed by: Dan Kimmel <dan.kimmel at delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: David Quigley <david.quigley at intel.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Issue #5078
DeltaFile
+47-7module/zfs/zio.c
+47-71 files

OpenZFS on OS X/zfs 99992e3tests/config commands-osx.m4

Missing commands-osx.m4

OpenZFS on OS X/zfs bfe2903module/zfs zvolIO.cpp zil.c

Correct ASSERT for --enable-debug case

OpenZFS on OS X/zfs 4ab7121module/zfs vdev_indirect.c vdev_removal.c

Correct ASSERTS with --enable-debug

OpenZFS on OS X/zfs bbc9d8clib/libdiskmgt Makefile.am

Fix typo in libdskmgt Makefile

OpenZFS on OS X/zfs 09fb3cfmodule/icp/algs/aes aes_impl.c aes_impl_generic.c, module/icp/asm-x86_64/aes aes_aesni.S aes_intel.S

Port ZOL assembler work

Update vdev_raidz.c from ZOL
add in various assembler versions, and with updated SPL
simd_x86.h CPU feature detection remains the same.

add string kstat zfs_vdev_raidz_impl to show picked and available
alg, and allow set preferred alg.

Assembler work: aes, gcm

Add ZOL assembler work for

gcm pclmulqdq
aes x86_64
aes aesni

ZOL zfs_fletcher assembler work

Bring in the assembler versions of fletcher, and kstat to select.
Due to missing kstat_set_ops_raw() functionality, the benchmarking is
not visible, added dprintf() for those curious.

OpenZFS on OS X/zfs ab65cdbmodule/zfs zfs_ioctl.c

zfs_ioc_create would eat error return

Creating a ZVOL too large would return confusing error message:

cannot open 'tank/vol': dataset does not exist

as the zvol_create_minors() tail code would overwrite the "error"
variable. Now returns with correct:

cannot create 'tank/vol': out of space

OpenZFS on OS X/zfs fb98be7module/zfs arc.c

Fix comparison signedness in arc_is_overflowing()

When ARC size is very small, aggsum_lower_bound(&arc_size) may return
negative values, that due to unsigned comparison caused delays, waiting
for arc_adjust() to "fix" it by calling aggsum_value(&arc_size).  Use
of signed comparison there fixes the problem.

Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by:  Alexander Motin <mav at FreeBSD.org>
Closes #8873
DeltaFile
+5-1module/zfs/arc.c
+5-11 files

OpenZFS on OS X/zfs 230c7e4module/zfs dmu_recv.c

Fix incorrect error message for raw receive

This patch fixes an incorrect error message that comes up when
doing a non-forcing, raw, incremental receive into a dataset
that has a newer snapshot than the "from" snapshot. In this
case, the current code prints a confusing message about an IVset
guid mismatch.

This functionality is supported by non-raw receives as an
undocumented feature, but was never supported by the raw receive
code. If this is desired in the future, we can probably figure
out a way to make it work.

Reviewed by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Issue #8758 
Closes #8863 

OpenZFS on OS X/zfs 87f7544module/zfs vdev_indirect.c

Remove warning in vdev_obsolete_sm_object

OpenZFS on OS X/zfs a5a4a1einclude/sys metaslab.h, module/zfs metaslab.c vdev.c

Allow metaslab to be unloaded even when not freed from

On large systems, the memory used by loaded metaslabs can become
a concern. While range trees are a fairly efficient data structure,
on heavily fragmented pools they can still consume a significant
amount of memory. This problem is amplified when we fail to unload
metaslabs that we aren't using. Currently, we only unload a metaslab
during metaslab_sync_done; in order for that function to be called
on a given metaslab in a given txg, we have to have dirtied that
metaslab in that txg. If the dirtying was the result of an allocation,
we wouldn't be unloading it (since it wouldn't be 8 txgs since it
was selected), so in effect we only unload a metaslab during txgs
where it's being freed from.

We move the unload logic from sync_done to a new function, and
call that function on all metaslabs in a given vdev during
vdev_sync_done().

Reviewed-by: Richard Elling <Richard.Elling at RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Paul Dagnelie <pcd at delphix.com>
Closes #8837

OpenZFS on OS X/zfs 2290dd2module/zfs arc.c

l2arc_apply_transforms: Fix typo in comment

Reviewed-by: Chris Dunlop <chris at onthe.net.au>
Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Richard Laager <rlaager at wiktel.com>
Signed-off-by: Allan Jude <allanjude at freebsd.org>
Closes #8822
DeltaFile
+3-3module/zfs/arc.c
+3-31 files

OpenZFS on OS X/zfs 3b20832man/man5 zfs-module-parameters.5, module/zfs metaslab.c

Reduced IOPS when all vdevs are in the zfs_mg_fragmentation_threshold

Historically while doing performance testing we've noticed that IOPS
can be significantly reduced when all vdevs in the pool are hitting
the zfs_mg_fragmentation_threshold percentage. Specifically in a
hypothetical pool with two vdevs, what can happen is the following:
Vdev A would go above that threshold and only vdev B would be used.
Then vdev B would pass that threshold but vdev A would go below it
(we've been freeing from A to allocate to B). The allocations would
go back and forth utilizing one vdev at a time with IOPS taking a hit.

Empirically, we've seen that our vdev selection for allocations is
good enough that fragmentation increases uniformly across all vdevs
the majority of the time. Thus we set the threshold percentage high
enough to avoid hitting the speed bump on pools that are being pushed
to the edge. We effectively disable its effect in the majority of the
cases but we don't remove (at least for now) just in case we hit any
weird behavior in the future.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8859 

OpenZFS on OS X/zfs b58168bmodule/zfs zfs_znode.c

Drop objid argument in zfs_znode_alloc() (sync with OpenZFS)

Since zfs_znode_alloc() already takes dmu_buf_t*, taking another
uint64_t argument for objid is redundant. inode's ->i_ino does and
needs to match znode's ->z_id.

zfs_znode_alloc() in FreeBSD and illumos doesn't have this argument
since vnode doesn't have vnode# in VFS (hence ->z_id exists).

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro at osnexus.com>
Closes #8841

OpenZFS on OS X/zfs 2c50581module/zfs bqueue.c

Wait in 'S' state when send/recv pipe is blocking

Reviewed-by: Paul Dagnelie <pcd at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: DHE <git at dehacked.net>
Closes #8733 
Closes #8752