Displaying 1 50 of 11,299 commits (0.010s)

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

Avoid instant panic
Delta File
+1 -1 module/zfs/zfs_vnops_osx.c
+1 -1 1 file

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Remove sa_lookup in vnop_lookup

Since we lookup parent in znode_alloc() and update it in zfs_create_link()
there is no need to call sa_lookup in vnop_lookup. Both cases;
importing already invalid parent (znode_alloc) and creating
a new symlink (zfs_create_link) then deleting parent, are handled.

We have no additional penalty in vnop_lookup in the case of hardlink
parents being correct. sa_update is needed if parent correction is
performed.

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Add code to detect when hardlink parentids are left invalid

As created by the commands

date > s ; mkdir a ; ln s a/b.lock ; rm a/b.lock ; rmdir a

would leave "s" with parentid of "a". The next vnop_lookup of "s"
corrects this situation.

Since we look up parent in znode_alloc() and update it in zfs_create_link()
there is no need to call sa_lookup in vnop_lookup. Both cases;
importing already invalid parent (znode_alloc) and creating
a new symlink (zfs_create_link) then deleting parent, are handled.

We have no additional penalty in vnop_lookup in the case of hardlink
parents being correct. sa_update is needed if parent correction is
performed.

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

Add code to detect when hardlink parentids are left invalid

As created by the commands

date > s ; mkdir a ; ln s a/b.lock ; rm a/b.lock ; rmdir a

would leave "s" with parentid of "a". The next vnop_lookup of "s"
corrects this situation.
Delta File
+74 -0 module/zfs/zfs_vnops_osx.c
+74 -0 1 file

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_znode.c

Add code to detect when hardlink parentids are left invalid

As created by the commands

date > s ; mkdir a ; ln s a/b.lock ; rm a/b.lock ; rmdir a

would leave "s" with parentid of "a". The next vnop_lookup of "s"
corrects this situation.

Since we look up parent in znode_alloc() and update it in zfs_create_link()
there is no need to call sa_lookup in vnop_lookup. Both cases;
importing already invalid parent (znode_alloc) and creating
a new symlink (zfs_create_link) then deleting parent, are handled.

We have no additional penalty in vnop_lookup in the case of hardlink
parents being correct. sa_update is needed if parent correction is
performed.

OpenZFS on OS X — module/zfs zfs_vnops_osx.c zfs_vnops_osx_lib.c

Last surviving hardlink would forget it was once a hardlink

Once the zp->z_links has returned back to 1, we would no longer correct the
parentid of the item, and it would be returned as "last used" parentid
which may have been deleted.

This corrects the panic from:

date > s ; mkdir a ; ln s a/d.lock ; rm a/d.lock ; rmdir a ; rm s

It is interesting to note that SA_ZPL_PARENT can be left "incorrect" on
disk by ZFS.

OpenZFS on OS X — module/zfs dbuf.c arc.c

Handle compressed buffers in __dbuf_hold_impl()

    In __dbuf_hold_impl(), if a buffer is currently syncing and is still
    referenced from db_data, a copy is made in case it is dirtied again in
    the txg.  Previously, the buffer for the copy was simply allocated with
    arc_alloc_buf() which doesn't handle compressed or encrypted buffers
    (which are a special case of a compressed buffer).  The result was
    typically an invalid memory access because the newly-allocated buffer
    was of the uncompressed size.

    This commit fixes the problem by handling the 2 compressed cases,
    encrypted and unencrypted, respectively, with arc_alloc_raw_buf() and
    arc_alloc_compressed_buf().

    Although using the proper allocation functions fixes the invalid memory
    access by allocating a buffer of the compressed size, another unrelated
    issue made it impossible to properly detect compressed buffers in the
    first place.  The header's compression flag was set to ZIO_COMPRESS_OFF
    in arc_write() when it was possible that an attached buffer was actually
    compressed.  This commit adds logic to only set ZIO_COMPRESS_OFF in
    the non-ZIO_RAW case which wil handle both cases of compressed buffers
    (encrypted or unencrypted).

    Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
    Signed-off-by: Tim Chase <tim at chase2k.com>
Delta File
+52 -11 module/zfs/dbuf.c
+12 -10 module/zfs/arc.c
+1 -1 module/zfs/zfs_vnops_osx.c
+65 -22 3 files

OpenZFS on OS X — tests/zfs-tests/tests/functional/cli_root/zpool_import zpool_import_errata3.ksh, tests/zfs-tests/tests/functional/rsend send_freeobjects.ksh send-cD.ksh

Add missing tests files.

OpenZFS on OS X — tests/zfs-tests/tests/functional/cli_root/zpool_import zpool_import_errata3.ksh, tests/zfs-tests/tests/functional/rsend send_freeobjects.ksh send-cD.ksh

Add missing tests files.

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

Revert "pageoutv2 must clear the dirty bit when committing the UPL"

This reverts commit 771dad6c9f0461345f68cff4ad99b79273c84d4e.

This causes Assertion failed: !(flags & UPL_PAGEOUT)

very early in testing.
Delta File
+2 -4 module/zfs/zfs_vnops_osx.c
+2 -4 1 file

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

Revert "pageoutv2 must clear the dirty bit when committing the UPL"

This reverts commit 771dad6c9f0461345f68cff4ad99b79273c84d4e.

This causes Assertion failed: !(flags & UPL_PAGEOUT)

very early in testing.
Delta File
+2 -4 module/zfs/zfs_vnops_osx.c
+2 -4 1 file

OpenZFS on OS X — module/zfs ldi_vnode.c

Merge branch 'master' of github.com:openzfsonosx/zfs
∈ Brendon Humphrey - 7a6334f1 - 2018-02-01 06:59:39
Eliminate use of uninitialized variable.
Delta File
+1 -1 module/zfs/ldi_vnode.c
+1 -1 1 file

OpenZFS on OS X — module/zfs zfs_vfsops.c

Allow zfs mount -o readonly=on option

More options could be supported beyond readonly.
Delta File
+65 -4 module/zfs/zfs_vfsops.c
+65 -4 1 file

OpenZFS on OS X — module/zfs zfs_vfsops.c

Allow zfs mount -o readonly=on option

More options could be supported beyond readonly.
Delta File
+65 -4 module/zfs/zfs_vfsops.c
+65 -4 1 file

OpenZFS on OS X — module/zfs zfs_vfsops.c

Allow zfs mount -o readonly=on option

More options could be supported beyond readonly.
Delta File
+65 -4 module/zfs/zfs_vfsops.c
+65 -4 1 file

OpenZFS on OS X — include/sys dmu_objset.h, module/zfs dmu_objset.c zil.c

Change os->os_next_write_raw to work per txg

Currently, os_next_write_raw is a single boolean used for determining
whether or not the next call to dmu_objset_sync() should write out the
objset_phys_t as a raw buffer. Since the boolean is not associated
with a txg, the work simply happens during the next txg, which is not
necessarily the correct one. In the current implementation this issue
was misdiagnosed, resulting in a small hack in dmu_objset_sync() which
seemed to resolve the problem.

This patch changes os_next_write_raw to be an array of booleans, one
for each txg in TXG_OFF and removes the hack.

Signed-off-by: Tom Caputi <tcaputi at datto.com>

OpenZFS on OS X — module/zfs ldi_vnode.c

Eliminate use of uninitialized variable.
Delta File
+1 -1 module/zfs/ldi_vnode.c
+1 -1 1 file

OpenZFS on OS X — module/zfs dmu_send.c

Raw sends must be able to decrease nlevels

Currently, when a raw zfs send file includes a DRR_OBJECT record
that would decrease the number of levels of an existing object,
the object is reallocated with dmu_object_reclaim() which
creates the new dnode using the old object's nlevels. For non-raw
sends this doesn't really matter, but raw sends require that
nlevels on the receive side match that of the send side so that
the checksum-of-MAC tree can be properly maintained. This patch
corrects the issue by freeing the object completely before
allocating it again in this case.

This patch also corrects several issues with dnode_hold_impl()
and related functions that prevented dnodes (particularly
multi-slot dnodes) from being reallocated properly due to
the fact that existing dnodes were not being fully cleaned up
when they were freed.

Signed-off-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Delta File
+43 -7 module/zfs/dmu_send.c
+43 -7 1 file

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx_lib.c

Add support for schg, sappend in chflags.

macOS has User and System variables of immutable and appendonly flags, so
we add two new flags types to accomodate them. Default ZFS immutable and
append will map to the User version.

Although manpage claims that clearing schg and sappend can only be done
in single-user mode, it is not the case for HFS.

OpenZFS on OS X — lib/libzfs libzfs_pool.c

zpool online -e to work with InvariantDisk paths

Change invd path to readlink() path to open disk without slice.
Delta File
+5 -1 lib/libzfs/libzfs_pool.c
+5 -1 1 file

OpenZFS on OS X — module/zfs zfs_vfsops.c

Correct readonly during mount
Delta File
+10 -5 module/zfs/zfs_vfsops.c
+10 -5 1 file

OpenZFS on OS X

Now with data in the test pool

OpenZFS on OS X — module/zfs zil.c

Fix for ZOL # 6916

When performing zil_claim() at pool import time, it is important that
encrypted datasets set os_next_write_raw before writing to the
zil_header_t. This prevents the code from attempting to
re-authenticate the objset_phys_t when it writes it out, which is
unnecessary because the zil_header_t is not protected by either objset
MAC and impossible since the keys aren't loaded yet. Unfortunately,
one of the code paths did not set this flag, which causes failed
ASSERTs during 'zpool import -F'. This patch corrects this issue.

Signed-off-by: Tom Caputi <tcaputi at datto.com>
Delta File
+2 -0 module/zfs/zil.c
+2 -0 1 file

OpenZFS on OS X — cmd/zfs zfs_main.c, cmd/zpool zpool_main.c

Encryption Stability and On-Disk Format Fixes

The on-disk format for encrypted datasets protects not only the
encrypted and authenticated blocks themselves, but also the order and
interpretation of these blocks. In order to make this work while
maintaining the ability to do raw sends, the indirect bps maintain a
secure checksum of all the MACs in the block below it along with a few
other fields that determine how the data is interpreted.

Unfortunately, the current on-disk format erroneously includes some
fields which are not portable and thus cannot support raw sends. It is
not possible to easily work around this issue due to a separate and
much smaller bug which causes indirect blocks for encrypted dnodes to
not be compressed, which conflicts with the previous bug. In addition,
the current code generates incompatible on-disk formats on big endian
and little endian systems due to an issue with how block pointers are
authenticated. Finally, raw send streams do not currently include
dn_maxblkid when sending both the metadnode and normal dnodes which
are needed in order to ensure that we are correctly maintaining the
portable objset MAC.

This patch zero's out the offending fields when computing the bp MAC
and ensures that these MACs are always calculated in little endian
order (regardless of the host system's byte order). This patch also
registers an errata for the old on-disk format, which we detect by

    [61 lines not shown]

OpenZFS on OS X — cmd/zfs zfs_main.c, cmd/zpool zpool_main.c

Encryption Stability and On-Disk Format Fixes

The on-disk format for encrypted datasets protects not only the
encrypted and authenticated blocks themselves, but also the order and
interpretation of these blocks. In order to make this work while
maintaining the ability to do raw sends, the indirect bps maintain a
secure checksum of all the MACs in the block below it along with a few
other fields that determine how the data is interpreted.

Unfortunately, the current on-disk format erroneously includes some
fields which are not portable and thus cannot support raw sends. It is
not possible to easily work around this issue due to a separate and
much smaller bug which causes indirect blocks for encrypted dnodes to
not be compressed, which conflicts with the previous bug. In addition,
the current code generates incompatible on-disk formats on big endian
and little endian systems due to an issue with how block pointers are
authenticated. Finally, raw send streams do not currently include
dn_maxblkid when sending both the metadnode and normal dnodes which
are needed in order to ensure that we are correctly maintaining the
portable objset MAC.

This patch zero's out the offending fields when computing the bp MAC
and ensures that these MACs are always calculated in little endian
order (regardless of the host system's byte order). This patch also
registers an errata for the old on-disk format, which we detect by

    [61 lines not shown]

OpenZFS on OS X — cmd/zfs zfs_main.c, cmd/zpool zpool_main.c

Encryption Stability and On-Disk Format Fixes

The on-disk format for encrypted datasets protects not only the
encrypted and authenticated blocks themselves, but also the order and
interpretation of these blocks. In order to make this work while
maintaining the ability to do raw sends, the indirect bps maintain a
secure checksum of all the MACs in the block below it along with a few
other fields that determine how the data is interpreted.

Unfortunately, the current on-disk format erroneously includes some
fields which are not portable and thus cannot support raw sends. It is
not possible to easily work around this issue due to a separate and
much smaller bug which causes indirect blocks for encrypted dnodes to
not be compressed, which conflicts with the previous bug. In addition,
the current code generates incompatible on-disk formats on big endian
and little endian systems due to an issue with how block pointers are
authenticated. Finally, raw send streams do not currently include
dn_maxblkid when sending both the metadnode and normal dnodes which
are needed in order to ensure that we are correctly maintaining the
portable objset MAC.

This patch zero's out the offending fields when computing the bp MAC
and ensures that these MACs are always calculated in little endian
order (regardless of the host system's byte order). This patch also
registers an errata for the old on-disk format, which we detect by

    [8 lines not shown]

OpenZFS on OS X — module/zfs ldi_vnode.c

    Provide correct error output if pool vdev (disk) is readonly.
    
    vnode_open returns EACCES when opening a readonly mounted device. This change 
translates EACCESS to EROFS, which then causes an appropraite error to be displayed in 
userspace - "cannot import '<pool>': one or more devices is readonly.
Delta File
+1 -1 module/zfs/ldi_vnode.c
+1 -1 1 file

OpenZFS on OS X — include/sys vnode.h, module/spl spl-vnode.c

Wrap required decmpfs calls in SPL

OpenZFS on OS X — include/sys vnode.h, module/spl spl-vnode.c

Wrap required decmpfs calls in SPL

OpenZFS on OS X — include/sys vnode.h, module/spl spl-vnode.c

Wrap required decmpfs calls in SPL

OpenZFS on OS X — include/sys vnode.h, module/spl spl-vnode.c

Wrap required decmpfs calls in SPL

OpenZFS on OS X — include/sys vnode.h, module/spl spl-vnode.c

Wrap required decmpfs calls in SPL

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_vfsops.c

Handle HFS decmpfs compression

The AppleFSCompression support library will compress files in Applications
like that of Mail. It does so by creating a "named stream" (resourcefork)
on the original file, and saving the compressed data. The namedstream
XATTR name is "com.apple.ResourceFork".
It also saves a "compression header" under the XATTR ""com.apple.decmpfs".
Then it truncates the original file to 0 bytes length, and calls
chflags to set UF_COMPRESSION.

Currently we handle this in zfs_vnop_setattr() when it sets UF_COMPRESSION,
we call decmpfs_decompress_file() to decompress the file back to
original form, which removes the named stream, and compression header
XATTR, and finally clears the UF_COMPRESSION flags.

This commit adds a new ZFS flags type "ZFS_COMPRESSION" to be the ondisk
value for UF_COMPRESSION. Technically, an on-disk value might not be required
as it is only set for a short time (the decmpfs_file_is_compressed() calls
getattr() to confirm the flag is set) before we decompress it, and clear it.

It is unfortunate that it fails to query the filesystem capabilities to
check for VOL_CAP_FMT_DECMPFS_COMPRESSION - as we do not advertise this
capability. (Amusingly the 'ditto --hfsCompression' command does check the
capability, so it needs to be enabled in ZFS to test with it)


    [23 lines not shown]

OpenZFS on OS X — include/sys zfs_znode.h, module/zfs zfs_vnops_osx.c zfs_vfsops.c

Handle HFS decmpfs compression

The AppleFSCompression support library will compress files in Applications
like that of Mail. It does so by creating a "named stream" (resourcefork)
on the original file, and saving the compressed data. The namedstream
XATTR name is "com.apple.ResourceFork".
It also saves a "compression header" under the XATTR "com.apple.decmpfs".
Then it truncates the original file to 0 bytes length, and calls
chflags to set UF_COMPRESSION.

Currently we handle this in zfs_vnop_setattr() when it sets UF_COMPRESSION,
we call decmpfs_decompress_file() to decompress the file back to
original form, which removes the named stream, and compression header
XATTR, and finally clears the UF_COMPRESSION flags.

This commit adds a new ZFS flags type "ZFS_COMPRESSION" to be the ondisk
value for UF_COMPRESSION. Technically, an on-disk value might not be required
as it is only set for a short time (the decmpfs_file_is_compressed() calls
getattr() to confirm the flag is set) before we decompress it, and clear it.

It is unfortunate that it fails to query the filesystem capabilities to
check for VOL_CAP_FMT_DECMPFS_COMPRESSION - as we do not advertise this
capability. (Amusingly the 'ditto --hfsCompression' command does check the
capability, so it needs to be enabled in ZFS to test with it)


    [23 lines not shown]

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Misc glitches

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Misc glitches

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Misc glitches

OpenZFS on OS X — include/sys hfs_internal.h, module/zfs zfs_vnops_osx.c

HFS ioctl defines seems to have changed at 10.13

OpenZFS on OS X — include/sys hfs_internal.h, module/zfs zfs_vnops_osx.c

HFS ioctl defines seems to have changed at 10.13

OpenZFS on OS X — include/sys hfs_internal.h, module/zfs zfs_vnops_osx.c

HFS ioctl defines seems to have changed at 10.13

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Implement VNOP_ALLOCATE

It turns out it is used more frequently than first expected.

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Implement VNOP_ALLOCATE

It turns out it is used more frequently than first expected.

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

Implement VNOP_ALLOCATE

It turns out it is used more frequently than first expected.
Delta File
+53 -4 module/zfs/zfs_vnops_osx.c
+53 -4 1 file

OpenZFS on OS X — module/zfs zfs_vnops_osx.c vdev_file.c

Implement VNOP_ALLOCATE

It turns out it is used more frequently than first expected.

OpenZFS on OS X — module/zfs arc.c

fix bad abd_return_buf_off size
Delta File
+2 -3 module/zfs/arc.c
+2 -3 1 file

OpenZFS on OS X — module/zfs zfs_vnops.c

move assertion to be in the same scope as the code it debugs
Delta File
+1 -1 module/zfs/zfs_vnops.c
+1 -1 1 file

OpenZFS on OS X — module/zfs zfs_vnops.c

missing argument
Delta File
+1 -1 module/zfs/zfs_vnops.c
+1 -1 1 file

OpenZFS on OS X — module/zfs zfs_vnops.c

small changes in clean_on_write path
Delta File
+10 -8 module/zfs/zfs_vnops.c
+10 -8 1 file

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

pageoutv2 must clear the dirty bit when committing the UPL

The UPL is created with the UPL_RET_ONLY_DIRTY flag set,
which causes the VM system to clear the hardware (HW) dirty bit and
set the software (SW) dirty bit on the pages gathered into
the UPL.   After writing the dirty pages out to the file,
it is our responsibility to clear the SW dirty bit (as is
done in zfs_pageout) using UPL_COMMIT_CLEAR_DIRTY.   This
clears the SW dirty bit, but leaves the HW dirty bit
alone so that the VM system can detect that the page has
been redirtied (typically by something that has mmapped it).

If we leave the SW dirty bit set, the next page-dirtying
operation will create a copy object.   Our shadow chain of
copy objects can grow quite large in priniciple, but we
get into trouble when it's even just one copy, since the
underlying paged-out-but-SW-dirty pages will linger until
reaped.  This reaping typically is likely to be deferred
until the file system is unmounted, at which point the
shadow chain is paged out from the top downwards, with the
result that all the data for that page in the file is
reverted to the point at which it was paged out for the
first time since mounting.

In master, the following chain of events will trigger this

    [30 lines not shown]
Delta File
+4 -1 module/zfs/zfs_vnops_osx.c
+4 -1 1 file

OpenZFS on OS X — module/zfs zfs_vnops_osx.c

pageoutv2 must clear the dirty bit when committing the UPL

The UPL is created with the UPL_RET_ONLY_DIRTY flag set,
which causes the VM system to clear the hardware (HW) dirty bit and
set the software (SW) dirty bit on the pages gathered into
the UPL.   After writing the dirty pages out to the file,
it is our responsibility to clear the SW dirty bit (as is
done in zfs_pageout) using UPL_COMMIT_CLEAR_DIRTY.   This
clears the SW dirty bit, but leaves the HW dirty bit
alone so that the VM system can detect that the page has
been redirtied (typically by something that has mmapped it).

If we leave the SW dirty bit set, the next page-dirtying
operation will create a copy object.   Our shadow chain of
copy objects can grow quite large in priniciple, but we
get into trouble when it's even just one copy, since the
underlying paged-out-but-SW-dirty pages will linger until
reaped.  This reaping typically is likely to be deferred
until the file system is unmounted, at which point the
shadow chain is paged out from the top downwards, with the
result that all the data for that page in the file is
reverted to the point at which it was paged out for the
first time since mounting.

In master, the following chain of events will trigger this

    [30 lines not shown]
Delta File
+4 -1 module/zfs/zfs_vnops_osx.c
+4 -1 1 file