Displaying 1 50 of 38,378 commits (0.010s)

DragonFlyBSD — sys/kern kern_synch.c

kernel - Fix SMP tsleep_interlock() vs wakeup() race

* Problem triggered by kern_mutex.c, probably only triggered by mutexes.
  The mutex sets state non-atomically and calls wakeup() with no
  intervening atomic op.  wakeup() loads the cpumask to wakeup
  non-atomically.  This can cause the wakeup() code to miss an
  interaction with the cpumask set by another cpu.

  cpu A:
        set state                        <-- sets state
        wakeup()                        <-- loads cpu mask non-atomically

  cpu B:
        tsleep_interlock()                <-- sets cpu mask atomically
        if (checkstate failed)                <-- checks state
                tsleep()

* Fix the problem by add a lfence() in the wakeup() code prior to
  loading the cpumask.  Even though I'm fairly sure only the kern_mutex.c
  code can causing this problem, putting the lfence in wakeup() is the
  safest mechanic.
Delta File
+9 -2 sys/kern/kern_synch.c
+9 -2 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_inode.c hammer2_xops.c

hammer2 - Cleanup

* Cleanup two debug messages that are no longer needed.

DragonFlyBSD — sys/vfs/hammer2 hammer2_bulkfree.c hammer2_io.c

hammer2 - Clean DIO invalidation flags in more cases

* DIO invalidation state allows a (64KB) buffer to be allocated from disk
  without reading it, retaining this knowledge across a freemap allocation
  into the chain.  It is primarily used by the freemap allocator when
  digging into new (previously fully unallocated) 64KB blocks to avoid
  unnecessary disk I/O.

* Clear DIO invalidation flags for data blocks being adjusted by
  bulkfree for any transition rather than just for the 11->10 transition.
  This is an attempt to cleanup corruption which can occur during heavy
  I/O + concurrent bulkfree operation.

* When clearing DIO invalidation state, clear both flags rather than just
  one flag.

DragonFlyBSD — sbin/hammer2 cmd_debug.c

hammer2 - limit show command output when crc error

* When a CRC error check fails on a recursive chain during the debugging
  'show' command output, limit recursion down that chain to one level.
  This allows the potentially corrupted data record to be dissected but
  does not attempt further indirection through it.
Delta File
+19 -6 sbin/hammer2/cmd_debug.c
+19 -6 1 file

DragonFlyBSD — sys/kern kern_timeout.c, sys/sys callout.h

kernel - Fix callout race and panic (2)

* Previous fix couldn't handle a callout_stop() vs callout_reset() race
  on different cpus.  When this race occurred, the callout_stop() would
  get stuck waiting for the ARMED bit to clear, which it never would
  because a new callout_reset() re-armed it.

* Refactor the callout code to clean it up.  Remove the ARMED flag, instead
  a callout is considered ARMED if the IPI_MASK counter bits are not zero
  or PENDING is set.  Use these rules to lock-in the owning cpu with an
  atomic op

* Integrate all flags settings in IPIs with the atomic decrement of the
  IPI_MASK count, rather than as separate operations.

* callout_stop() now just waits for the IPI sequence to end.  callout_reset()
  now re-tests whether the callout is still armed or not after callout_stop()
  returns (since it needs to know for sure) before it tries to take over
  the callout.  This fixes the race.
Delta File
+202 -345 sys/kern/kern_timeout.c
+48 -17 sys/sys/callout.h
+250 -362 2 files

DragonFlyBSD — sys/vfs/hammer2 hammer2_chain.c hammer2_ioctl.c

hammer2 - Try a different approach to bulkfree

* Attempt to fix an issue in the previous version where some dirty chains
  could slip through the gaps and prevent bulkfree from working properly.

* Do a full filesystem sync instead of a partial sync.

* Use the post-sync hmp->volsync structure to obtain the blockref array,
  interlocked against the sync.

DragonFlyBSD — sbin/hammer2 main.c hammer2.8

hammer2 - Cleanup manual page, add help, remove bulkfree-async

* Remove bulkfree-async.  Just stick with the synchronous bulkfree for
  now.

DragonFlyBSD — sys/kern kern_timeout.c kern_event.c

kernel - Fix callout race and panic

* Fix race (found by ivadasz) related to the IPI/WAIT code unblocking
  before the IPI is able to finish adjusting the knode and callout.
  The wait code was only waiting for the IPI counter to reach 0 via
  IPI_MASK, it also had to wait for the ARMED bit to get cleared.

* Avoid retesting c->c_flags to handle WAITING bit change races.  Instead,
  fully integrate the test-and-clear of the WAITING bit into
  callout_unpend_disarm().

* Fix an issue where callout_terminate() fails to IPI the remote cpu
  due to the function dispatch code clearing the ARMED bit.  No
  longer clear the ARMED bit.  This ensures that a termination or stop
  waits for the callout to return.

  This change means that synchronous callout operations to other cpus will
  be more expensive.  However, the kernel generally does not do cross-cpu
  callouts any more so its generally non-problem.

* Remove the now unused callout_maybe_clear_armed() inline.

* Also clear kn->kn_hook for EVFILT_TIMER when removing a callout, as
  a safety.

Reported-by: ivadasz (Imre Vadasz)
Delta File
+41 -31 sys/kern/kern_timeout.c
+1 -1 sys/kern/kern_event.c
+42 -32 2 files

DragonFlyBSD — sys/vfs/hammer2 hammer2_xops.c hammer2_inode.c

hammer2 - Fix improper bzero length in rename code

* When renaming to a long filename the data buffer is sized according
  to chain->bytes.  sizeof(chain->data->buf) is too much.  Fix the
  bzero.

* Fixes a panic.

DragonFlyBSD — etc/mtree BSD.root.dist

build - Add /home to the mtree

* It is reasonable to create a /home directory in the base system
  template.
Delta File
+2 -0 etc/mtree/BSD.root.dist
+2 -0 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_vnops.c hammer2.h

hammer2 - Remove LOCK* debugging

* Remove LOCKSTART, LOCKSTOP and friends.  This debugging code was used to
  check lock counts on entry and exit from VOPs and is no longer needed.

DragonFlyBSD — sys/vfs/hammer2 hammer2_vnops.c hammer2_inode.c

hammer2 - Flesh out kqueue support

* Do a quick run through and add missing KNOTE's.  The support now matches
  H1.

* Quickly tested via the monitor utility, appears to work.

DragonFlyBSD — lib/libc/sys kqueue.2

kqueue.2: Improve explanation of EV_EOF for fifos.

* The EV_EOF flag can't be cleared with EV_CLEAR. For Fifos it will be
  cleared when a new reader (for EVFILT_WRITE) or writer (for EVFILT_READ)
  connects.
Delta File
+3 -5 lib/libc/sys/kqueue.2
+3 -5 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_iocom.c

hammer2 - cleanup

* Cleanup some of the SPAN debug messages
Delta File
+11 -7 sys/vfs/hammer2/hammer2_iocom.c
+11 -7 1 file

DragonFlyBSD — lib/libc/sys kqueue.2

kqueue.2: Mention EV_NODATA flag, which may be set in addition to EV_EOF.

* The EV_NODATA flag was added in 3bcb6e5e80cb05cecaddd866d4b15b428334b9aa.
Delta File
+4 -0 lib/libc/sys/kqueue.2
+4 -0 1 file

DragonFlyBSD — sys/kern kern_event.c

kqueue: Make EVFILT_USER event behaviour more consistent.

* Stop abusing the kn->kn_sfflags value for storing the current state of
  the EVFILT_USER filter. Instead use kn->kn_fflags like other filters.
  Similarly store the data value in kn->kn_data instead of kn->kn_sdata.
  This means that the fflags value gets reset when EV_CLEAR was specified
  when adding the event, and the event is received by userspace. This
  behaviour is consistent with existing kqueue filters, and allows using
  EVFILT_USER properly as an edge-triggered event when using the fflags,
  and not just level-triggered.

* Don't clear kn->kn_fflags when the event is modified with EV_CLEAR. Doing
  this wasn't affecting the actual state of the EVFILT_USER event before
  this change (since the state was kept in kn->kn_sfflags instead).

* All this also avoids blindly copying the fflags value that was specified
  when adding the event. Instead the NOTE_FFLAGSMASK mask is applied, and
  the NOTE_FF* options are used, so the returned fflags value should now
  always only have the lower 24 bits set.

* Make setting the fflags and data value when adding the event work as
  might be expected.
Delta File
+43 -7 sys/kern/kern_event.c
+43 -7 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_ioctl.c hammer2_vfsops.c

hammer2 - Fix snapshots and multi-label mounts

* Allow the same /dev/blah at DIFFERENTLABEL to be specified in a mount
  command, so multiple labels from the same device can be mounted.

* Devfs can throw different vnodes for the same device.  When matching
  up hammer2_dev, check devvp->v_rdev for a match as well.

* Fix a number of bugs in the snapshot code that left a hammer2_inode
  structure hanging and caused a panic.

* Fix races in admin thread flag messaging that could lead to 60-second
  delays during umount.

DragonFlyBSD — sys/vfs/hammer2 hammer2_bulkfree.c hammer2_flush.c

hammer2 - revamp comments

* Adjust a few comments

DragonFlyBSD — sbin/hammer2 cmd_snapshot.c main.c

hammer2 - Refactor snapshot frontend

* We can no longer snapshot a subdirectory (due to inodes being indexed
  under the root and no longer embedded in directories), so simply
  snapshot naming.

* Name snapshots only after their PFS label plus the date and time.

* Flesh out command help

DragonFlyBSD — sys/kern kern_event.c

kqueue: Fix typo in filt_userattach function: kn_fflags vs. kn_sfflags.

* This typo meant that adding an EVFILT_USER event with NOTE_TRIGGER already
  set, would fail to trigger the user event.

* So far I didn't find any EVFILT_USER usages in opensource code, where
  the NOTE_TRIGGER flag is set when adding the EVFILT_USER event, so this
  fix seems to be a cornercase in practice.
Delta File
+1 -1 sys/kern/kern_event.c
+1 -1 1 file

DragonFlyBSD — sbin/mount_hammer2 mount_hammer2.c

hammer2 - Fix root remount for hammer2 root

* mount_hammer2 should not fail in early boot trying to remount /
  if it is unable to connect to the cluster controller, as it might not
  be able to start it due to lo0 not being configured yet.

DragonFlyBSD — sys/vfs/hammer2 hammer2_vfsops.c

hammer2 - Fix ro -> rw mount upgrade

* Fix panic when upgrading a ro mount to rw.
Delta File
+18 -6 sys/vfs/hammer2/hammer2_vfsops.c
+18 -6 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_strategy.c

hammer2 - Fix bug in comp:autozero mode

* comp:autozero mode did not properly set the check code in the live
  de-dup case, resulting in I/O errors later on.

* Fixes issues installing /boot (newfs_hammer2 on an 'a' partition uses
  comp:autozero by default in order to disable compression, so the boot
  loader can read the files without needing decompression code).
Delta File
+11 -3 sys/vfs/hammer2/hammer2_strategy.c
+11 -3 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_flush.c hammer2_bulkfree.c, test/debug h2chains.c

hammer2 - Fix bulkfree bugs

* Fix a bug where the HAMMER2_CHAIN_ONFLUSH flag could become desynchronized,
  preventing some modifications from chaining the flag to the root.

  This caused the bulkfree's flush to miss some chains and thus free
  blocks that should not have been freed.

  With this fix concurrent bulkfree and modifying operations appear to
  work properly.

* For now do two quick flush passes in the bulkfree code.  We probably
  need only one.

  Note on the bulkfree code: This code operates by flushing currently
  modified chains and then taking a snapshot of the vchain blockmap.
  The bulkfree code can then operate on the snapshot concurrent with
  active filesystem operations, without interfering with or being
  interfered with by those operations.

  The bulkfree code does not have to flush modified in-memory inodes or
  buffer cache buffers, it need only flush modified hammer2_chain structures
  which may have done allocations, in order to properly account for those
  allocations.  Bulkfree runs in two stages precisely in order to allow
  concurrency.

    [5 lines not shown]

DragonFlyBSD — sbin/hammer2 cmd_bulkfree.c main.c, sys/vfs/hammer2 hammer2_admin.c hammer2_bulkfree.c

hammer2 - Add kernel-thread-based async bulk free

* Add an async bulk-free feature and a kernel thread to allow the hammer2
  vfs itself to run bulk frees.

* Revamp the hammer2 thread management code a bit to support the new use,
  and clean-up the API.

DragonFlyBSD — lib Makefile, lib/libstand hammer2.c

hammer2 - buildworld components, boot2 and loader support

* Compile hammer2 components unconditionally in buildworld now.
  Leave conditionals commented out for the moment, they will be
  removed entirely later on.

* Finish revamping the boot code to properly handle H2 DIRENTs.
  The boot code can now boot from a HAMMER2 'a' partition if desired.
  UFS is no longer required.

* Change newfs_hammer2, mount_hammer2, and the hammer2 vfs to supply
  a default super-root label if none specified.  Supply "@BOOT" when
  mounting from the 'a' partition, "@ROOT" when mounting from the 'd'
  partition, and "@DATA" when mounting from any other partition.

  mount_hammer2 no longer defaults to using @LOCAL for the label when
  none is specified.  Instead we want hammer2 mounts that fail to supply
  a label to behave in a manner that is more likely to be expected by
  the superuser.

* Do not yet include hammer2 in X86_64_GENERIC.  At the moment it must
  be supplied in /boot/loader.conf if the user wishes to boot from it.

DragonFlyBSD — share/misc pci_vendors

Update the pciconf(8) database.

August 10, 2017 snapshot from http://pciids.sourceforge.net/
Delta File
+126 -10 share/misc/pci_vendors
+126 -10 1 file

DragonFlyBSD — sys/sys uio.h

<sys/uio.h>: Clean up the namespace.
Delta File
+12 -8 sys/sys/uio.h
+12 -8 1 file

DragonFlyBSD — sys/vfs/msdosfs msdosfs_vfsops.c

kernel/msdosfs: Add additional sanity checks before mounting.

This is FreeBSD revisions 113979 and 206098.

Taken-from:    FreeBSD
Reported-by:   <open.source at ribose.com>
Dragonfly-bug: <https://bugs.dragonflybsd.org/issues/3050>
Delta File
+7 -0 sys/vfs/msdosfs/msdosfs_vfsops.c
+7 -0 1 file

DragonFlyBSD — sys/vfs/msdosfs msdosfs_vfsops.c

kernel/msdosfs: Add additional sanity checks before mounting.

This is FreeBSD revisions 113979 and 206098.

Taken-from:    FreeBSD
Reported-by:   <open.source at ribose.com>
Dragonfly-bug: <https://bugs.dragonflybsd.org/issues/3050>
Delta File
+7 -0 sys/vfs/msdosfs/msdosfs_vfsops.c
+7 -0 1 file

DragonFlyBSD — sys/kern subr_csprng.c kern_synch.c

kernel - More high-process-count fixes

* The recent tsleep/wakeup work allows multiple cpumasks to associate with
  the same pcpu sleep queue.  This created a degenerate condition where
  the cpu bit would not get cleared in all masks, causing spurious IPIs
  to be sent to the cpu.

  Fixed by ensuring that the bit is cleared after receiving an IPI, if the
  queue is found to be empty.  That is, we allow spurious IPIs to happen,
  but ensure that the condition is cleared so the same driving event does
  not cause additional ones.

* Remove debugging kprintf()s from usched_dfly that were not intended to
  be committed.

* Cleanup the csprng code style a bit and add some missing parenthesis.

DragonFlyBSD — sys/kern usched_dfly.c

kernel - Fix uload calculation in usched_dfly

* When a very large number of processes are runnable (greater than 32767),
  the uload rollup field in dfly_pcpu[] can overflow.  In addition, the
  delta calculation for individual processes can also overflow.

  Fix both cases.

* Symptoms: one cpu starts getting an excessive percentage of the process
  load as the system attempts to continuously rebalance into it.
Delta File
+31 -15 sys/kern/usched_dfly.c
+31 -15 1 file

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_subpart_hammer.c flow.c, usr.sbin/installer/libinstaller diskutil.c diskutil.h

install - Fix a few issues

* sleep 1 second before [re]surveying drives to give devfs time to
  [re]probe the drive.

* fn_subpart_hammer was apparently trying to check the capacity of the
  current slice when there is no current slice.  Check the disk capacity
  instead.

DragonFlyBSD — test/sysperf Makefile

test - Add various 900,000 process tests (2)

* Connect tests to Makefile
Delta File
+11 -1 test/sysperf/Makefile
+11 -1 1 file

DragonFlyBSD — test/sysperf pipe900k.c loop900k.c

test - Add various 900,000 process tests

Add tests involving a large number of user processes.  These tests default
to creating 900,000 user processes and require a machine with at least
128GB of ram.  The machine must be booted with kern.maxproc=4000000 in
/boot/loader.conf.

These test kernel resource management in the face of a large number of
processes.  scheduler, pipe kvm, concurrent fork and exec, general kernel
handling of large numbers of processes, callout, tsleep, and wakeup.
Also concurrent teardown for 900,000 processes when you hit ^C.

* loop900k -        Fork/exec's 900,000 processes which sleep 80 seconds and
                then go into an infinite loop.

                Tests the scheduler and kernel resource management.

* pipe900k -        Fork/exec's 900,000 processes which chain 900,000 pipe()s
                between them and runs an end-to-end test.

                Tests pipe chaining, kernel pipe resource creation and
                teardown, and kernel resource management.

* sleep900k -        Fork/exec's 900,000 processes which sleep 60 seconds and
                then go into an infinite loop with a 1/10 second sleep.

    [13 lines not shown]

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_subpart_hammer.c, usr.sbin/installer/libinstaller diskutil.c diskutil.h

installer: Change fn_mapper_name() to subpartition_get_mapper_name().

Similar to subpartition_get_device_name(). It also helps cleaning up
long lines.

DragonFlyBSD — sys/kern subr_param.c

kernel - Increase flock() / posix-lock limit

* Change the scaling for kern.maxposixlocksperuid to be based on maxproc
  instead of maxusers.

* Increase the default cap to approximately 4 * maxproc.  This can lead
  to what may seem to be large numbers, but certain applications might use
  posix locks heavily and overhead is low.  We generally want the cap to
  be large enough that it never has to be overridden.
Delta File
+1 -1 sys/kern/subr_param.c
+1 -1 1 file

DragonFlyBSD — share/man/man7 tuning.7 build.7

doc - Update build(7) and tuning(7)

* Further modernization, adjust tuning to reflect the new reality.
Delta File
+63 -30 share/man/man7/tuning.7
+44 -0 share/man/man7/build.7
+107 -30 2 files

DragonFlyBSD — sys/kern uipc_mbuf.c

kernel - Increase default mbufs

* Increase default mbuf clusters by 4x

* Increase default mbuf jclusters by 2x

* Change the minimum (for now) to a smaller portion of the total mbuf
  capacity.
Delta File
+18 -12 sys/kern/uipc_mbuf.c
+18 -12 1 file

DragonFlyBSD — sys/net/ipfw ip_fw2.c

ipfw: Use SEQ_GEQ
Delta File
+3 -5 sys/net/ipfw/ip_fw2.c
+3 -5 1 file

DragonFlyBSD — sys/kern sys_pipe.c, sys/vm vm_zone.c vm_map.c

kernel - Remove bottlenecks related to mass pipe closures

* These changes fix teardown latencies involved with destroying a large
  number of pipe()s at the same time, for example when a large number
  of processes are killed simultaneously.  Testing with 900,000 processes
  chaining 900,000 pipe()s, the system could become unusable for upwards
  of an hour when the test is killed with ^C prior to these changes.

  After these changes the system balks for less than 30 seconds in the
  900,000 process case.

* Wrap pipe teardowns in a pcpu mutex.  Such teardowns only occur when the
  normal pcpu pipe cache / hysteresis gets blown out due to a large number
  of closures occurring simultaneously and are not typically in the
  critical path.

  The pcpu mutex allows good concurrency while also limiting the number of
  processes which can be tearing down pipe KVM at the same time.  If
  we have say 900,000 processes all exiting, this mechanism limits the
  teardown to (ncpus) concurrent processes.

  The mutex uses a queued lock (whereas the lockmgr spams wakeup()s),
  so we use a mutex to avoid a degenerate O(N^2) wakeup situation.
  This removes an enormous amount of overhead during the teardown.


    [14 lines not shown]
Delta File
+30 -2 sys/kern/sys_pipe.c
+15 -6 sys/vm/vm_zone.c
+7 -0 sys/vm/vm_map.c
+1 -1 sys/vm/vm_zone.h
+53 -9 4 files

DragonFlyBSD — sys/platform/pc64/acpica OsdEnvironment.c

kernel/acpica: Name a variable more appropriately (from FreebSD).

DragonFlyBSD — usr.bin/truss Makefile

i386 removal, part 62/x: Fix per-architecture config file naming in truss.
Delta File
+1 -1 usr.bin/truss/Makefile
+1 -1 1 file

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c

installer: Don't add an entry for 'root' to /etc/crypttab.

It is not necessary and will in fact generate a 'Device root already
exists.' warning because root has already been configured and mounted
at this point.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c

installer: Don't add an entry for 'root' to /etc/crypttab.

It is not necessary and will in fact generate a 'Device root already
exists.' warning because root has already been configured and mounted
at this point.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_configure.c

installer: Fix mapper name generation for encrypted installation.

fn_mapper_name() is supposed to take the mountpoint name, not the
device name.

This actually broke the install in environments where no /dev/serno/...
device was available and both root and /build were selected for
encrypting because the device name would never contain a '/' and so
the mapper name 'root' was returned for both.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_configure.c

installer: Fix mapper name generation for encrypted installation.

fn_mapper_name() is supposed to take the mountpoint name, not the
device name.

This actually broke the install in environments where no /dev/serno/...
device was available and both root and /build were selected for
encrypting because the device name would never contain a '/' and so
the mapper name 'root' was returned for both.

DragonFlyBSD — . Makefile_upgrade.inc, sys/sys sysvipc.h

<sys/sysvipc.h>: Remove another no longer used header file.

It should have been removed in 0951a8e1611f221f03c7cb219aae011dbca8a17d.
Delta File
+0 -39 sys/sys/sysvipc.h
+1 -0 Makefile_upgrade.inc
+1 -39 2 files

DragonFlyBSD — sys/kern kern_synch.c init_main.c, sys/sys proc.h kernel.h

kernel - Scale tsleep() performance vs many (thousands) of processes

* In situations where a huge number of processes or threads are present
  and sleeping (that is, more than a few thousand), the global cpumask hash
  table used by tsleep() would saturate and effectively cause any wakeup()
  call to broadcast to all CPUs.

* Refactor the tsleep initialization code to allow the global cpumask
  hash table and the pcpu hash tables to be dynamically allocated.

* Allocate a MUCH larger global cpumask hash table, and significantly
  smaller pcpu hash tables.  The global cpumask hash table is now
  sized to approximate 2 * maxproc, greatly reducing cpumask collisions
  when large numbers of processes exist in the system.

  The pcpu hash tables can be smaller without effecting performance.  This
  will simply result in more entries in each queue which are trivially
  iterated.

  Nominal maxproc ~32,000 -> in the noise (normal desktop system)
  Nominal maxproc ~250,000 -> 16MB worth of hash tables (on a 128G box)
  Maximal maxproc ~2,000,000 -> 122MB worth of hash tables (on a 128G box)

* Remove the unused sched_quantum sysctl and variable.


    [7 lines not shown]

DragonFlyBSD — sys/kern subr_param.c

kernel - Change maxproc cap calculation

* Increase the calculation for the maxproc cap based on physical ram.
  This allows a machine with 128GB of ram to maxproc past a million,
  though it should be noted that PIDs are only 6-digits, so for now
  a million processes is the actual limit.
Delta File
+4 -3 sys/kern/subr_param.c
+4 -3 1 file