*BSD Commit Log Search.
The first two items will only affect future commits for the time being. Reindexing that doesn't involve nuking everything first is a TODO.
Recent fixes and tweaks:
Thanks to Alexey Dokuchaev for some of these suggestions.
Domain Name: FRESHBSD.ORG
Creation Date: 2006-03-06T18:30:19Z
10 years ago today, woo.
Let's pretend I did something cool for it :P
v4's getting close to general usability. Filtering works, indexing works, style.. is usable. And slightly responsive.
I'll make some effort now to keep a test version of the site running at v4.freshbsd.org. Most things should generally work, please mess about with it and report issues.
As an aside, here's some code I made in the process of making v4. More will follow:
The site's been migrated to a fresh Elasticsearch instance - several shards on the previous one have failed, and reindexing from scratch was the quickest way to get things going again.
This is unfortunately going to break some old links for projects that are being indexed using git mirrors of their CVS repositories (NetBSD, OpenBSD, pkgsrc). I've been hoping the band-aid which kept them working would hold until v4 rolled out, but.. meh.
Current state of v4 is a largely complete backend and a skeleton frontend. Should be ready for some testing next month \o/
FreshBSD v4 development is going well. Woo.
New git importer is pretty much done and much better than the existing hackjob. Currently working on the new SVN importer, which will be more of a tidy-and-refactor of the existing one.
CVS is very much planned - I have a promising prototype using a modified version of rcshist so hopefully I can stop depending so much on third party git mirrors or import tools.
With a little luck, we could be looking at a test site within a few weeks, but I think we all know how these things can go, so no promises :P
I'll be making a best-effort to keep any existing links working. This includes mapping commits from git mirrors to their CVS counterparts.
FreshBSD's master server has been upgraded.
First and most important, a Final Solution to the occasional bout of swapping has been enacted:
Mem: 2901M Active, 378M Inact, 6415M Wired, 3280K Cache, 131G Free
That should do, right? I bet I could run two java's now!
Second, the case has been replaced, so now it's finally back inside one instead of being precariously balanced on top of a sheet of wood and a cardboard box.
More importantly, all the lights on it blink when they're supposed to.
Many thanks to Silverstar Components Ltd for the great price on the case, and to the friend who gave me a great price on the memory (and wishes to remain nameless).
Made some tweaks to git indexing, excluding future HardenedBSD merge commits (temporary band-aid), and stripping off gpg signatures.
Will sort out something better for merge handling when I get the chance. Thanks to Shawn Webb for giving me some ideas.
The opnsense-tools repository is now being indexed. The commits are smooshed together with -core (for now), but they shouldn't be too tricky to distinguish.
The server's now upgraded. It didn't go entirely according to plan, but thanks to the magic of wood and cardboard it works. So long as I don't knock it.
Individual search performance seems reduced about as much as you'd expect going from 2.6 to 2.1GHz. I'll look into resharding elasticsearch to try and make better use of the extra cores.
Thanks to my Monday morning delivery arriving Sunday (thanks Scan ), and after one faulty PSU, I now have the new server motherboard running, with dual 2.1Ghz Xeon L5639's and 18GB of ECC DDR3.
Bar further issues, I'll be transferring it into the current server (bumping it to 42GB) in the morning. Expect a bit of degraded service shortly, followed by the opposite \o/
Upgrade is now scheduled for Monday 2nd Feb, while I wait for another pair of LGA1366 coolers. The ones I have are the push-button type, while the motherboard I've got has very securely glued on screw-in backplates.
It's an X8DTN+ in case anyone was wondering. Yes, I still use my PCI-X cards.
This weekend should be seeing a server upgrade, bringing us to 12 Xeon cores, 42GB of memory, and twice the memory bandwidth.
If someone can find me a CT51272BA1339.18FR (PC3-106000E-9-10-E0) UDIMM, I can make that 48GB.
After a few tweaks the dupe checker's tagged about a million commits and pointed them at the currently active versions. Took about 9 hours. Remaining duplicates were still alive in git since the last run, so you'll at least get diffs and file views with them until git prunes them.
OpenBSD ports is still lacking a usable source git repository. If anyone can suggest one, please do.
Alongside a reindexing of OpenBSD commits due to a change in repository comes a long-awaited feature: duplicate removal. Not yet reflected in facet counts, so some projects appear to have way more commits than they actually do.
Direct links to old commits will still work, but diffs and file views won't be available. Later on I'll have such links forward you to the "current" version where everything will work.
git fetching has been tweaked, unwedging certain repos that weren't updating properly (OPNsense, OpenBSD Xenocara, HardenedBSD). I've also removed a couple of stale svnsync locks (Elftoolchain, NAS4Free) preventing them from updating.
secure.freshbsd.org's SSL certificate has been updated. Yay StartSSL.
The search form is back. Oops.
Git merge commits are rendered much more sensibly now. No more amusingly deep meaningless nesting when there's a lot of child commits being pulled in.
Facets counts for branches/committers should be considerably more accurate now. Thanks to John-Mark Gurney for pointing out the problem.
I've also increased the default number you see (10 -> 15) and how many more results you get each time you expand them (50 -> 75).
Git merge commit diffs should be saner now.
Bandaid git merge commit diffs (such as this one) which were exceptioning out. They now incorrectly follow the wrong revision behind for diff generation, but at least it doesn't explode.
In other news, FreshBSD's main server is now sporting a pair of nice SSDs. Yay.
FreshBSD finally has its promised redundancy. Both the web app and search backend are replicated onto the Portfast virtual machine which also acts as the frontend proxy for the site.
Performance will be significantly poorer, and diffs will be unavailable since the server doesn't have enough disk space to store all the repositories, but it's better than nothing.
More downtime today, due to electrical work which was postponed from last week.
Will be sorting out some redundancy and failover soon. Searches on the backup machine will be significantly slower, and disk space limitations will mean no diffs/file views unless they're in cache, but it's better than a Varnish error page.
Apologies for the downtime earlier - my VPN was down, and it took me entirely too long to notice.
DragonFlyBSD indexing has caught up following an odd git problem that had it thinking some old commit was still the tip of HEAD. This also took me entirely too long to notice.
Monitoring. Push notifications. Yes. Good idea.
FreshBSD's backend will be down for a few hours around 8am GMT due to electrical work. Hopefully Varnish can hold the fort for a while.
ZFS-OSX is now OpenZFS on OS X, and has a new Github address. Chase up the indexing location and name. Thanks to "ilove zfs" for the heads-up. So do I :)
Project tags for it and ZFS on Linux have changed to make them more readable. Basic /commit and /project links will automatically redirect, but queries like "q=project:zfsonlinux" will need updating manually.
NetBSD and OpenBSD indexing is currently suspended because the upstream git repositories I'm pulling from aren't receiving updates. They're:
Hopefully they'll be back soon.
(Update: Rivo Nurges's estpak.ee mirrors are back. Thanks :)
Alternative suggestions are very welcome, whether it's other repos or pointers on maintaining my own.
Diff and file views now have <wbr> tags inserted in any runs of non-breaking text longer than 16 characters to assist in wrapping long lines. Inspired by this OpenBSD commit.
Doesn't work in Internet Explorer, but the alternative of using ­ or unicode interferes with clipboard use.
Also, leading spaces in git commit messages are now stripped.
I've improved diff and file rendering, including line numbers and friendlier links to toggle between file diff/view. Please tell me if you notice any regressions.
I've cobbled together basic support for diffing and viewing individual files. Not great, but should tide us over until I get around to a proper rewrite. Thanks to Melissa Elliott for the request.
I've also made the file list a little friendler by limiting its height in the viewport and copying the file/delta count to the top of the table when it's longer.
Reindexing pkgsrc and NetBSD to get rid of duplicates has unfortunately triggered a nasty issue where indexing hangs for dozens of seconds when sending transactions to elasticsearch.
I've bumped up the batch size from 100 to 1,000 to try to amortize the problem across more commits, but they're still filling in slower than usual. Hopefully I'll find a more reasonable solution soon - sorry for any inconvenience.
Update: Turning it off and on again helped. Thus solving the problem forever.
FreshBSD is now running on elasticsearch 1.1.0 on OpenJDK 8. I've also switched the Ruby appserver from Unicorn to Puma. There should be no functional changes.
I've tweaked the site's SSL options to better ensure forward secrecy in most browsers, now the site is running FreeBSD 10 end-to-end and thus running a suitably up-to-date OpenSSL. Let me know if it causes you any problems.
I've expanded indexing on FreeBSD to include both vendor/ and user/ commits. Branch names are currently a bit wrong unfortunately - e.g. "illumos" instead of "vendor/illumos" or "VENDOR_ILLUMOS" - will fix shortly.
The repository syncer has been made more robust - there should be fewer instances of the index getting behind in future.
Added SVN commit links. In the name of avoiding nasty edge-cases, they currently need a whitespace character preceeding them, but will hopefully do for now.
Fixed a problem in Atom rendering that resulted in commit messages being empty.
Illumos. It's called Illumos. Ahem.
Experimental SSL support is now available via https://secure.freshbsd.org. This includes proxying via my VPN behind the scenes, so such requests should never hit cleartext.
It's just a free StartSSL certificate, but it's not like you're sending me your credit card details. Opera 12 rather dislikes it out of the box, but every other browser I've tried seems happy.
The default sort order should be commit, not commit date. Simple enough for Subversion, slightly more work for git. Then people can use broken clocks as much as they like without sitting at the top of the front page for 3 days.
OpenBSD src and ports are behind because the upstream repositories at anoncvs.estpak.ee are offline. If anyone has suggestions for alternatives they'd be very welcome.
Note FreshBSD's CVS support is rudimentary and not really suited to large projects. Much like CVS itself really :P
I've added OpenZFS repositories, including illuminos, SmartOS, ZFS-OSX and ZFS on Linux.
The illuminos repository exposes some parsing bugs in the indexer, so there's a few dozen commits missing there.
FreshBSD is now running on a Xeon L5639 - a six core 2.1GHz 60W Westmere. This is replacing the old Xeon W3520 - four core 2.6GHz 130W Nehalem.
More double-escaping fixes: project links in commits and "More…" links in the sidebar.
I've modified the SVN importer to import newest-first, seeing as they're likey the commits people are most interested in. I've also enabled the multithreaded indexer to repopulate things faster
Yeah. I broke it again.
I'm currently doing a reindexing run after faffing about with elasticsearch 0.90.7 (and failing). I've downgraded to 0.90.5, but it's spewing exceptions despite appearing to work, so this seemed best. First time an ES upgrade hasn't gone smoothly. Bah.
Normal service will be resumed shortly. Sorry for any inconvenience.
More repositories. Git history deduplication. Project/Repository distinction. Indexing statusboard.
No broken URLs. Best Effort.
Orders of magnitude faster handling of commits with large numbers of files.
SSL. SNI works these days, right?
User accounts? Comments?
ETA: $n weeks.
Add GhostBSD, a FreeBSD fork.
Avoid double HTML-escaping query strings in a few places. Most notable in search queries.
Fix man page links; seem to have been broken by some interpreter bug, with regexp capture references not being set when they should. Work around by applying the regexp twice. Now I have three problems.
While I'm here, point FreeBSD man pages to 10-current, send NAS4Free manpage references to FreeBSD, and EdgeBSD to NetBSD.
News item in navbar is now bold if updated in the past 24 hours.
The About page has been updated with images and extra blurb.
Upgraded hosts to FreeBSD 9.2. Upgraded elasticsearch from 0.90.0 to 0.90.5.
Add EdgeBSD src and infrastructure repositories. Both squished into the same project.
Skipped EdgeBSD pkgsrc for now - lot of duplicate commits with this as it is since it includes most of NetBSD's history. Better handling of forks like this is on the todo list.
Added a quick news page in the most lazy way possible.
Fixed double HTML-escaping of git merge messages.
Added support for tracking multiple repositories and repository-types per-project.
Track PC-BSD's new github repositories - skipping pbi for now since it doesn't look particularly useful. Let me know if I'm mistaken.
They're currently all squished into the same project instead of dedicated PB-BSD/PC-BSD Projects/PC-BSD PBI sections. Again let me know if you'd prefer them splitting up.
Belatedly track repository migration to git. Somehow I'd had the local repo mirror set up but neglected to tell the indexing engine about it. Oops.
Very belatedly add to repository syncer so it's actually tracked.
Upgrade to latest Padrino and tire gems.
Fix Atom rendering - should now actually be valid XML.
Disable CSRF prevention tokens, seeing as there's nothing to protect right now.
Add NAS4Free, a fork of FreeNAS.