Coming up in Tor 0.2.6...

Hi, all! We've begun the feature-freeze for Tor 0.2.6, so we know with pretty high probability what new features and changes will be in the next stable release. The ChangeLog and ReleaseNotes files will have a pretty comprehensive overview of what changed since 0.2.5, but I thought I'd like to write up some better descriptions of the major changes.

AF_UNIX socket support for clients and hidden services

This is going to help developers of applications that use Tor make their designs more secure. On Unix and OS X, applications can bind to objects in the file namespace and use them as if they were network connections. Now, Tor can receive connections and make connections to local hidden service applications over this mechanism. That's pretty cool, because it means that integrators can prevent other applications from using the network at all, to ensure that they never make connections that don't go through Tor. Maybe we could use this eventually to turn off networking in Tor Browser entirely on OSX and Unix.

(Sorry, no Windows NamedPipe support yet. That would be cool for a future version, but it'll require some major hacks in our network stack.)

Jake, Andrea, David and I have all worked on the implementation here; it ended up being pretty elegant.

More info: #12585, #11485, #14451.

Faster cpuworker design

We want to use multithreading to spread Tor's cryptography across more cores, but one challenge in doing so has been that our old code for dividing work across multiple cores was written 12 years ago, and designed to accommodate systems that didn't support threading at all. To support no-threading builds, we allowed the CPU workers to be processes instead of threads, and we always communicated with workers over a socket pair.

No-threading systems are no longer relevant, though, and we can totally do better now. We now have a nice simple condition-and-queue based thread pool.

More info: #9682.

Much more testing

In 0.2.5, we introduced a practice of requiring high test coverage for all new or modified code whenever possible. This is continuing to bear fruit for us: We've had fewer weird undiagnosable bugs around this time than previously.

We've also added more external testing: The "make test-stem" target can run the Stem unit tests on Tor to make sure they pass.

Also, a few volunteers have stepped up to contribute to our testing infrastructure. Now test-networks can start in seconds, not minutes. This has already encouraged me to run the network tests more frequently.

Improved circuit scheduler

In older versions of Tor, we used a priority-based scheduler on each connection to decide which circuits to relay traffic for. But connections were scheduled individually rather than globally. That meant that even though the most appropriate circuit got attention on each connection, we might write to connections that did not contain the most important circuit.

(To improve latency, we prefer circuits that have not sent many cells recently.)

Now, thanks to ticket #9262, we have an improved, "global" circuit scheduler. Tor relays now consider all circuits on all connections that are ready to write at once. This is not expected to yield immediate performance improvements, but once the KIST scheduler is deployed, it should lead to lower latency for more connections.

(Kudos go to Andrea for her software engineering skills here: since we've merged it, it has actually '''worked''. That's impressive for a thing of this complexity.)

More info: #9262

Even better support for AutomapHostsOnResolve

We've got a cool feature "AutomapHostsOnResolve feature" for DNSPort users where, when you ask Tor to resolve an address that doesn't have an IP (such as for a hidden service), it can instead give you a temporary IP address (like 127.192.34.56 or fcf0:1234:9312:1111:2222:9803:1bda:20ef), and treat later connections to that address as if they were to the original address.

This feature has had some longstanding warts, though. It didn't always work with IPv6; it interacted weirdly with MapAddress, and it tended to reset itself if you changed the configuration. Worse, it was in one of our messiest functions, so making changes to it was a bit difficult.

In 0.2.6, we've fixed that code (!) I hope, made the feature more robust, cleaned up the function that implements it, and written a bunch of tests for it to help development in the future.

More info: #7555, #12509, #13811, and #12831.

Proposal 227: Include package fingerprints in consensus documents

This one should help the TorBrowser updater: it allows directory authorities to vote on, and include in their consensus documents, a set of versions, URLs, and fingerprints for various software packages. Now every Tor client will have an easy way to learn when a new package is available, and to authenticate the package digest.

More info: Proposal 227, #10395.

More efficient directory requests

Back when I wrote the code for compressing responses on directory requests, I didn't use all the right flags for zlib. By turning off the "Z_FLUSH" flag on our compression objects, I think we should be saving something like 7% of the bandwidth currently used for compressed microdescriptors and descriptors, freeing that bandwidth up for other purposes.

More info: #11787.

Improved memory-based DoS prevention

In 0.2.5, we introduced features to prevent accidental or deliberate out-of-memory conditions from taking down Tor nodes. (The "Sniper Attack" research explains why this is really important.)

In 0.2.6, we identify more kinds of memory usage that could grow without bound, and improve our memory tracker to consider them and clear them as needed.

Also, we add a notion of a "low memory" mode. When a Tor relay is low on memory, it tries to use less memory-heavy versions of its algorithms. (Currently, this only effects zlib compression.)

More info: #13806, #10115, #11792.

Hidden service statistics support

Tor relays can now collect statistics on the amount of traffic that is due to hidden service usage, and the total number of hidden services in existence. (Yes, we have made a major effort to ensure this is done safely, by aggregating results, adding laplace noise, and doing lots and lots of analysis.)

This code is off by default; you can help us learn more about the network by enabling the HiddenServiceStatistics option.

More info: Proposal 238, #13192.

Dropped support for various old things

Panta rhei; the world is ever-changing. Once upon a time, it might have made sense for us to support the C89 C standard, versions of Windows before XP, kernels without threading support, and the WinCE (!) architecture.

Windows XP is a special case. In spite of its lack of security updates, some folks are still on it, and I'm not going to deliberately drop support for it for a while longer. Nevertheless, we can't keep the romance alive forever here: Sooner or later (probably) sooner, we'll wind up accidentally breaking WinXP. When that happens, we're not likely to fix it: running an OS without security patches is the internet equivalent of rejecting vaccines.

More info: #11446, #12439, #13233.

Proposal 232: Pluggable Transport through SOCKS proxy

When Tor is configured to use a pluggable transport, *and* configured to send data through an outgoing SOCKS proxy, it can now tell its transports about that proxy, too. This isn't a feature most people need, but the people who need it really need it.

More info: Proposal 232, #8402.

Longer guard-node lifetimes, single chosen guards

This one is going to be important for security. Earlier research indicates that we need to have each client use fewer guards for their traffic, and keep each guard around for longer, if we want to strengthen their resistance to attackers running a large number of nodes. We've tweaked these values a bit already; current defaults are one entry guard, three guards for directory info, and sixty to ninety days for a guard lifetime. But once we finally merge the #9321 patch, we can increase the guard lifetime to something much higher without unbalancing clients' bandwidth distributions.

More info: #9321 and its child tickets, "Improving Tor's Anonymity by Changing Guard Parameters", and "One Fast Guard For Life".

And miscellaneous features and bugfixes too numerous to mention...

See the ChangeLog for all the exciting details.

Additional acknowledgments:

Thanks go to everybody who has helped with this release, either by writing code, reporting bugs, testing fixes, making comments, diagnosing issues, or just generally helping out. This includes, but is not limited to: Adrien BAK, Andrea Shepard, Anthony G. Basile, Arlo Breault, Arthur Edelstein, Craig Andrews, David Goulet, David Stainton, Francisco Blas Izquierdo Riera (klondike), George Kadianakis, Jens Kubieziel, Dana Koch, Isis Lovecruft, Jacob Appelbaum, Karsten Loesing, Kevin Murray, Magnus Nord, Mansour Moufid, Matthew Finkel, Michael Scherer, Peter Palfrader, Philip Van Hoof, Roger Dingledine, Sebastian Hahn, Mark Smith, Gisle Vanem, Tom van der Woerdt, Tomasz Torcz, Yawning Angel, anonymous, cypherpunks, intrigeri, meejah, rl1987, teor, skruffy, special, wfn, fpxnns, jowr, Device, and chobe.