Removing End-Of-Life Relays from the Network

by dgoulet | October 8, 2019

Today, the Tor network is composed of more than 6000 relays. These relays are running Tor software versions that go all the way back to the 0.2.4.x series, which was released on December 10th 2013. Other cutting edge relays are running our latest code in nightly builds and alpha releases. These relay versions represent roughly 5 years of Tor development. There are a total of 85 different Tor versions, from alpha to stable, in use by relays today.

Currently the Tor Network Team maintains 5 Tor version series:

  • 0.2.9.x (LTS)
  • 0.3.5.x (LTS)
  • 0.4.0.x
  • 0.4.1.x
  • 0.4.2.x (Stable on Dec 15th, 2019)

Maintaining these versions means that the Network Team intends to fix major stability issues, security vulnerabilities, and portability regressions. We may also fix smaller bugs that significantly impact user experience.

Impact of EOL Relays

Unfortunately, End-Of-Life relays have some negative impacts on the network. Any relay in the network that runs an obsolete version puts network stability and security at risk. Outdated relays make it harder for us to roll out important fixes. And they can also make it harder to roll out some new features.

One example is the Denial of Service defenses that we rolled out at the start of 2018 as an emergency reaction to a large scale attack on the network. Unfortunately, that defense is only available for relays running supported versions.

Tor's circuit padding defense is in a similar position. Tor 0.4.1.x introduced circuit padding, to make it harder for network observers to recognize client onion service requests. The feature only works for circuits that have a 0.4.1.x (or later) relay as their middle hop.

Because of a bug in the 0.3.2.x series, some out of date relays are badly affecting HSv3 reachability.  These relays cause higher latency than normal for users. And they add overall network load, from all the Tor clients retrying failed HSv3 connections.

Removal of EOL relays

Because of the impacts of keeping unmaintained relays on the network, we've decided to remove End-Of-Life relays from the network.

In the past weeks, we've taken steps to contact every relay operator with a valid ContactInfo field to ask them to upgrade to the latest stable release. The Tor relay community was informed via the tor-relays mailing list on September 3rd 2019 of this upcoming change.

The End-Of-Life relays in the network currently make up just over 12% of the total bandwidth, or around 750 relays. Out of these, only 62 are Exit relays accounting for only 1.68% of the total Exit traffic. We expect a minor impact on the size of the network, and a small drop in the Metrics graph.

Specifically, today we're starting the process of having the 9 directory authorities refuse End-Of-Life relays. The affected series are:

  • 0.2.8.x (278)  [w: 0.55%]
  • 0.3.0.x (20)    [w: 0.04%]
  • 0.3.1.x (34)    [w: 0.15%]
  • 0.3.2.x (136)  [w: 1.37%]
  • 0.3.3.x (81)    [w: 0.84%]
  • 0.3.4.x (205)  [w: 9.12%]

We expect our next Tor stable release (around November 2019) will contain a software change that will reject End-Of-Life relays by default. Until then, we will reject around 800 obsolete relays using their fingerprints.

Obsolete bridges won't be rejected yet; they will be rejected later in 2019, when we deploy the Tor software change.

Upgrading Your End-of-Life Relay

If you see this line in your logs, then your relay was part of the EOL list:

Fingerprint is marked rejected -- if you think this is a mistake please set a valid email address in ContactInfo and send an email to bad-relays@lists.torproject.org mentioning your fingerprint(s)?

In order to re-join the network, please upgrade to a still supported version and ideally our latest stable (currently 0.4.1.6).

After you upgrade your relay, there are two ways to get back on the Tor network:

  • If you want to keep your relay keys, email the bad relay list, and ask them to stop rejecting your relay fingerprint.
  • Or change your relay keys:
    1. Go to your Tor data directory (DataDirectory).
    2. Then delete the keys/ directory (Linux: "rm -rf keys/")
    3. Start Tor again.

You can find Tor packages for your distro / OS here.

Debian and Ubuntu users: Please note that those instructions use our deb.torproject.org repository. Using our package repositories (or Debian Backports) will ensure that you always get the latest Tor stable, which will help ensure that the network is as secure and as reliable as can be.

If you need any help, please feel free to email the Network Health team at network-health@lists.torproject.org.

If you don't yet run a relay but want to, our guide can help get you set up.

Thank You, Relay Operators

Support from relay operators is essential to keep the network healthy. Operators must keep their relays and machines up to date. Relays are the backbone of all software that relies on Tor, and each operator helps immensely in providing people with privacy and freedom online around the world. We cannot thank them enough.

Again and forever, a huge thank you to all relay operators out there that volunteer their time, resources, and effort to the Tor network. Your contributions are indispensable and everlasting.

The Tor Project

Comments

Please note that the comment area below has been archived.

Anonymous (not verified) said:

October 08, 2019

Permalink

Hurraaaay! Still not <3 police, but loving this kind of network police!

Seriously, this is a great and much awaited move from the Network team.

However I have a question: you have mentioned some possible (and very limited) drawbacks due to the removal of EOL relays. A part this, I am wondering if you have hypothesized the possible benefits the network could gain, or if you will just sit in front of your monitors and observe the situation, as if it was an experiment

(something like: "OK, this is a test to get experience for the future. In this way, when we will decide to kick 0.2.8 relays off the network, we will already be prepared, we will already know what kind of problems can emerge and we will already know how to manage the situation").

Second question: how many relay operators answered to your call and the mail you sent them?

Thanks.

It is probably pretty simple logic. For security for everyone the Version Firewall will protect everyone. The only downside is that tor authority directories could become malicious but that is the same situation how it is established using Directory Authorities anyway so why not act out some police work against bad actors since the equation stays the same from the top.

You always have to "hypothesized the possible benefits" when maintaining the plumbing system. It's a big world and the plumbing needs lots of TLC. Nothing is perfect. But improvement is the only outcome that is judged. If no increase of logical pathways than the system breaks.

Hope it helps!

Yes definitely we've analysed the drawbacks of such a removal. As stated in the post, the benefits are in terms of security and stability mostly against loosing network capacity.

The more features we roll out, especially security ones, the more the network relays gets partitioned in cluster of relays supporting X or/and Y or/and Z.

Then, client picks those relays based on what they support and depending on what the client wants to do. For instance, the Tor circuit padding from client to Guard node, a client can not stop picking the other Guard nodes because they don't support padding. That would leak information about the client but also narrow down to a very small number of relays this client could pick and become an easier target. So to provide this new security feature to be used all the time, we need to wait that the entire network upgrade to at least a version supporting it. And that can be a long time...y

In other words, the more clusters of relays we have, the less the network is agile to provide new safety measures to everyone. And the more security issues linger in the network due to out dated and unmaintained versions.

And, this is not really an experiment to prepare us for the future although it was a learning experience for sure. We do reject relays regularly from the network to protect against attackers. Sometimes, it is large numbers.

For you second question, that is difficult to say without counting all the emails but I would say a reasonable number. Ideal world, they would all respond but some do upgrade without bothering to reply :).

Anonymous (not verified) said:

October 10, 2019

Permalink

Dear both,
thanks for your insightful and detailed answers. I am fascinated by plumbing so, if you do not mind, I have a few more things I'd like to ask.

1) Was there a kind of "salient" and specific reason (like an attack to the network or something) that led the Network team to kick out EOL relays? Or it was more like: "Well, the topology of Tor is just becoming too partitioned and fragmented. These are the drawbacks, these are the benefits. Let's do this"? Or both?

2) On the organizational side, I was wondering... Why do you think this happens? It is difficult to provide an answer for me. I am not a coder or an hacker, but I know that anonymity and privacy rely on security. Hence, as bridge operator, I am very keen in checking the security of my nodes (compatibly with my technical skills which are not high). Actually, there is not much to do: once you set up unattended-upgrades and some tool for log audit a good part of the work is already done. So why? Is it just because people quickly setup a VPS with Debian (without using deb.torproject or backports) and forget about it? Is it negligence? Is it intentional? Or do you have other ideas?

3) Directory Servers are charged with few but important technical functions (i.e. network topology and node state) that make Tor a distributed but semi-centralized network. This is no secret and the reasons for this choice are written in the Tor design paper. I am wondering... what else they do? Are they appointed with other technical functions? How did their design evolved since 2003/4? The real question is: is there one or more paper/link about this topic?

Thanks

0.2.9.x series is EOL in January 2020 so another 3 months. We'll simply roll that one up the end since expect maybe one other release, maybe two if something happens.

However, the 0.3.5.x LTS series, we will be rejecting every version up to 0.3.5.7 due to issues and alpha code.

Anonymous (not verified) said:

October 09, 2019

Permalink

I do not have an email account. How can I report a possibly malfunctioning relay?

When the guard node assigned to your debian-tor instance (used for accessing the Debian onions), how can edit the torrc (which file exactly?) to choose a new guard?

DontIgnoreBugReports (not verified) said:

October 09, 2019

Permalink

During the past week, despite repeated attempts, I have been unable to report bugs to Tails Project using Whisperback (which emails their onion mail server). Is it possible that the cause is owing to an EOL server not being properly maintained by Tails Project?

I have seen two types of error messages in multiple attempts: a TLS failure (I first guessed an expired certificate) and "Pysocks does not support IPv6". In the first case in Onion Circuits I could see a failed attempt to contact an onion. The the second case it appears that no circuits are being built.

I hope someone with a secure comms link with Tails Project will report this to them. TIA.

Anonymous (not verified) said:

October 10, 2019

Permalink

I didn't know the Tor Project gave support to relays with old builds of the Tor software.
I'm glad this decision was made, supporting many old builds only makes it harder for the developers.

Anonymous (not verified) said:

October 15, 2019

Permalink

Further questions that the Network Health team should be asking:

  1. Why are those top most prevalent Obsolete or Unrecommended versions still being used? Repository update lag? Operator or packager neglect? Discontinued devices or support? (as well as hardware such as Anonabox) Malicious or researcher farms to exploit the network by way of vulnerabilities in old versions? Something else?
  2. What other ways can we encourage and facilitate their update and penalise dawdling?

Anonymous (not verified) said:

October 17, 2019

Permalink

>Fingerprint is marked rejected -- if you think this is a mistake please set a valid email address in ContactInfo and send an email to bad-relays@lists.torproject.org mentioning your >fingerprint(s)?
you still do not have onions email. (!)