Tor at the Reproducible Builds Workshop in Athens 2015

by gk | December 8, 2015

Being able to build Tor Browser several times in a row and getting exactly the same result each time has been an important feature for a while now. It provides a direct link between the source code we provide and the binary that Tor users are downloading and using to surf the web. This offers a number of benefits to all parties involved:

  • Users can verify that they really got the binary they were supposed to get
  • Pressure on developers to provide a bullet-proof build and signing setup is reduced
  • Incentives to pressure release engineers into inserting backdoors into the code are reduced

From December 1-3, 2015 we had the opportunity to discuss these and other topics around reproducible builds with members of different projects. Thanks to the Linux Foundation, the Open Technology Fund and Google, developers from Debian, FreeBSD, NetBSD, Google, the Guardian Project, Coreboot and Tor (to name just a few) were able to attend. The workshop started with exchanging experiences with already existing systems (like Gitian, which we use for Tor Browser). During the three days of the meeting, work went on to explore together future directions for advocacy, commonly used tools, infrastructure and documentation.

We were especially pleased to see the fruitful collaboration on the operating systems level. While it is good to have a reproducible Tor Browser, the security guarantees that it provides are even stronger if the operating systems and the toolchains used to build it can be created reproducibly as well. Moreover, all participants agreed that non-reproducibility is essentially a defect that needs to be fixed. This allows us to treat workarounds (like using libfaketime to avoid timestamp differences in binaries) as mere band-aids and instead focus on addressing the root causes of non-determinism directly upstream.

Thanks to Allen Gunn and the Aspiration team for the excellent facilitation and all participants for the productive and exciting time. See all of you at the next workshop!

Comments

Please note that the comment area below has been archived.

Anonymous (not verified) said:

December 08, 2015

Permalink

Could Reproducible Builds offer protection against a rogue certificate being inserted into the TB running on an ordinary user's computer?

Users can verify that they really got the binary they were supposed to get
Pressure on developers to provide a bullet-proof build and signing setup is reduced
Incentives to pressure release engineers into inserting backdoors into the code are reduced

i do not understand how a rogue certificate could be 'inserted' implemented.
do you mean that someone is corrupted in the team (s) ?
are you speaking about compromised laptop/tablet/computer like lenovo in the past ?
are you asking to the tor team (reproducible builds workshop) if a special feature in the tb should prohibit the usage of a rogue certificate ?

> are you speaking about compromised laptop/tablet/computer like lenovo in the past?

Yes, "superfish" style rogue cert. See also Kazakhstan.

> Pressure on developers to provide a bullet-proof build and signing setup is reduced

Please don't reduce that pressure just yet, since few Tor users have the ability to do a build! For example, I don't. Hard enough getting ordinary users to verify the signature of the iso image (Tails) or tarball (TBB).

it is up to you !
You must learn :
how to verify&install properly the operating system-software (easy)
how to choose the right computer-hardware for you (easy)
etc.

a patch, a removal tool, some instructions were added after the "superfish" style rogue cert problem.

i read that linus torwald is working about an universal operating system ; maybe this one will contain a special feature protecting against these false or hidden certs.
you could make a request to him !

(unfortunately; i am not informed about 'See also Kazakhstan' and you have not let a https link.)

The reproducible build workshop is a success.

Thanks, but I don't think any of that addresses the question I tried to ask. Which I now see was not sufficiently clear.

> The reproducible build workshop is a success.

Agreed.

"Please don't reduce that pressure just yet, since few Tor users have the ability to do a build! For example, I don't. Hard enough getting ordinary users to verify the signature of the iso image (Tails) or tarball (TBB)."

I don't think the post is talking about end-users so much as users like:
- Linux distro package maintainers, who build a binary package from source and distribute it to the distribution's (e.g. Debain) users. They can verify that the binary matches the one that Tor Project says it should before signing and shipping their package.

- Deployment operations people, who do basically the same thing but within a specific company or organization. This would mean something like doing a reproducible build and verifying it before putting it on a shared NFS /usr filesystem, or the Windows-equivalent way of sharing preinstalled software within a network.

- IT security people, who are responsible for whitelisting specific binaries for execution on computers within the organization. If the binary doesn't match Tor Project's, the OS will refuse to execute it.

- Users of source-based distributions like Gentoo, where the reproducible build and binary comparisons can happen automatically behind the scenes.

- Users of package managers like apt-get, yum, pacman and many more. The package manager can extract the binary from the package and verify it against the Tor Project's hash before installing the package. This won't tell the end-user that the binary is really based on the source they think it is, but it will tell them that the source (and resulting binary) was not modified by the package maintainer or package build server, or in transit. Windows users could do the same thing manually, but it is less useful because Windows binaries are distributed directly from Tor Project while Unix packages are distributed from the distribution's developers.

- End users of any OS would be able to verify their downloaded binary's hash against hashes from other users of the same OS who have actually done the reproducible build. All of those users would have to have been compromised, as opposed to (currently) just the Tor Project developer who PGP signs the binaries.

Some more than others, but any of these methods could be helpful in detecting a compromised build system or machine-code-level backdoors somewhere along the path.

Well, we still try to have the signing setup and our build machines as secure as possible. But these are no single point of failures anymore as they were before thanks to our reproducible builds. Thus, the pressure to not mess up these things is not as high anymore as it has been which is what I meant.

> if a special feature in the tb should prohibit the usage of a rogue certificate?

Not sure if that appears to be technically feasible, but anything which

o warns ordinary Tor users that their TB stores a cert which could be used to MITM their connections via Tor to https websites

o removes said cert (if possible)

would be useful in any software provided by Tor Project or an allied project such as Tails Project.

If your operating system (or anything with enough privileges to change something for a running TB instance) is compromised/malicious, you can't trust it. Doesn't matter if you have a good binary or not.

Do Linux From Scratch once and you'll be appalled by how much unsigned code must be built and installed just to get a functioning system. GNU and kernel.org require signatures on releases fortunately, but projects like systemd, "file" from NetBSD, and wpa_supplicant have never heard of PGP, let alone deterministic builds. And FYI, your distro's packager doesn't have magical "unsigned code verification" abilities.

Both OP's and your comments are true, and with that being said, the only Right Way to do deterministic builds is to build

    everything

deterministically. If we want that, we'll likely have to form a complete deterministic build distro, with source auditing and packager keys a la Debian-style.

That's why Linux Foundation, FreeBSD, NetBSD, and Coreboot were there (presumably). There's no reason reproducible builds couldn't extend all the way down to the kernel, AFAIK. It's just a matter of making it reality. (Yes, the firmware and hardware are still a problem, but that's a whole different game.)

> There's no reason reproducible builds couldn't extend all the way down to the kernel, AFAIK.

Non-expert, but I too worry about the kernel. Tails is based on Debian which uses the Linux kernel, and Linus is famous for saying that he treats security flaws just like any other bug, which attitude is not only wrongheaded and actually endangers Linux users. His mind must be changed on this point, but this project may be as quixotic as converting James Comey into an ardent onion lover.

To put linus's stance on bugs back into context, he says that he treats security bugs like other bugs because all bugs can become security bugs. Take the glibc gethostbyname() function with an off-by-one buffer overflow. Everyone assumed you couldn't fit a malicious payload into one byte, so it was assumed noncritical. It later turned into a critical remote code execution when given a specially crafted hostname (IIRC; search for the paper for details).

Nnevertheless, he also has stated that he cares more about performance than security. He also doesn't like to break userspace, which is often necessary when introducing security features.

That's how the Grsec patch came to be. Spender and Linus simply couldn't agree on how to mainline the changes. IMO there will be more interest in deterministically building Grsec than mainline, for the reasons I mentioned.

> Could Reproducible Builds offer protection against a rogue certificate being inserted into the TB running on an ordinary user's computer?

I now see my question was not clear. After some thought, I decided I probably should have asked a different question:

As a non-coder, with no experience in building an OS from source code, who cares about security and who regularly uses TB and Tails, can I try to learn to use Reproducible Builds in some way to enhance my security, for example against rogue certificates?

Or perhaps better put: I think I have some idea why Reproducible Builds is awesome for developers, but can the Awesomeness explainers post a blog explaining how ordinary users can try to leverage it on a generic PC or laptop to do something which might improve their security against sophisticated GCHQ/Kazakhstan style "equipment interference" (cyberattacks)?

Reproducible Build is an integrity test.
Beta testers were invited.

° to do something which might improve their security against sophisticated GCHQ/Kazakhstan style "equipment interference" (cyberattacks)
° https://www.gov.uk/government/uploads/system/uploads/attachment_data/fi…
° few years ago , someone proposed a mesh network as counter measure. Later, wifi, bluetooth, laser, ir were tried : it seems that it offers better protection than wired cable for lan and short distance.

° can I try to learn to use Reproducible Builds in some way to enhance my security, for example against rogue certificates ?
° you must choose/verify carefully your hardware and software
° you can build it from binary.
° you could join the dev by gnupgp_e-mail
° you could ask to the dev a personalized version according on your threat model.

° how ordinary users can try to leverage it on a generic PC or laptop ?
° it is the chain of trust.
° vm _ installing the o.s only for this purpose on a smart card:usb _ with tail , on the cloud , vps ,
° an encrypted electronic card pki could be added to your computer enhancing seriously your private surfing:communication.
https://en.wikipedia.org/wiki/Hardware_security_module
an initiative from a team - kit hardware - presented a prototype few months ago but i forgot the link , sorry.
° implementing a special cert/pin/lines of code allowing only few sites or few correspondents restricting access and a perfect setting of the firewall maybe, a server serving relay.

hope that help

° you can build it from binary.
° you could join the dev by gnupgp_e-mail
° you could ask to the dev a personalized version according on your threat model.

Clear & Neat & easy.

> you can build it from binary.

Over my head.

> you could join the Dev by gnupgp_e-mail

Don't know how to do this, and fear it might be very hard to use email lists anonymously. NSA has had decades to figure out how to traduce them for their own evil purposes, eh?

> you could ask to the Dev a personalized version according on your threat model.

Seriously? Even if I could, I don't think it would be proper for anyone but a very high value Tor user indeed (e.g. Greenwald) to demand a personalized version. And even if it were proper, your proposal seems contrary to the goal of achieving anonymity.

Perhaps it's just me, but the question is still a little unclear. Firefox (and Tor Browser) do embed root certs into their packages, so TBB should be unaffected by OS-level rogue certificates (like Superfish; see "Windows Certificate Store" and OpenSSL documentation). However, reproducible builds really only apply to code, and won't protect data -- like certs -- any more than PGP signatures will.

I'm not exactly sure what you mean by "equipment interface cyberattacks". If you mean things like rootkit/bootkit/firmware/hardware-level backdoors, then no, reproducible builds will not help.

This blog post wasn't really meant to explain the scope of reproducible builds and what they mean in the grand scheme of things. See these earlier blog posts for background information:
https://vbdvexcmqi.oedi.net/blog/deterministic-builds-part-one-cyberwar…
https://vbdvexcmqi.oedi.net/blog/deterministic-builds-part-two-technica…
Tagged: https://vbdvexcmqi.oedi.net/category/tags/deterministic-builds

> Perhaps it's just me, but the question is still a little unclear.

To be clear, I am not sure exactly what I am asking, because I lack sufficient knowledge to know what to ask! So please bear with me a little longer.

> Firefox (and Tor Browser) do embed root certs into their packages, so TBB should be unaffected by OS-level rogue certificates (like Superfish; see "Windows Certificate Store" and OpenSSL documentation).

I think you are saying that the root certs included in current genuine TB offer protection against the initial connection from my computer to Tor entry node being MITM'd, but a rogue or compromised exit node could still MITM the connection from exit node to some https website. But in the latter case, the enemy would not be able to easily tie the exit stream to my real IP, we hope. Do I understand your point correctly?

> I'm not exactly sure what you mean by "equipment interface cyberattacks". If you mean things like rootkit/bootkit/firmware/hardware-level backdoors

Yes.

Here's my third try to ask a sensible question: if we speculate (on the basis of Snowden leaked documents and more recently leaked documents) what attacks we might reasonably guess that our more sophisticated enemies (FVEY, Chinese/Russian/Iranian government cyberspies) might try to use against us, could Reproducible Builds possibly be helpful for a non-developer such as journalist who has good reason to fear being attacked by sophisticated attackers?

< Reproducible Builds > reproducible build is a beta step done before to be released.
it is more a professional dev entertainment/training/test team than an original hardened tor version world shield universal against bad guys ....
< helpful for a non-developer such as journalist > as user, it is depending on more where you are living than you are doing.
< has good reason to fear being attacked by sophisticated attackers > it is about random data, they recollect blindly ; then they choose the target, sometimes ten years after e.g.
< (FVEY, Chinese/Russian/Iranian government cyberspies) > it is their job. it helps tor to be better.
5eyes are more cruel than Chinese/Russian/Iranian government,
iranian government is far less dangerous than eu countries. russian government is a democratic and modern state. chinese government is between a closed proprietary internet and an open lan which the state is the admin.
< could TOR possibly be helpful for a non-developer such as journalist who has good reason to fear being attacked by sophisticated attackers?>
tor is proposed at the download page after to be passed by a quality test (reproducible built).
As all the users ; we are spied and protected against such 'enemies' using tor.

> 5eyes are more cruel than Chinese/Russian/Iranian government,
iranian government is far less dangerous than eu countries. russian government is a democratic and modern state. chinese government is between a closed proprietary internet and an open lan which the state is the admin

I suspect at least two of those statements may represent sarcasm, but I nonetheless leap at the opportunity to point out that both Russia and Iran were named as Enemies of the Internet by RSF in 2012:

https://en.rsf.org/russia-russia-12-03-2012,42075.html

https://en.rsf.org/iran-iran-12-03-2012,42070.html

Thanks to Snowden's 2013 leaks, we now know why the current US/UK governments should have been at the top of that list, but the current governments of Russia and Iran are hardly angelic.

It's all of them (authoritarian governments) against all of us (the People).

$ may represent sarcasm ? no, it is the truth on a blog , not a tweet on cellphone.
$ Russia and Iran are hardly angelic ? An angelic place can be accessed from a decent death ; people has not obtained it yet (euthanasia) and it is also included in the word 'privacy'.
$ RSF give us news about unclear/unknown people who are not at all angels ; tor and the net is opened for this purpose.
$ these countries are not authoritarian governments.
$ the term 'people' does not exist in most european countries (eu).

@ improve their security against sophisticated GCHQ/Kazakhstan style "equipment interference" (cyberattacks)

The better way is to attack them , they are acting as 'good angels' protecting the economy against awful devils.

unfortunately, their targets are not outside of the economy and even not dark or hidden.

for example, a crime is not punished if you do not pay it , not judged if it is not transmitted, not known if the few persons who are involved did not speak about.
(so money allow almost all ; more in the usa than anywhere else)
for example, slavery, extortion,torture, pornography, missing persons are the same subject and not included in the mind, usage, education, mental health, social class as a dirty thing so it is not punished.
(if it was not on the internet/deepweb, it should be in the street managed by the mayors of every town in every country of course)

a victim is often chosen before, a long time before, by official, authorities, elected people, jet-set, artist, to become it.
for example in india ;some strange things happened to a couple of us tourist and even to the resident too.
officialy, a clash, but the reality is that eu enterprises have payed some criminals gang to do it.(german in this case)

now that you gave these information to the GCHQ , you are the target ; they are in front of themselves and must choose, filter, decide, make the right option and correlate with the fact, the persons and the hazards ... you are not anymore a devil, you are the angel ... with them.
because it is a program without feeling and an argument with you launches another research in another direction.
because it is legitimated by their own line of code, it is built, structured in a such manner that you can answer.
an attack is a response from an aggression.

prove to them that this employe(us) of nato or another free & clean organizations was working for an italian maffia (nato) like the mayor of montreal ; operation planified by the ministry of defense of the usa for avoiding/erasing the independence, the autonomy,
prove to them that thatcher has humiliated 5 millions of uk citizens putting them in one the worst poverty during a long decade.(someone said few hours after her death - one person - that she was honorable ; a very big opposite opinion came from the people suddenly _ since this negative reaction of the people, no one dares speaking about her as a vip or as a good person.)

The reality that you can prove is stronger than all the lies.

Everyone can work better if the facts are true (the computers too) ; it could frozen a lot of thing even the GCHQ.

Take the interference, appropriate it for yourself as a freedom voice.

so, what about the Kazakhstan ?

Anonymous (not verified) said:

December 10, 2015

Permalink

The Tor reproducible build project is wonderful! High praise and gratitude is due to all the Tor devs and contributors. Maybe someday we will get all software developers to follow the lead of the Tor project and implement the reproducible build concept.

Anonymous (not verified) said:

December 10, 2015

Permalink

با درود قبلا فیلتر شکن تور را دانلود کردم .اما بعد از مدتی غیر فعال شد الان برای اینکه دوباره فعالش کنم راحتترین راه حل چیه.....ممنون

Anonymous (not verified) said:

December 11, 2015

Permalink

Great! And what did you guys come up with? I.e. where (when?) can we get videos, slide decks, papers, etc. from the event?