Backdoored Linux Mint, and the Perils of Checksums
Someone hacked the website of Linux Mint — which, according to Wikipedia’s traffic analysis report is the 3rd most popular desktop Linux distribution after Ubuntu and Fedora — and replaced links to ISO downloads with a backdoored version of the operating system. This blog post explains the situation.
From the post and comments, the key points includes:
- Links to the malicious version of the ISO were added, detected, and removed on the same day, February 20. If you’re already running Linux Mint, this doesn’t affect you — all files installed or updated using the package manager are digitally signed and the signatures are verified.
- Linux Mint 17.3 Cinnamon was the only version that was compromised
- The website was hacked because of a WordPress exploit. Project leader Clement Lefebvre says, “Yes, the breach was made via wordpress. From there they got a www-data shell.”
- The backdoored ISO contains Linux Mint with Tsunami botnet malware running on it.
The blog post includes instructions for checking your ISO files to ensure that they’re valid by comparing MD5 checksums. MD5 checksums!
Besides the fact that the website isn’t available over HTTPS so network attackers could change those MD5 checksums to whatever they want as you load the blog post, MD5 is entirely broken and has been for many years. MD5 should never be relied on for verifying that you have the legitimate version of a file. It would not be difficult for someone to generate a backdoored Linux Mint ISO that has the same MD5 checksum as the legitimate ISO. Likewise, while SHA1 is considerable stronger, it also should not be used for security purposes anymore. Wikipedia’s SHA1 article says: “SHA-1 is no longer considered secure against well-funded opponent.”
It would be great if the Linux Mint project can completely stop relying on MD5 and started using a checksum algorithm that is considered secure today, like SHA256.
But it’s also important to note that comparing the checksum of a file you downloaded with what you see on the website you downloaded it from isn’t secure either, even if you are using SHA256. If a hacker can hack the website to modify the download link, they can modify the checksum at the same time to match their malicious download.
The only solution to this problem is to use public key cryptography. The ISOs should be digitally signed with an OpenPGP secret key, and users should verify the signature using the associated public key. Linux Mint actually does in fact sign releases with a PGP key, but there’s no information on the download page about this, or how to go about verifying the signature.
If you look at the directory structure in the Linux Mint folder on one of the download mirrors, like http://mirrors.kernel.org/linuxmint/stable/17.3/ for example, you’ll see a bunch of ISO files as well as sha256sum.txt and sha256sum.txt.gpg. The sha256sum.txt file includes SHA256 checksums of all of the ISO files, and you can use sha256sum.txt.gpg to verify the signature of that file.
This appears to be the signing key:
pub dsa1024/0FF405B2 2009-04-29 [SC] Key fingerprint = E1A3 8B8F 1446 75D0 60EA 666F 3EE6 7F3D 0FF4 05B2 uid [ unknown] Clement Lefebvre (Linux Mint Package Repository v1) sub elg2048/0F346519 2009-04-29 [E]
Verifying is PGP signatures is more complicated and harder to explain than comparing checksums, but it’s actually secure. It’s the only way to be sure that a Linux installer ISO you download hasn’t been tampered with since the image was built by the developers. Tails is an example of an operating system that does an excellent job at explaining how to verify PGP signatures when you download their ISO.
Legacy comments, imported from previous version of this blog:
I greatly enjoy reading your articles at The Intercept. I thought you might like to know that as of version 18 the Mint team looks to be using 256 keys to verify the iso's now. Better late than never I guess! Happy holidays!
Any speculation on who was behind this hack or why? Why Linux Mint Cinnamon and why that particular malware? Was it a nation state type actor deploying crude malware to get a particular target and trying to look like an ordinary hacker? Or an ordinary hacker targeting that particular OS for some reason?
I think it's unlikely that it was a nation-state actor. It was probably whoever runs the Tsunami botnet.
Also, the whole PGP part is not correct either.
The problem is that those keys aren't verified most of the time. An attacker that has control over the server can just put their own public key in, add it to a few public servers and pose as Clement from Linux Mint.
see these comments (not me, I'm just a research monkey): https://www.reddit.com/r/netsec/comments/46ujhv/linux_mint_isos_were_replaced_with_trojanized/d08c3lm?context=3 https://www.reddit.com/r/netsec/comments/46ujhv/linux_mint_isos_were_replaced_with_trojanized/d08e231
But it is in fact possible to verify the legitimacy of a key, and I do it myself frequently. Without using public key crypto signatures, there is no way at all for this to work at all.
Sometimes I'm connected in the web of trust with the signing key. When I'm not I normally verify that people have posted the fingerprint in multiple locations online (over https if it's available), and I come from multiple IP addresses. I also often find that TOFU works out great with OS distributions. When I download a new ISO and try to verify it's signature, I often find that I already have the signing key from a previous time I downloaded an ISO from the same distro.
It looks like the mint team does not care too much about security.
Is it interesting that the Tails link in this blog about verifying signatures gives me a certificate error on multiple browsers in Windows and Linux? Is this a test?
Never mind. It was OpenDNS blocking the Tails site. Yes it was a test :)
1024-bit DSA OpenPGP keys are not considered secure.
indeed. the linux mint team should address this problem with a set of updated keys, ISO signatures and signed checksums
"MD5 should never be relied on for verifying that you have the legitimate version of a file. It would not be difficult for someone to generate a backdoored Linux Mint ISO that has the same MD5 checksum as the legitimate ISO. "
Sorry, but that's a myth that keeps being brought up. What has been done by academics is: create two files from scratch that have the same MD5 sum, basically a "collision attack". (This is why MD5 should not be used for passwords.)
This has nothing to do with the Linux Mint iso. You would need what is called a "pre-image attack" (https://en.wikipedia.org/wiki/Preimage_attack), where you try to create a file that has the same MD5 sum as an existing sum. This is realistically impossible (I think it was 2^120 or something...trillions of years). Then you also can't just add a few bytes to the iso either, you would need the new file to work as the Linux Mint iso does. So basically a "secondary pre-image attack" So no, this would not have been possible with the Linux Mint security breach.
Still, to me the fact that they use MD5 is just one more argument for their security strategy being weak. They use Wordpress, they have 6-character long passwords containing "mint", they don't use HTTPS, they dont automatically install security updates on Mint etc etc.
I don't think you're correct. It looks like this attack was successfully completed in 2005 with postscript files (https://www.schneier.com/blog/archives/2005/06/more_md5_collis.html) and again in 2008 with X509 certificates (https://www.win.tue.nl/~bdeweger/CollidingCertificates/). In both cases, both pieces of data getting hashes were valid files. But it does look like it takes a lot more work than simply finding a collision.
Again, what they did was generate two completely new files (collision attack).
Your second link explains this in the "application" section. Some quotes: "A successful attack on an existing certificate (or some other data structure such as an executable) requires second preimage resistance of one message: given a pre-specified value and its hash, it must be practically infeasible to find another value with the same hash. As far as we are aware, the results announced in  do not imply that second preimages are essentially easier to find than they should, namely with an effort proportional to 2^n for an n-bit hash function" and "We are not aware yet of real life practical implications of our results."
Your first link: see the comments, e.g. https://www.schneier.com/blog/archives/2005/06/more_md5_collis.html#c6016 "All that the researchers are demonstrating here is that it is possible to create two functionally different yet valid files with the same signature in a very short amount of time.
This and the other referenced attacks all take advantage of the length extension property of hash functions after finding two pieces of data with colliding hash values. Effectively, all they demonstrate is the possibility of wrapping identical content around these two separate sets of colliding bits.
While this is important, this is not nearly the same thing as demonstrating that it is possible to create two ARBITRARY documents with the same hash value or that it is possible to generate a NEW document with the same hash value as an EXISTING document."
So, what you could do, for instance, is to fake you own public key by generating two with the same MD5. Or if the Linux Mint founder was bad, he could create two iso images with the same MD5, one with a backdoor. But it does not work if you already have an existing iso and MD5 (this would be a pre-image attack).
If you say that Clem, if evil, could make two files with same md5, one backdoored and one not. Why wouldn't anybody be able to do that then, whether a file with an md5 already existed or not? What is the difference?
"Or if the Linux Mint founder was bad, he could create two iso images with the same MD5, one with a backdoor."
or Shneier's statement :
"“All that the researchers are demonstrating here is that it is possible to create two functionally different yet valid files with the same signature in a very short amount of time."
Seems pretty straightforward to me so pardon my ignorance. I'm trying to understand how these statements back your claims that such a thing is not possible?
I think I understand now, you are saying the only way is to modify both files to get their md5 to matchup... like with the collision generator.
So if true, we would still then have to hope that, theoretically, a non backdoored iso was not uploaded for a while on the site before changed and that mint actively monitors the hashes as well as the iso. I wonder how long they were compromised for. And Still kind of crazy regardless they don't use https and that they even have a md5 hash in the first place, and not just sha256 and a sig, like most other distros.