Ignore the actual time count. Answer me this: When would you feel more secure, knowing that your favorite distribution has been attacked and is now under scrutiny, or thinking — or rather not thinking — that your Linux may have been a target of a dedicated attack? In other words, if there’s a security problem, and you’re not aware of it, are you not in fact a happier camper?
These two philosophical questions relate directly to the Linux Mint website breach in February, which resulted in a compromised ISO being served from the official domain. Since, the problem has been addressed, various security measure put in place to improve and ensure the robustness of the Mint domain, and normal operation resumed. But … is that good enough?
What is seen cannot be unseen
The Linux Mint site attack is a very interesting one. Not because of the technical details surrounding the vulnerability, the exploit, the subsequent hacking attempts and the forum database exposure, and the cleanup activity. That can be interesting on a particularly uneventful day. What makes this event fascinating is the way the community reacted to this whole issue.
I told you so is probably the best phrase that summarizes the collective indignation of the angry, hungry-for-blood crowds raising their pitchforks against Mint. After all, the difference between Ubuntu and Mint default security levels has always been a hot topic, and the contention also brought the question of quality and professionalism into play. But the debate went as far as questioning the validity of Linux Mint as an independent distro, and it touched lots of other tangential topics, none of which helped resolve the security problem or allay fear with the confused users.
Now that the fires of doom and justice have settled, there are two very clear phenomena rising from the ashes, smoke, brimstone, and an odd snippet of GPL-ed code. The first one is, the Mint team has worked toward improving the security of their Web-facing services, a worthy lesson that will probably affect every single CMS user, be it WordPress or any other, at some point in time. But the more intriguing one is the sense of insecurity surrounding Mint now. People are whispering:
Is Linux Mint safe for use?
Let us check!
Now that you are AWARE, you will start doubting and questioning Linux Mint. And we’re not talking only about the official site and downloads and what methods you ought to use to verify the integrity of distro images. We’re also not limiting ourselves to the website, the forum, the use of TLS and other methods to secure the Web traffic in transit, or how to minimize the exposure vector of a WordPress site. No. That’s trivial.
People are wondering if Linux Mint is a safe operating system to login into email, use for banking, or best yet, actually run updates! After all, the distro will connect to Linux Mint mirrors and pull packages, and some of these could potentially be compromised.
Practically, nothing has changed between how the repo management and security mechanisms work between February 19, the day before the breach, and now, but the perception of user has, and that’s all that matters. Many users also lack knowledge and understanding into the finer details of these mechanisms, and ignorance leads to fear, and fear breeds worst case scenarios. All because you are no longer taking distro security for granted. But can you actually do something about it, so you can have more than just a hunch? How about an educated guess? Or a proper examination of what you’re dealing with?
How to verify distro contents
All right. Let’s say you have Linux Mint 17.3 installed. Let’s say this distro has been on your system for a good few months now, and you have definitely not installed it using the compromised image. Till now, you have been absolutely certain that it is safe, secure and good for you, and that Linux is not as exposed to the online malware games as other platforms.
However, you do want to update your system, and that means changing its state — downloading packages from Linux Mint repositories, as well as other sources, and since the breach, you do not really know for sure that they have not been touched or changed. Mint folks assure us that the repos have not been compromised in any way, and there’s even a nice little thread on the official forums, but is there a way to do a self-test?
In a nutshell, what we want to do is download packages from Mint servers, then check their integrity using checksum tools and compar them to the online information. This is somewhat like asking the Internet if the information you have is true. And overall, on average, since it is highly unlikely that every server on the Web has been hacked and is serving bogus data at the same time, you should be able to converge to an accurate truth of how things really are. In other words, if you download package updates, check their sum and then compare them to online sources, one or more, official, mirrors, affiliated, or alike, then you should be fine.
We will use the commandline apt interface to do our testing. To just download packages, we need the -d flag, and to upgrade individual programs, we want the –only-upgrade flag. This will allow us to handpick packages and check them. For instance, Firefox (in bold what you have to type):
sudo apt-get install -d --only-upgrade firefox Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: fonts-lyx Recommended packages: xul-ext-ubufox The following packages will be upgraded: firefox 1 upgraded, 0 newly installed, 0 to remove and 159 not upgraded. Need to get 42.6 MB of archives. After this operation, 861 kB disk space will be freed. Get:1 http://www.mirrorservice.org/sites/packages.linuxmint.com/packages/ rosa/upstream firefox amd64 45.0+linuxmint1+rosa [42.6 MB] Fetched 42.6 MB in 10s (4,203 kB/s) Download complete and in download only mode
This command will download a new version of Firefox and place the DEB package into the local cache, which goes under /var/cache/apt/archives. If you check the directory listing inside this folder, you will most likely see several Firefox packages, like version 43, 45 and more.
-rw-r--r-- 1 root root 42589560 Mar 11 10:04 firefox_45.0+linuxmint1+rosa_amd64.deb
The next step is to calculate the file hash using one of the available programs. Since md5 and sha1sum are considered vulnerable to collision attacks, you should use SHA-2 variants, like for instance sha256sum. But for the sake of this exercise, we will also try sha1sum, just to go through the motions.
sha1sum firefox_45.0+linuxmint1+rosa_amd64.deb cb1f09d6cf0064030aab3308358888091293aaf6 firefox_45.0+linuxmint1+rosa_amd64.deb
Now, the final step is to search online for “firefox-whatever.deb” sha1sum to get the desired information. In this particular case, I was only able to find the sums on a Japanese domain, which listed the necessary sums, and they were identical for this Debian file.
This one example is not too indicative, so we should try some more.
sudo apt-get install -d --only-upgrade libgnutls26 Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libgnutls-openssl27 libgnutls26:i386 Suggested packages: gnutls-bin gnutls-bin:i386 The following packages will be upgraded: libgnutls-openssl27 libgnutls26 libgnutls26:i386 3 upgraded, 0 newly installed, 0 to remove and 157 not upgraded. Need to get 787 kB of archives. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n]
Libgnutls26 is another package, almost randomly chosen from a long list of available updates. BTW, if you’re wondering what updates your system may have, you can run apt-get dist-upgrade. This will give you a long list of upgradeable packages. You can choose any one to test. However, if you have the right expertise, you can narrow down your testing to programs that could potentially be used to glean or intercept user information.
Again, we will do a hash check, using the robust sha256sum program:
-rw-r--r-- 1 root root 18364 Feb 24 17:14 libgnutls-openssl27_2.12.23-12ubuntu2.5_amd64.deb -rw-r--r-- 1 root root 393294 Feb 24 17:14 libgnutls26_2.12.23-12ubuntu2.5_amd64.deb -rw-r--r-- 1 root root 375044 Feb 24 17:15 libgnutls26_2.12.23-12ubuntu2.5_i386.deb
And the sum:
sha256sum libgnutls26_2.12.23-12ubuntu2.5_amd64.deb 75485b03d5548bdd135112831121c0625ccc2fa5571a0cb748134180f84de11f libgnutls26_2.12.23-12ubuntu2.5_amd64.deb
This particular package actually comes from upstream, from Ubuntu. You will find detailed information on the Ubuntu site. Once again, if you compare the two hashes, the one available on the official domain and the file you have just downloaded, you will see that they are identical. They SHOULD be, mind. Otherwise, we might have a problem.
Issues with this method
If you’re even slightly savvy, you will notice that this approach is totally unscalable. First, you cannot possibly manually check every single package and then correlate to online information. Second, if you want to narrow the testing down to just critical packages, you need deep expertise in understanding which of the available binaries, libraries and utilities might be used against you, and it still does not guarantee any real understanding in what might be different. Third, there’s a tiny but this-road-leads-to-madness probability that the online information is unreliable.
This means that, for all practical purposes, while you could actually spend hours testing and verifying all your packages, basically doing something that servers running software repos and package managers actually do on their own as part of their internal checks, you will not be able to test the validity of your distro sources in any human-digestible manner. Even with automation. Moreover, discrepancies, be they real or false positives, will absolutely disrupt your work without any easy resolution.
This method does show you that things aren’t magically mysterious. There are simple ways to check and correlate information. There are also ways to ensure that package signatures are not tampered with, and hashes can be stored independently of the packages for additional redundancy and security. With Linux Mint, some of the sources are shared with Ubuntu, which runs on millions of home computers and quite a few servers, under tight scrutiny by people with vested, financial interest in keeping their distros clean and tidy and hacker-free.
If you cannot find it within yourself to let go, and believe there are decent enough mechanisms in place to ensure you received clean packages to your system — this automatically raises the question around private, unofficial channels like the PPA — then you cannot really use Mint anymore, or any other distribution for that matter.
Waiting is always good
Another approach is to actually wait whenever there’s a bunch of fresh new updates available. The law of large numbers, or rather, the law of large deployments dictates there will be someone else with your problem first, so let them solve it. This approach is somewhat selfish, but it guarantees you are not the the troubleshooting pioneer. Not only because there could be compromised packages coming to your system. Actually, far from it. The primary reason is to avoid bugs and errors.
More technology to our rescue
There are additional tools you can use to examine your distribution. A simple check of the network stack, to see what open connections you may have on your system, is also a good way to correlate the state of your Linux Mint installation before and after your update process. Just run:
sudo netstat -tulpan > ~/netstat.<before|after>
Compare the two files after you have completed the updates. You will notice that ports will probably change, and all browser sessions will be different. If you sanitize these, you should see similar output. Indeed, in my test, there were exactly 26 connections both before and after the update, corresponding to Samba, CUPS, DHCP client, and others. Nothing strange or weird.
Summary & when enough is enough
All right, let’s briefly wrap this up. Linux Mint had a problem, E-I-E-I-O. That problem was related to the website, ISO downloads and the forum database. These have since been fixed. The repos were not affected. But if you’re skeptical, you can manually download one, some or all update packages and check them yourself using a hash program like sha256sum.
Your next step is to compare the hash to what is advertised online, both on the official Mint site, the repos, as well as independent third-party sources, like university mirrors and such. This means you will have multiple samples for correlation, increasing the likelihood that the data is correct — it is not feasible that all websites hosting Mint information could/would be hacked. This should be peachy, and it should increase your peace of mind.
You can also check your network stack, as well as the process table, scheduled tasks, user session startup, and similar, to search for any weird processes, services, open ports, and the like. This requires skill and understanding of the inner workings of a Linux system. Again, you should not see any differences before and after the update when it comes to running services. Port numbers may change. Browser related connection will also be different and should be discounted from the comparison.
If you are still worried, you may want to read my article examining a false positive of a rootkit infection and all the available steps you can undertake to analyze such a situation. Then, I have also discussed the question of whether Ubuntu contains spyware, and how you can run the necessary tests to examine and verify claims of this nature. Also 100% valid for Linux Mint.
Linux Mint has suffered a reputation damage, which has led to doubts and questions being raised in the community. Valid questions, because you don’t want any private, confidential data to be transmitted to a third party without your knowledge and consent. So you may want to check if Mint is clean, and whether it can be used safely. This article outlines the technical methods.
However, you should also not forget that this is not the first, nor the last hack of a website related to a distro project. All the big names have had similar issues in the past. Moreover, obscurity does not guarantee security. If you’ve never thought about this topic before Feb 20, then you really should not be focusing too much energy on it now. Because all the other times you downloaded packages and updates your system, there could have been a breach somewhere, but since you were not aware of it, you did not do anything about it. Now you are aware, but it does not change the reality, only your perception. You may want to double-check everything now, it’s a natural reaction, but it’s not really grounded in any hard, solid facts. If anything, the hack only helps put more security highlight on the distros and their management, so they should now be more secure than ever before.