Announcing the new version of SecureDrop, with the results from our third security audit
Garrett Robinson
March 22, 2015
Today, we’re announcing the latest major release of SecureDrop, our open-source whistleblower submission system. SecureDrop 0.3 uses the same basic architecture found in 0.2, but contains numerous improvements focused on better usability for both journalists and sources, a radically simplified installation process, and an auto-updating procedure that allows us to deliver important fixes to all SecureDrop installations in a timely manner.
SecureDrop 0.3 was installed at the Toronto Globe and Mail and Gawker Media in February, and both organizations announced their deployments in early March. Based on their feedback, we have made some minor fixes and today are announcing SecureDrop 0.3.1 as the newest stable release of SecureDrop. All current installations are upgraded or in the process of upgrading to the latest version.
If you are interested in trying SecureDrop, start by reading the installation guide. For this and future releases, just check out the master branch of the SecureDrop git repository, which will always point to the latest stable release. Additionally, we recommend verifying the signed git tag for the release. Starting with SecureDrop 0.3, all of the release tags are signed with the Freedom of the Press Foundation master signing key.
Threats to press freedom around the world are at an all-time high. Sign up to stay up to date and take action to protect journalists and whistleblowers everywhere.
Thanks for signing up for our newsletter. You are not yet subscribed! Please check your email for a message asking you to confirm your subscription.
To accompany this release, we’re publishing the latest security audit of SecureDrop. As always, we are committed to having every major release audited by a different team of experts. This is SecureDrop’s third major security audit and you can read the first two here and here.
This time, we worked with a team from iSec Partners lead by Valentin Leon and Jonathan Chittenden. The audit was conducted in June 2014 and lasted several weeks, during which we communicated with the iSec team while they examined a demo instance we had set up for them. While previous audits have focused on the SecureDrop web application, this time we specifically asked the iSec researchers to examine the full stack.
We're happy to report that iSec did not find any "critical" vulnerabilities. They did find two "high"-severity vulnerabilities (one of which was in a third party dependency). We are also happy that all of the vulnerabilities they found were considered difficult to exploit, and "most of these issues can only be exploited in combination with some other, unknown, vulnerabilities". iSec also noted that "many of the issues in the report affect the environment rather than the web application... because the application presents a very limited attack surface." We feel this finding justifies our request to focus on the full stack, and we'll continue to encourage future auditors to look at the big picture which, while it may be more challenging, can also be much more rewarding.
All of the issues that iSec reported are resolved in this release of SecureDrop, most with the exact solutions recommended by iSec.
We’d like to thank iSec Partners and all of the individual researchers who contributed to the report.
You might notice that the audit was completed by mid-July last year, and that we are only now announcing the release, nearly eight months later. This delay was caused by a number of factors.
First of all, several of the proposed solutions for issues found in the audit required significant work and research to resolve. For example, resolving iSEC-14FTC-003-10 and iSEC-14FTC-003-1 required writing a new authentication backend for journalists, complete with two-factor authentication, into the web application. Writing this and integrating it into our installation and workflow processes was non-trivial.
Another example was iSEC-14FTC-003-9, which recommended sanitizing memory to avoid the retention of the plaintext of submissions submitted to the SecureDrop server. We spent over a month investigating a variety of techniques for mitigating this issue, but in the process found that the mitigations proposed by iSec were insufficient and a complete solution would be very difficult to implement. We ultimately decided to cut our losses and implement a simple kludge to mitigate the issue, but the time invested in trying to find a sophisticated solution significantly delayed the release.
We also came to the conclusion that SecureDrop’s installation, maintenance, and upgrade processes were in need of a complete overhaul. In 0.2, these processes were all highly manual, and often complex and error-prone due to the numerous security hardening protections used in SecureDrop. Since each SecureDrop installation is maintained independently (for security reasons), this created a tremendous support burden for us. It posed a threat to our future ability to scale to support more SecureDrop instances, and posed an even more serious threat to our present ability to secure existing SecureDrop installations.
While installations of SecureDrop 0.2.1 received automatic security updates for software provided by the Linux distribution and the Tor Project, there was no way for us to automatically deliver updates for SecureDrop itself, or for other important components of the SecureDrop architecture such as the OSSEC security monitoring system or the grsecurity hardened kernel.
As a team, we agreed that improving this situation was too important to wait for a later release, and so we began working to revamp the processes for the 0.3 release.
While this overhaul meant we had to wait six months to release the current version, we are confident that undertaking this project was worthwhile in the long run. We now have a straightforward provisioning system that can be used not just for production installations, but also for virtualized development and testing environments. This has made development and troubleshooting significantly easier and more efficient. Production installations are now much faster and less error-prone, and it is easier than ever for organizations and individuals to set up SecureDrop on their own. Finally, future upgrades will be significantly easier for SecureDrop admins to do by themselves, since we will be able to automate the process and provide an Ansible playbook to perform the upgrade. We plan to continue this trend of automation in future releases to improve the efficiency of our development, testing, and release processes, and will have some exciting news to announce on this front very soon!
Additionally, we have split the key components of SecureDrop into a set of Debian packages, and part of the new installation process establishes access to the FPF package repository so the instance can receive future updates automatically. This allows us to provide updates quickly and efficiently to resolve potential issues across our entire stack. We are also providing packages for the OSSEC monitoring tool and a hardened Linux kernel with grsecurity.
Previously, the use of grsecurity was strongly recommended, and we provided a simple set of instructions for SecureDrop admins to build and maintain hardened kernels on their own. Unfortunately, due to the hassle involved in maintaining a custom kernel, few installations actually did this. With SecureDrop 0.3, all installations now have grsecurity installed automatically and with no extra effort on the part of each admin.
Freedom of the Press Foundation will maintain the hardened kernel and will provide timely updates to all of the installations. This is a major step towards enhancing the security of every SecureDrop installation - making kernel hardening a default part of the platform means every installation now benefits from the sophisticated protections provided by grsecurity.
For those interested in the more technical details of the improvements made to this version of SecureDrop, please see below.
The error in the unattended-upgrades configuration was initially fixed in #443. After we started testing automatic upgrades with our Debian package repository, we encountered unexpected problems with unattended-upgrades and ultimately switched to using cron-apt in #717to resolve these issues. cron-apt is more configurable and helps us avoid unnecessary complexity in the layout and maintenance of our package repository.
The point of this issue is to ensure that security updates are actually being downloaded and installed regularly. We have verified that this is the case in 0.3 using cron-apt, so the issue is resolved in a manner analogous to the short term solution recommended by iSec.
Our initial approach to resolving this issue was to look for alternative Apache modules to implement two-factor authentication with the same UX that we had with `google-authenticator-apache-module`. At first, we wanted to make the minimum of changes to resolve the issue in order to minimize the amount of unaudited code (code written after the audit, because of the audit) in the final release.
Two factors led us to take a different path. First, the UX provided by `google-authenticator-apache-module` was very poor - a modal dialog pop-up the browser, similar to a website using HTTP authentication. It was not possible to customize the content of this dialog to indicate what information should be specifically entered in the text field it provided, and there was no way to provide helpful error messages if authentication failed, which is especially important in the context of time-based two-factor authentication. Second, there was no way to extend it to support a password in addition to a two-factor code, which was the recommended short-term solution for another issue reported by iSec (iSEC-14FTC-003-1). A proposed solution to ask users to enter the password followed by the two-factor code in the single field provided by the authentication prompt was rejected as having extremely poor usability.
Ultimately, we decided to implement two-factor authentication in the web application in #522, which was the long term solution suggested by iSec.
This was one of the most challenging issues to mitigate. SecureDrop is a web application that receives plaintext submissions over an end-to-end encrypted transport (thanks to the Tor Hidden Service protocol) and encrypts them to specific journalists. Since the plaintext is streamed into the web server and handed off to the web application, it is naturally retained in several places. iSec included a proof of concept in their report demonstrating that small amounts of plaintext (such as the content of a message sent by a SecureDrop source) can remain in memory for hours after submission.
Initially, the iSec researchers believed that this issue was primarily due to our use of Python for the SecureDrop web application. Since Python is a garbage-collected language, it is impossible to reliably free memory, and even when memory is freed it is difficult to sanitize it (e.g. by overwriting the allocated area with zeros or random data before returning it to the allocator).
We approached this issue by first developing a framework to test for the presence of submission plaintext in memory, using a virtualized environment and the Volatility memory forensics framework. We used VirtualBox, which allowed us to dump images of the VM’s entire address space that could then be analyzed with Volatility. This allowed us to do a more comprehensive test that simulated forensic memory acquisition.
Unfortunately, analysis using this framework revealed that the mitigations proposed by iSec were insufficient. Even when we were able to prevent the Python web application processes from retaining plaintext in memory, plaintext was retained in other areas of the system, such as in the Apache web server’s memory. A few examples of the things we tried include:
The mitigations based around PAX_MEMORY_SANITIZE were also difficult to test because it was impossible to get a grsecurity kernel working with Volatility (due at least in part to grsecurity’s anti-forensic protections), and so we had to fall back on inferior methods of analysis (strings and grep).
In the end, we concluded that a truly comprehensive mitigation would only be possible by re-writing the application in a low-level language affording complete control over memory, such as C/C++. To mitigate the issue in 0.3, we used a kludge. Since the servers automatically install updates every 24 hours, and sometimes automatically reboot if the updates require a reboot, we changed their scheduling to always reboot every 24 hours, reducing the maximum potential lifetime of submission plaintext in memory to 24 hours or less.
This issue was tracked in #99.
As part of the authentication flow for journalists (implemented to mitigate iSEC-14FTC-003-10), we started requiring passwords in addition to the Authenticated Tor Hidden Service "auth-cookie" and a TOTP (or HOTP) based second factor code, which was the short-term solution suggested by iSec. This issue was tracked in #435 and resolved as part of #522.
We implemented the short term solution proposed by iSec in #445.
We implemented the short term solution recommended by iSec.
The first part of the solution was to strip email headers that may contain sensitive information about the installation. This was tracked in #442 and was implemented for the 0.3 release.
The second part of the solution was to avoid potentially including the IP address of either the App Server or the Monitor Server in the subject of the Postfix emails. When we discussed this mitigation, we determined that this was not really a threat because the IP addresses leaked in the subject line of the OSSEC alerts are the private addresses of the servers, on a network interface that blocks all incoming traffic that is not related to initial outgoing traffic from the private network. Additionally, the host firewall rules for both servers block all incoming connections unless they are made in response to an outgoing connection. As a result, we did not believe this was a real vulnerability and did not mitigate it for 0.3. However, we recently decided that we might as well implement the solution since it was so simple, and so it was ultimately implemented in #915.
We implemented the short term solution proposed by iSec in #437.
This was ultimately fixed when we switched to using Ansible for configuring and installing SecureDrop. The playbook configuration file allows the SecureDrop admin to specify the external DNS server to use, and the host firewall rules restrict DNS traffic to the specified server. This is an implementation of the short-term solution proposed by iSec, and was tracked in #438.
As explained in the section on iSEC-14FTC-003-10, we ultimately switched from unattended-upgrades to cron-apt to manage automatic package upgrades for the 0.3 release. Thanks to iSec reporting this issue, we made sure to have OSSEC monitor cron-apt’s logs. We configured cron-apt to emit logging to syslog, which is monitored by
OSSEC, and wrote a custom OSSEC decoder to catch and always send alerts on cron-apt errors. This is the cron-apt equivalent of the short-term solution proposed by iSec and was implemented in #853.
We implemented the short term solution proposed by iSec in #440.
Many of the improvements in this release would not have been possible without SecureDrop's awesome contributors. We'd especially like to thank the entire ThoughtWorks community, which fostered the project and contributed tremendously to its evolution; Jack Singleton and Charlie Austin, whose passion for the project brought in numerous contributors in addition to their own significant contributions; Max Griffiths, Chris Ng, and Vlad, whose sage devops wisdom lead to our significant progress in, and ongoing project toward, improving our development, test, and release processes; David Stainton, for his crucial early assistance in writing our Ansible playbooks; spender, for answering all of our questions about grsecurity and helping us debug and patch tricky hardware compatibility issues; Rosalie Tolentino and Karlyn Murphy for their valiant efforts to impove and facilitate our project management; Nick Bailey and Pamela Ocampo for doing a fantastic UI refresh over the course of a single hackathon; Bill Budington, Yan Zhu, and Micah Lee for their significant and ongoing contributions to the project; and Runa Sandvik for her major contributions to our documentation and organizational efforts.