Today, we’re announcing the latest major release of SecureDrop, our open-source whistleblower submission system. SecureDrop 0.3 uses the same basic architecture found in 0.2, but contains numerous improvements focused on better usability for both journalists and sources, a radically simplified installation process, and an auto-updating procedure that allows us to deliver important fixes to all SecureDrop installations in a timely manner.

SecureDrop 0.3 was installed at the Toronto Globe and Mail and Gawker Media in February, and both organizations announced their deployments in early March. Based on their feedback, we have made some minor fixes and today are announcing SecureDrop 0.3.1 as the newest stable release of SecureDrop. All current installations are upgraded or in the process of upgrading to the latest version.

If you are interested in trying SecureDrop, start by reading the installation guide. For this and future releases, just check out the master branch of the SecureDrop git repository, which will always point to the latest stable release. Additionally, we recommend verifying the signed git tag for the release. Starting with SecureDrop 0.3, all of the release tags are signed with the Freedom of the Press Foundation master signing key.

Security Audit

To accompany this release, we’re publishing the latest security audit of SecureDrop. As always, we are committed to having every major release audited by a different team of experts. This is SecureDrop’s third major security audit and you can read the first two here and here.

This time, we worked with a team from iSec Partners lead by Valentin Leon and Jonathan Chittenden. The audit was conducted in June 2014 and lasted several weeks, during which we communicated with the iSec team while they examined a demo instance we had set up for them. While previous audits have focused on the SecureDrop web application, this time we specifically asked the iSec researchers to examine the full stack.

We're happy to report that iSec did not find any "critical" vulnerabilities. They did find two "high"-severity vulnerabilities (one of which was in a third party dependency). We are also happy that all of the vulnerabilities they found were considered difficult to exploit, and "most of these issues can only be exploited in combination with some other, unknown, vulnerabilities". iSec also noted that "many of the issues in the report affect the environment rather than the web application... because the application presents a very limited attack surface." We feel this finding justifies our request to focus on the full stack, and we'll continue to encourage future auditors to look at the big picture which, while it may be more challenging, can also be much more rewarding.

All of the issues that iSec reported are resolved in this release of SecureDrop, most with the exact solutions recommended by iSec.

We’d like to thank iSec Partners and all of the individual researchers who contributed to the report.

Why the delay?

You might notice that the audit was completed by mid-July last year, and that we are only now announcing the release, nearly eight months later. This delay was caused by a number of factors.

First of all, several of the proposed solutions for issues found in the audit required significant work and research to resolve. For example, resolving iSEC-14FTC-003-10 and iSEC-14FTC-003-1 required writing a new authentication backend for journalists, complete with two-factor authentication, into the web application. Writing this and integrating it into our installation and workflow processes was non-trivial.

Another example was iSEC-14FTC-003-9, which recommended sanitizing memory to avoid the retention of the plaintext of submissions submitted to the SecureDrop server. We spent over a month investigating a variety of techniques for mitigating this issue, but in the process found that the mitigations proposed by iSec were insufficient and a complete solution would be very difficult to implement. We ultimately decided to cut our losses and implement a simple kludge to mitigate the issue, but the time invested in trying to find a sophisticated solution significantly delayed the release.

We also came to the conclusion that SecureDrop’s installation, maintenance, and upgrade processes were in need of a complete overhaul. In 0.2, these processes were all highly manual, and often complex and error-prone due to the numerous security hardening protections used in SecureDrop. Since each SecureDrop installation is maintained independently (for security reasons), this created a tremendous support burden for us. It posed a threat to our future ability to scale to support more SecureDrop instances, and posed an even more serious threat to our present ability to secure existing SecureDrop installations.

While installations of SecureDrop 0.2.1 received automatic security updates for software provided by the Linux distribution and the Tor Project, there was no way for us to automatically deliver updates for SecureDrop itself, or for other important components of the SecureDrop architecture such as the OSSEC security monitoring system or the grsecurity hardened kernel.

As a team, we agreed that improving this situation was too important to wait for a later release, and so we began working to revamp the processes for the 0.3 release.

While this overhaul meant we had to wait six months to release the current version, we are confident that undertaking this project was worthwhile in the long run. We now have a straightforward provisioning system that can be used not just for production installations, but also for virtualized development and testing environments. This has made development and troubleshooting significantly easier and more efficient. Production installations are now much faster and less error-prone, and it is easier than ever for organizations and individuals to set up SecureDrop on their own. Finally, future upgrades will be significantly easier for SecureDrop admins to do by themselves, since we will be able to automate the process and provide an Ansible playbook to perform the upgrade. We plan to continue this trend of automation in future releases to improve the efficiency of our development, testing, and release processes, and will have some exciting news to announce on this front very soon!

Additionally, we have split the key components of SecureDrop into a set of Debian packages, and part of the new installation process establishes access to the FPF package repository so the instance can receive future updates automatically. This allows us to provide updates quickly and efficiently to resolve potential issues across our entire stack. We are also providing packages for the OSSEC monitoring tool and a hardened Linux kernel with grsecurity.

Previously, the use of grsecurity was strongly recommended, and we provided a simple set of instructions for SecureDrop admins to build and maintain hardened kernels on their own. Unfortunately, due to the hassle involved in maintaining a custom kernel, few installations actually did this. With SecureDrop 0.3, all installations now have grsecurity installed automatically and with no extra effort on the part of each admin.

Freedom of the Press Foundation will maintain the hardened kernel and will provide timely updates to all of the installations. This is a major step towards enhancing the security of every SecureDrop installation - making kernel hardening a default part of the platform means every installation now benefits from the sophisticated protections provided by grsecurity.

For those interested in the more technical details of the improvements made to this version of SecureDrop, please see below.

Audit Issues and their Resolutions

  • 1. Security patches incorrectly installed (iSEC-14FTC-003-5)

The error in the unattended-upgrades configuration was initially fixed in #443. After we started testing automatic upgrades with our Debian package repository, we encountered unexpected problems with unattended-upgrades and ultimately switched to using cron-apt in #717to resolve these issues. cron-apt is more configurable and helps us avoid unnecessary complexity in the layout and maintenance of our package repository.

The point of this issue is to ensure that security updates are actually being downloaded and installed regularly. We have verified that this is the case in 0.3 using cron-apt, so the issue is resolved in a manner analogous to the short term solution recommended by iSec.

  • 2. Apache Two Factor Authentication (2FA) module bypass (iSEC-14FTC-003-10)

Our initial approach to resolving this issue was to look for alternative Apache modules to implement two-factor authentication with the same UX that we had with `google-authenticator-apache-module`. At first, we wanted to make the minimum of changes to resolve the issue in order to minimize the amount of unaudited code (code written after the audit, because of the audit) in the final release.

Two factors led us to take a different path. First, the UX provided by `google-authenticator-apache-module` was very poor - a modal dialog pop-up the browser, similar to a website using HTTP authentication. It was not possible to customize the content of this dialog to indicate what information should be specifically entered in the text field it provided, and there was no way to provide helpful error messages if authentication failed, which is especially important in the context of time-based two-factor authentication. Second, there was no way to extend it to support a password in addition to a two-factor code, which was the recommended short-term solution for another issue reported by iSec (iSEC-14FTC-003-1). A proposed solution to ask users to enter the password followed by the two-factor code in the single field provided by the authentication prompt was rejected as having extremely poor usability.

Ultimately, we decided to implement two-factor authentication in the web application in #522, which was the long term solution suggested by iSec.

  • 3. Sensitive data remains in memory (iSEC-14FTC-003-9)

This was one of the most challenging issues to mitigate. SecureDrop is a web application that receives plaintext submissions over an end-to-end encrypted transport (thanks to the Tor Hidden Service protocol) and encrypts them to specific journalists. Since the plaintext is streamed into the web server and handed off to the web application, it is naturally retained in several places. iSec included a proof of concept in their report demonstrating that small amounts of plaintext (such as the content of a message sent by a SecureDrop source) can remain in memory for hours after submission.

Initially, the iSec researchers believed that this issue was primarily due to our use of Python for the SecureDrop web application. Since Python is a garbage-collected language, it is impossible to reliably free memory, and even when memory is freed it is difficult to sanitize it (e.g. by overwriting the allocated area with zeros or random data before returning it to the allocator).

We approached this issue by first developing a framework to test for the presence of submission plaintext in memory, using a virtualized environment and the Volatility memory forensics framework. We used VirtualBox, which allowed us to dump images of the VM’s entire address space that could then be analyzed with Volatility. This allowed us to do a more comprehensive test that simulated forensic memory acquisition.

Unfortunately, analysis using this framework revealed that the mitigations proposed by iSec were insufficient. Even when we were able to prevent the Python web application processes from retaining plaintext in memory, plaintext was retained in other areas of the system, such as in the Apache web server’s memory. A few examples of the things we tried include:

  1. Replacing the stream class used by our web application framework (Python’s `io.BytesIO`) with a custom version that sanitized the underlying buffer whenever it was reallocated or freed. We also added a method to explicitly trigger sanitization to work around our inability to guarantee when the object might be freed. This technique reduced the number of instances of submission plaintext in memory, but did not eliminate them.
  2. Since our new package layout made it possible for us to rely on the presence of grsecurity, we attempted to use PAX_MEMORY_SANITIZE, a feature of PaX/grsecurity that sanitizes all freed memory and is enabled in our kernel builds. The reasoning here was that if we could reliably cause the memory to be freed, PaX would take care of sanitizing it for us. We focused on marrying this technique with recycling processes, since all process memory is freed on exit. We tried two approaches:
    1. Apache allows you to configure how it interacts with external programs to respond to web requests. In the case of our web application, we configured it so it would use a single Python process for each request, and only allow that process to serve a single request. This caused the process to die immediately after serving the request, and the memory (including the submission plaintext) was subsequently sanitized. Unfortunately while this technique substantially reduced the number of instances of plaintext in memory, it did not eliminate them. Analysis with Volatility showed that Apache was also keeping request data around in memory for unpredictable amounts of time. This lead to our next approach:
    2. We tried periodically restarting the web sever, which kills all of the attached Python processes as a byproduct. This was effective at sanitizing memory, but it was difficult for us to determine when to restart the server without harming availability. In particular, since the server is only available over Tor, bandwidth can be quite limited and uploads can be very time consuming. We wanted to avoid the case where restarting the server to sanitize memory might kill a long-running upload, because that would frustrate sources and potentially prevent them from uploading large amount of data entirely. We confirmed that restarting the server during an upload would kill it (even when we used Apache’s "graceful restart"), and at this point decided we had invested too much time and discontinued this line of inquiry.

The mitigations based around PAX_MEMORY_SANITIZE were also difficult to test because it was impossible to get a grsecurity kernel working with Volatility (due at least in part to grsecurity’s anti-forensic protections), and so we had to fall back on inferior methods of analysis (strings and grep).

In the end, we concluded that a truly comprehensive mitigation would only be possible by re-writing the application in a low-level language affording complete control over memory, such as C/C++. To mitigate the issue in 0.3, we used a kludge. Since the servers automatically install updates every 24 hours, and sometimes automatically reboot if the updates require a reboot, we changed their scheduling to always reboot every 24 hours, reducing the maximum potential lifetime of submission plaintext in memory to 24 hours or less.

This issue was tracked in #99.

  • 4. Lack of two-factor authentication for journalists (iSEC-14FTC-003-1)

As part of the authentication flow for journalists (implemented to mitigate iSEC-14FTC-003-10), we started requiring passwords in addition to the Authenticated Tor Hidden Service "auth-cookie" and a TOTP (or HOTP) based second factor code, which was the short-term solution suggested by iSec. This issue was tracked in #435 and resolved as part of #522.

  • 5. Collisions can occur during code name generation (iSEC-14FTC-003-11)

We implemented the short term solution proposed by iSec in #445.

  • 6. OSSEC alerts disclose system details (iSEC-14FTC-003-7)

We implemented the short term solution recommended by iSec.

The first part of the solution was to strip email headers that may contain sensitive information about the installation. This was tracked in #442 and was implemented for the 0.3 release.

The second part of the solution was to avoid potentially including the IP address of either the App Server or the Monitor Server in the subject of the Postfix emails. When we discussed this mitigation, we determined that this was not really a threat because the IP addresses leaked in the subject line of the OSSEC alerts are the private addresses of the servers, on a network interface that blocks all incoming traffic that is not related to initial outgoing traffic from the private network. Additionally, the host firewall rules for both servers block all incoming connections unless they are made in response to an outgoing connection. As a result, we did not believe this was a real vulnerability and did not mitigate it for 0.3. However, we recently decided that we might as well implement the solution since it was so simple, and so it was ultimately implemented in #915.

  • 7. Iptables rules allows for any traffic on port 123 (iSEC-14FTC-003-3)

We implemented the short term solution proposed by iSec in #437.

  • 8. Iptables rules allow for DNS traffic with any server (iSEC-14FTC-003-2)

This was ultimately fixed when we switched to using Ansible for configuring and installing SecureDrop. The playbook configuration file allows the SecureDrop admin to specify the external DNS server to use, and the host firewall rules restrict DNS traffic to the specified server. This is an implementation of the short-term solution proposed by iSec, and was tracked in #438.

  • 9. OSSEC not watching unattended-upgrade log file (iSEC-14FTC-003-6)

As explained in the section on iSEC-14FTC-003-10, we ultimately switched from unattended-upgrades to cron-apt to manage automatic package upgrades for the 0.3 release. Thanks to iSec reporting this issue, we made sure to have OSSEC monitor cron-apt’s logs. We configured cron-apt to emit logging to syslog, which is monitored by

OSSEC, and wrote a custom OSSEC decoder to catch and always send alerts on cron-apt errors. This is the cron-apt equivalent of the short-term solution proposed by iSec and was implemented in #853.

  • 10. Outdated OSSEC and AppArmor configuration (iSEC-14FTC-003-4)

We implemented the short term solution proposed by iSec in #440.

Thanks

Many of the improvements in this release would not have been possible without SecureDrop's awesome contributors. We'd especially like to thank the entire ThoughtWorks community, which fostered the project and contributed tremendously to its evolution; Jack Singleton and Charlie Austin, whose passion for the project brought in numerous contributors in addition to their own significant contributions; Max Griffiths, Chris Ng, and Vlad, whose sage devops wisdom lead to our significant progress in, and ongoing project toward, improving our development, test, and release processes; David Stainton, for his crucial early assistance in writing our Ansible playbooks; spender, for answering all of our questions about grsecurity and helping us debug and patch tricky hardware compatibility issues; Rosalie Tolentino and Karlyn Murphy for their valiant efforts to impove and facilitate our project management; Nick Bailey and Pamela Ocampo for doing a fantastic UI refresh over the course of a single hackathon; Bill Budington, Yan Zhu, and Micah Lee for their significant and ongoing contributions to the project; and Runa Sandvik for her major contributions to our documentation and organizational efforts.