EDP Coming to More Platforms, Keeping TCB Small

The Fortanix Enclave Development Platform targets Intel SGX and confidential virtual machines. The next steps are more platforms combined with a trusted computing base as close to SGX as possible.

In 2019, I announced the Fortanix Enclave Development Platform (EDP) as “the preferred way to write Intel SGX enclaves from scratch”. It is the foundation of every Fortanix product.

Half a year before that, I had explained why Fortanix chose Intel Software Guard Extensions (SGX), calling it “the best choice for our Runtime Encryption® platform”. Since then, most new Confidential Computing platforms have taken a route different from SGX. Instead of protecting a region inside an ordinary process, as SGX does, they protect an entire virtual machine (VM). For these platforms, EDP has a second compilation target called Fortanix VME, short for VM-based Enclaves. With VME, EDP applications already run on AWS Nitro Enclaves and on AMD Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP).

A confidential VM (CVM) usually runs a full Linux distribution, and the large trusted computing base (TCB) this brings is treated as a given. The architecture, however, does not require Linux at all. EDP started on SGX, but its design pattern is not tied to any specific platform. An EDP application reaches the untrusted world through a single interface that is small enough to audit. Inside the trust boundary, there is hardly any code beyond the application itself. A CVM can be built the same way.

Today, VME still places a Linux kernel inside the trust boundary, but only as an interim solution. In the future, it will run without Linux, and also, like SGX, reach upstream Rust tier 2 platform support, as well as be able to target Intel Trust Domain Extensions (TDX).

Enclave Development Platform (EDP) Recap

For a developer, an EDP application is an ordinary Rust program. Execution begins in fn main, and threads and network connections work the way they do on any other platform. You don’t have to write a separate untrusted component to accompany the application, because EDP’s runner loads the enclave and handles its requests.

On SGX, where EDP began, the enclave contains only your application and Rust’s standard library, std. That makes SGX the gold standard for TCB size. There, std communicates with the runner through fewer than 20 usercalls. These were designed to withstand Iago attacks, in which a malicious operating system manipulates a protected program through carefully chosen return values. The EDP architecture documentation describes this interface and the rest of the design in more detail.

A Misconception About Confidential VMs

A CVM is easily mistaken for an ordinary VM with a confidentiality switch turned on, which would mean it has to contain firmware, a kernel and a full userland. Seen that way, a large TCB is inherent to the technology.

The mistake comes from an approach known as lift and shift, which treats a CVM as a drop-in replacement for an ordinary VM. Existing server software moves into the CVM together with the firmware and operating system it expects, so the whole stack ends up inside the trust boundary. Even so, the move is rarely as seamless as the name suggests, since protecting the software properly still requires changes.

Bringing that stack along has a cost. Firmware and the Linux kernel add megabytes to the code that has to be trusted. The firmware runs first, so it handles the host’s initial data and has to measure or verify everything it loads, or attestation no longer covers what runs after it. The kernel then exposes interfaces such as port I/O, memory-mapped I/O, hypercalls, shared memory pages and injected interrupts, which its own threat model for confidential VMs lists. Each of these interfaces carries input controlled by the infrastructure provider, which is precisely the party Confidential Computing is meant to protect against. In effect, the Iago problem returns, with the guest’s firmware and kernel now in the role of the protected program.

In reality, such a large TCB is not inherent to the technology. A CVM protects whatever runs inside it, whether that is a complete OS stack or a single application.

Fortanix VME Today

In its current form, a VME enclave contains a Linux kernel and one multi-threaded application, without the userland that a Linux system would normally have. (In the next section, I’ll discuss how we plan to get rid of Linux.)

On VME, the runner interface has two parts. The first part is defined by the architecture, which includes the hypercalls, shared memory and injected interrupts through which any guest deals with its hypervisor. The second is the Fortanix VME vsock protocol that carries networking. Vsock is a type of socket designed for communication between a VM and its host. On top of that protocol, std provides TcpStream and TcpListener, exactly as it does with usercalls on SGX.

As a result, your source code stays the same whichever target you build it for. For example, the program below runs in an SGX enclave, on Nitro Enclaves and in an SEV-SNP VM:

use std::io::Write;
use std::net::TcpListener;

fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("0.0.0.0:8080")?;
    for stream in listener.incoming() {
        stream?.write_all(b"Hello from inside the trust boundary\n")?;
    }
    Ok(())
}

EDP-VME is generally available for evaluation on Nitro Enclaves and on SEV-SNP. The two platforms package the enclave differently and produce different attestation evidence, but the application never sees either difference.

Virtual firmware and Linux handle the architectural part of the interface, along with tasks such as booting, memory management, thread scheduling and attestation. The downside is that both sit inside the trust boundary in their entirety, together with all the code the application never uses. Much of this guest stack faces the host directly and was originally written for a trustworthy hypervisor. This temporary arrangement let EDP reach new platforms early, ahead of the smaller design that is still taking shape.

EDP-VME Without Linux

Linux can be removed from VME in one of two ways. One option is for std to take over the work that firmware and Linux do today, as it already does on SGX. The other is to put a formally verified microkernel in place of Linux.

Either way, the plan is for the runner to take on the role of the VM monitor, using the Linux Kernel-based Virtual Machine (KVM) interface directly instead of QEMU. Both options also need code specific to each platform for memory management, attestation and CVM virtualization exceptions.

The Standard Library as the Platform

If std replaces firmware and Linux, the trust boundary contains nothing but the application and the standard library.

Each thread gets a dedicated virtual CPU (vCPU), much as each thread in an SGX enclave occupies its own thread control structure. No scheduling or preemption is needed, except for the cooperative task scheduling that a Rust async runtime already performs inside the application, if it uses one. A waiting thread simply halts its vCPU until it is woken.

There is no firmware either, because the entry code in std takes over its start-up work, including setting up page tables. The page tables can also be built before launch, letting the guest read its memory map from them. Because the launch measurement then covers the tables, the layout is fixed and can be verified just like the size of an SGX enclave.

While the Enarx project is no longer under active development, it offers evidence that the approach outlined here is practical. Its KVM shim boots an SEV-SNP guest, configures paging and handles virtualization exceptions in a thin layer of Rust.

Why Not a Verified Microkernel?

The other option replaces Linux with a kernel backed by machine-checked proofs, such as seL4. Those proofs show that the kernel behaves as specified and, on some architectures, that it keeps processes apart from one another. VME, however, runs a single application and needs protection from the hypervisor, which the proofs do not address.

The proofs also leave out the boot code and the machine interface, which is exactly where the guest first handles state supplied by the host. On x86-64, where SEV-SNP and TDX run, the verified configuration covers functional correctness alone, and the multicore kernel that a multi-threaded application needs is still being verified. The code a CVM adds for virtualization exceptions, page validation and the transport would fall outside the proofs as well.

A microkernel would still bring kernel code that is proven correct and fine-grained isolation between components, but it would also need a runtime on top of it before it could host an EDP application. All things considered, this explains why we’re leaning toward the other approach.

An Open Question: The Transport

A separate question, orthogonal to the minimization approach chosen, is whether vsock should remain the transport. Nitro Enclaves requires it. However, SEV-SNP and TDX don’t, which leaves the guest free to talk to the runner through an interface like the SGX usercall interface instead. That would carry over the hardening of the SGX target, including the Rust types that stop enclave code from accessing user memory directly, and avoid implementing virtio as new host-facing code. The cost is a second transport to maintain alongside vsock.

TDX and the Road Ahead

The platforms are at different stages today. On SGX, EDP applications already run without any operating system inside the trust boundary. On Nitro Enclaves and SEV-SNP, EDP is available for evaluation, although Linux still sits inside the trust boundary there. Support for TDX is on the roadmap and will come through the same VME target. Regardless of platform, you’ll end up with nothing but your application and std inside the trust boundary.

In upstream Rust, the SGX target is a tier 2 target. Tier 2 means that the target keeps building as the compiler changes and that official builds of its standard library can be installed with rustup. The intention is to bring EDP-VME to the same status.

The way you write an EDP application stays the same throughout, and the program shown earlier keeps running unchanged as the platforms beneath it evolve. The resulting TCB should be as close to SGX as possible for every EDP application, wherever it runs.

Help shrink the TCB on every platform and join the EDP community!