News

Cap'n Proto 0.6 Released: Two and a half years of improvements

kentonv on 01 May 2017

Today we’re releasing Cap’n Proto 0.6, the first major Cap’n Proto release in nearly 2.5 years.

Cap’n Proto has been under active development the entire time, as part of its parent project, Sandstorm.io. The lack of releases did not indicate a lack of development, but rather a lack of keeping the code running on every platform it supports – especially Windows. Without a working Windows build, we couldn’t do a release. But as Sandstorm didn’t need Windows, it was hard to prioritize – that is, until contributors stepped up!

Note that this release encompasses the core tools and the C++ reference implementation. Implementations in other languages have their own release schedules, but it’s likely that several will be updated soon to integrate new language features.

Brought to you by Cloudflare

As announced on the Sandstorm blog, most of the Sandstorm team (including myself) now work for Cloudflare. Cloudflare is one of the largest users of Cap’n Proto, as described in this talk by John-Graham Cumming, and as such maintaining Cap’n Proto is part of my job at Cloudflare.

What’s New?

Full Windows / Visual Studio Support

With this release, all of Cap’n Proto’s functionality now works on Windows with Visual Studio 2015 and 2017. That includes the serialization, dynamic API, schema parser, async I/O framework (using I/O completion ports), RPC, and tools. This is a huge step up from 0.5, in which Cap’n Proto could only be built in “lite mode”, which supported only basic serialization.

Most of the work to make this happen was contributed by Harris Hancock (with some help from Gordon McShane, Mark Grimes, myself, and others). It was no small feat: Visual Studio’s C++ compiler is still quite buggy, so lots of work-arounds were needed. Meanwhile, the Cap’n Proto developers working on Linux were continuously introducing new issues with their changes. Harris sorted it all out and delivered a beautiful series of patches. He also helped get us set up with continuous integration on AppVeyor, so that we can stay on top of these issues going forward.

Security Hardening

The 0.6 release includes a number of measures designed to harden Cap’n Proto’s C++ implementation against possible security bugs. These include:

• The core pointer validation code has been refactored to detect possible integer overflows at compile time using C++ template metaprogramming, as described in this old blog post.
• The core test suite – which runs when you type make check – now includes a targeted fuzz test of the pointer validation code.
• We additionally tested this release using American Fuzzy Lop, running several different test cases for over three days each.

JSON converter

Cap’n Proto messages can now be converted to and from JSON using libcapnp-json. This makes it easy to integrate your JSON front-end API with your Cap’n Proto back-end.

See the capnp/compat/json.h header for API details.

This library was primarily built by Kamal Marhubi and Branislav Katreniak, using Cap’n Proto’s dynamic API.

HTTP library

KJ (the C++ framework library bundled with Cap’n Proto) now ships with a minimalist HTTP library, libkj-http. The library is based on the KJ asynchronous I/O framework and covers both client-side and server-side use cases. Although functional and used in production today, the library should be considered a work in progress – expect improvements in future releases, such as client connection pooling and TLS support.

See the kj/compat/http.h header for API details.

Smaller things

With two years of development, there are far too many changes to list, but here are some more things:

• KJ now offers its own unit test framework under kj/test.h, as well as a compatibility shim with Google Test under kj/compat/gtest.h. The KJ and Cap’n Proto tests no longer depend on Google Test.
• New API capnp::TextCodec in capnp/serialize-text.h provides direct access to parse text-format Cap’n Proto messages (requires libcapnpc, the schema parser library). (Contributed by: Philip Quinn)
• It is possible to compare Cap’n Proto messages for equality (with correct handling of unknown fields, something Protocol Buffers struggled with) using APIs in capnp/any.h. (Contributed by: Joshua Warner)
• A function capnp::canonicalize() has been added which returns the canonical serialization of a given struct. (Contributed by: Matthew Maurer)
• AnyPointer fields can now be assigned in constant values, by referencing another named constant (which itself is defined with a specific type).
• In addition to AnyPointer, the types AnyStruct, AnyList, and Capability can now be used in schemas.
• New class capnp::CapabilityServerSet in capnp/capability.h allows an RPC server to detect when capabilities to its own local objects are passed back to it and allows it to “unwrap” them to get at the underlying native object.
• A membrane framework library was added (header capnp/membrane.h). This makes it easy to set up a MITM layer between RPC actors, e.g. to implement revocability, transformations, and many other useful capability patterns.
• Basic flow control can now be applied to an RPC connection, preventing new messages from being accepted if outstanding calls exceed a certain watermark, which helps prevent excessive buffering / malicious resource exhaustion. See RpcSystem::setFlowLimit().
• KJ’s networking API now includes datagram protocols (UDP).
• In .capnp syntax, all comma-delimited lists can now have a trailing comma. (Contributed by: Drew Fisher)
• Hundreds more small feature additions and bug fixes.