Evaluating the usability of differential privacy tools with data practitioners

Abstract

‍
Which differential privacy library is easiest to use safely? 

Thanks to new research by Ivoline Ngong, Brad Stenger, Joseph Near, and Yuanyuan Feng, we now have initial answers, and they're a testament to the usability work that went into building Tumult Analytics: 83% of test users successfully performed 3 differential privacy analysis tasks in just one hour using Tumult Analytics, even though none of them had any prior experience with the framework!

Equally importantly, everyone who completed the task also did it correctly, and got exactly the privacy guarantees that they wanted. This is critical, because getting strong protection is the whole point, and privacy failures are invisible — the output looks reasonable! This is fantastic independent validation of all the work we did making the API not only simple to use, but also difficult to inadvertently mis-use.

In the authors’ words:

Tumult Analytics seems to strike the best balance [between safety and usability]. Its API was effective at preventing DP violations. And the API users had high completion rates and satisfaction scores. This success is likely due to careful design of the API and its error messages.

The research also identifies multiple possible areas of improvement, like organizing the documentation in a more user-friendly way, or making some DP-specific parameters optional. Count on us to continually invest in making differential privacy even easier to use and deploy!