The researchers tested Flow, Facebook’s static type system, and TypeScript – from Microsoft, of course – and found that, at a conservative estimate, each product can spot approximately 15% of bugs which would otherwise have ended up in committed code.
As the report introduction puts it: “Proponents of static typing argue that it detects bugs before execution, increases runtime efficiency, improves program understanding, and enables compiler optimisation. Dynamic typing, its advocates claim, is well-suited for prototyping, since it allows a developer to quickly write code that works on a handful of examples without the cost of adding type annotations.”
Yet as the research shows, it may be worth taking the effort to add type annotations after all.
Of the 400, Flow successfully detected 59 and TypeScript 58 at first analysis, although after labelling all the bugs as either detectable or undetectable under Flow and TypeScript, the number rose to 60 for each. Factoring in the confidence level, the eventual percentage of between 11.5% and 18.5%, with the mean at 15%. According to the data, the vast majority of the bugs picked up were done so by both programs, with only three on each side only being preventable by Flow and TypeScript alone.
The study aimed to focus on the effects of annotations rather than the annotator – much in the same way surgery trials seek to draw conclusions over the surgeries rather than the surgeon – with the researchers opting for methods to negate ‘learning effects’ during the tests.
Ultimately, the question is one of whether a 15% reduction in bugs is worth the extra work in annotation. As Adrian Colyer put it in a blog post assessing the study, Flow’s more lightweight approach is potentially better suited to dynamic typing advocates, with TypeScript a better option for strong-typing advocates coming from other languages. The researchers added the results from the study were “encouraging.”
You can read the full report here (pdf, 12 pages).