Tools we love Vol.9: Darwin

About the series

As you know, we at Humans in the Loop have a great love and appreciation of a well-designed annotation tool. After the great feedback on the reviews we published of our the best platforms on the market here and here, we decided that it’s time for a deep dive in some of our all-time favorites!

This article is the second to last one from a series of 10 reviews which were published each week. You can find the rest of the series on our blog!

The whole series is based on the premise of transparency and honesty and none of these reviews are sponsored. They are just our way to give props to the best teams out there working on making annotation easier for AI teams, and to share some of the know-how that we have been accumulating over the past few years as a professional annotation company.

As in previous reviews, our parameters are:

• price
• functions
• project management
• automation

If you have additional questions or want to get in touch with us to beta test or feature your tool in an upcoming article, feel free to email us at hello@humansintheloop.org!

The tool

Darwin is the brainchild of V7 Labs, a UK company founded in 2018 which is also behind exciting computer vision projects like AI Poly and Autonomous Retail. Darwin is one of the most versatile and advanced tools for image and video annotation out there and what we love about it is that the team is continuously fixing bugs and releasing new features.

Some of the best features of the platform which we will cover in more detail below are its ability to handle complex polygonal and semantic segmentation annotation in a smart way including interpolation of polygons on video. Its ability to manage ultra-high resolution imagery and a variety of formats makes it the go-to platform for complex use cases such as medical annotation.

The pricing of Darwin starts at $150 per month which gets you 7500 automated annotations or 75 labeling hours monthly. At higher volumes it goes all the way down to $0.60 per labeling hour and $0.006 per automated annotation. Users can also buy model training and inference services starting at $99 per training cycle.

Features

The annotation interface of Darwin is really user-friendly and clean, with a detailed annotation panel, hotkeys, and image manipulation settings. Our favorite feature which we will cover in the ‘Automation’ section is their Auto-annotate smart polygon tool but there are tons of other aspects which make it great for annotators and project managers alike.

The platform supports a variety of annotation formats: bounding boxes, polygons, keypoints, cuboids, tags, lines, ellipses, and skeletons. When setting up a class, the project manager can define a description of the class and set up a thumbnail so as to make it as easy as possible for annotators, as well as create subtypes like an instance ID, attributes, text, a directional vector, etc.

The supported file types include .mp4, .mov, .avi, .bmp, .jpg, .jpeg, .png, .svs, .tif, .tiff, and .dcm in both RGB and YBR color space. For each video uploaded, users can choose the frame rate for annotation and whether to annotate it as a video or as separate images. The export is equally versatile, with export formats such as COCO, CVAT, PASCAL VOC, PNG masks, and Darwin’s own format in JSON or XML. At Export, admins can create a version of the dataset which acts as a snapshot and cannot be edited but which they can refer to in the future.

Project management

At project setup time, admins can define the stages which each image will go through, including ‘Annotation’ and ‘Review’ (including random sampling of a percentage of the data). At each stage, admins can assign a certain portion of the work to go to specific annotators or to be split between all annotators (annotators may be enabled to ‘self-assign’ batches of work indefinitely).

During ‘Review’ stages, reviewers can work in a ‘read-only’ mode where they only pinpoint mistakes and return the work to the previous stage, or they could be expected to make corrections on the annotations. There is also a very handy feature for ‘Comments’ where both annotators and reviewers can select an area and discuss potential issues or questions.

Dataset management is another strong side of the platform. Data can be sorted in folders and assigned a priority score for faster annotation. Admins can monitor its progress through the workflow by filtering data that is ‘Complete’, ‘New’, in ‘Annotation’ stage, in ‘Review’ stage, or ‘Processing’. Finally, detailed statistics are available for all projects and workflows, including class distribution, annotation progress, time spent annotating, review pass rate, and many more.

Automation

Without a doubt, Darwin’s ‘Auto-annotate’ polygon and semantic segmentation tool is the best one on the market. It’s not a superpixel or edge-detection approach but rather a generalized object detection and segmentation model which is domain-agnostic, meaning that you can run it on any type of imagery and it will still detect shapes with incredible accuracy.

This becomes even more awesome when applied to video: most annotation platforms only offer linear interpolation for bounding boxes (so a box moving at a steady speed across the frames). In Darwin’s case, you have access to a turbo interpolation engine which allows you to apply auto-annotation on polygons on keyframes and observe how the interpolated polygons on the frames in between smoothly morph into each other.

Finally, for very specific use-cases, V7 can train a custom model for polygon detection which you can use. The platform comes with an AutoML engine made to fine-tune Auto-annotate to new datasets, even very small ones. They also offer a service called “Neurons” which are applications combining a neural network architecture, a training pipeline, and augmentations flow which are suited for specific use cases: like microscopy, anomaly detection, and anonymization.