Tools we love Vol.8: Kili Technology

About the series

As you know, we at Humans in the Loop have a great love and appreciation of a well-designed annotation tool. After the great feedback on the reviews we published of our the best platforms on the market here and here, we decided that it’s time for a deep dive in some of our all-time favorites!

This article is the eighth from a series of 10 reviews which will be published each week. You can find the rest of the articles on our blog.

The whole series is based on the premise of transparency and honesty and none of these reviews are sponsored. They are just our way to give props to the best teams out there working on making annotation easier for AI teams, and to share some of the know-how that we have been accumulating over the past few years as a professional annotation company.

As in previous reviews, our parameters are:

• price
• functions
• project management
• automation

If you have additional questions or want to get in touch with us to beta test or feature your tool in an upcoming article, feel free to email us at hello@humansintheloop.org!

Kili Technology

Kili Technology is one of the most versatile tools on the market and it’s perfect for large companies which need to manage a variety of ML projects at once, including the annotation of images, videos, text, documents, and audio.

Established in France in 2018 by a team of experienced entrepreneurs, the platform provides for a wide range of customizable interfaces and an open ecosystem to plug your own interface and integrate in the data science workflow. Beyond being a simple annotation tool, it’s an entire suite of features for data and project management which is meant to help teams deliver AI faster.

Pricing is available by inquiry depending on the needs of the client (SaaS, on-premise, or a hybrid between the two where the platform is available as a Saas but the data is hosted on-premise) and there is a free online version available for anyone to try it out.

Features

One of the best things about Kili is that it provides a really wide range of labelling tools and formats. When creating a project, users can choose between image classification (single and multi-class), image object detection (bounding box, polygon and semantic), optical character recognition, PDF classification (single and multi-class), text transcription, text classification, named entities recognition/relation, as well as video annotation (frames classification or object tracking, suitable also for DICOM imagery).

The annotation interface is quite basic but it supports all basic tools for image labeling, such as points, polygons, lines, and semantic segmentation. There are options to ‘subtract’ or make holes in polygons as well as to toggle between drawing over or below existing objects when they intersect.

In terms of input, it accepts images, videos, which can be uploaded through the interface or through an integration with cloud providers (through a csv file). The export is only available as a JSON file in the Google API format.

Project management

Setting up the annotation interface

Kili is built with large corporations in mind so project management features are arguably its strongest side. There are quite a lot of handy features for dataset management (like asset/label search and filtering by status, labeler, honeypot or consensus score, etc.) These metrics are also available on an annotator level so annotators can be ranked by their stats and results.

There are two ways to validate the quality of annotation: one is consensus (measuring the agreement between several annotators who label the same asset) and the other one is honeypot (measuring the agreement between a golden standard annotation and the work of the annotator). Manual review is also enhanced with tons of features for filtering assets in order to easily find the problematic ones and correct them.

Finally, one of our favorite things about the platform is the fully-customizable labeling task interface (editable as a form or as a JSON text) where the admin can set up multiple labeling assignments for the same asset (e.g. first annotate the text with a bounding box and then transcribe it) which can be nested, made ‘required’ or ‘optional’, etc.

Automation

Kili can be used both in training phase (where the projects are set up and some sort of pre-annotation may be uploaded before labeling) or in production phase (where the trained model is being applied on the data and predictions with low confidence can be routed towards manual annotation).

There is no particular automation feature within the platform itself (e.g. smart segmentation) but it allows for users to easily connect their own models or use pre-annotations. In terms of pre-annotations, users can upload labels which are either predictions from your custom model, predictions from a weakly supervised learning framework or human-labeled data from other sources. The pre-annotations can also be set up as gold standard honeypot data.

In addition, the platform’s ‘AutoML’ feature allows for orchestrating online learning using different AutoML frameworks. Once the model has seen enough examples, it starts to make predictions and the rest of the process becomes reviewing pre-annotated data. Finally, the tool also lets you customize the labeling queue for active learning purposes, as well as for real-time annotation, through asset prioritization.

Kili Technology is a great solution for complex AI projects and what makes it stand out is the fact that it’s suitable for managing the entire lifecycle of the labeled data. All in all, highly recommended!