After reading Human-in-the-Loop Machine Learning: a data-centric and suggestive book

This is a review after reading Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, which I received from Junya Ueda, one of the translators.

---

Recently, with the rise of generative AI and LLMs, I hear more often about how to collect and create "high-quality data". Not only for data used to train LLMs, but also for solving familiar problems, it has become natural to define the task needed for the problem, analyze data for it, and create data in order to solve social problems.

For many of these problems, you do not need to invent a new model yourself. Defining the task, collecting data, and training can often produce enough performance. This is the so-called data-centric way of thinking, focused on data.

There are many model-centric discussions in the world, about models and algorithms, while data is often treated as something evaluated on an already published dataset. Human-in-the-Loop Machine Learning is rare because it focuses on data and explains it in depth.

For what the book contains, it is best to read the table of contents and reaction summary, so please look there. Personally, the first part that strongly interested me was data sampling methods for active learning. It is easy to think that data near a clear linear decision boundary, where confidence is low, should be annotated. But the book discusses from many perspectives how to interpret uncertainty and diversity, and what strategies to use when deciding which data should be annotated. It is full of ideas, and implementations, that made me stop and think.

The second was collaboration with annotators. As the book says, "people management is essential." It is not at all a matter of saying "label it like this, thanks" and leaving the rest to them. The book strongly argues that you should treat collaboration with annotators like ordinary people management: how to make requests, what skills are needed, how to give feedback, how to remove annotator bias, how to handle uncertainty for each annotator, and how to communicate and provide feedback. Of course, it also contains many hints beyond people management, such as bias.

There were useful points everywhere, and because I read it carefully, it took about two months to finish. It was that interesting, and as someone who works with machine learning, I am truly glad I encountered this book.

Now that the LLM-driven AI era has begun, it will probably become normal to have AI evaluate data, use that feedback to create high-quality training datasets, and perform reinforcement learning. The original Human-in-the-Loop Machine Learning was written before GPT-4 appeared and before LLMs drew as much attention as they do now, but I think the book's viewpoint will be very useful when combined with LLMs too.