Wonder

Log in

Research Outline

Prepared for Alison M. | Delivered July 11, 2020

Views On Human-In-The-Loop AI

Goals

The research aims to identify a series of views on the concept of human-in-the-loop in artificial intelligence (AI) and machine learning. These insights demonstrate how humans could best partner with AI.

Early Findings

  • Paco Nathan from O'Reilly Media believes that the human-in-the-loop (HITL) concept is a design pattern, which combines both technical and management aspects. HITL builds on the semi-supervised machine learning model while a special case of this model is active learning. Active learning is a combination of machine learning models that automatically reach an agreement on how to label input data. When models cannot reach an agreement, the task is then passed onto humans, which is a major cost in the space of AI.
  • VSINGHBISEN states that when machines or computer systems cannot solve a problem, human intervenes by testing, training, tuning and validating algorithms. This creates a feedback loop for continuously improving the accuracy of algorithms, which is essential for developing AI-enabled devices, systems or machines that perform tasks or make predictions without human assistance. Human is essential in the process of developing instructions, such as bounding box annotation, which could be understood by machines. Data labelling is the first step of the algorithm training process.
  • Compare The Cloud wrote that HITL is optimal for machine learning, however, faces practical problems, such as risk and compliance issues involved in the outsourcing of HITL. This is because the business or market change often requires constant model refreshing. Optimized Learning, a new technology developed by Warwick Analytics, aims to reduce the need for human intervention and data labelling, through the use of a labelling machine, PrediCX.
  • Joint research by the University of Michigan and IBM revealed that some machine learning models "do not allow users to directly interact with the model," and the generalizability of models becomes questionable especially when there is limited labelled data. This problem can be solved by making abstractions and expressions easily understood by users and allow users "to assess, select and build on the model." The process of developing text analytics models could realize the greater potential of AI if it involves human or uses human-AI hybrid intelligence methods.
  • AnHai Doan from the University of Wisconsin-Madison discussed the imperative of "user communities that develop data repositories and tools," which forms part of the human-in-the-loop data analysis solutions (HILDA). AnHai recommends HILDA solutions should allow users to learn how to use different tools or algorithms in the process of solving data analysis problems, as "there is no single tool that can automate the entire process." For example, HILDA solutions are often developed to solve certain abstract problems, such as entity matching, whereas users care more about the accuracy of the matching or whether the new tool is better than their existing one than the actual matching. This creates a new problem of how to measure the accuracy of tools and what if the tools could not reach users' target accuracy.
  • Genpact's Sanjay Srivastava shared this view on how humans and machines co-evolve. There are three key stages of a problem-solving process, namely prediction, judgement/decision-making, and taking actions. The machine is in charge of prediction while the human is responsible for the latter two. The more accurate the machine learning models are, the more decision-making choices are available to humans. He recommends technology vendors focus on improving 20% of predictions that were inaccurate in the past so that machines can augment humans to make better decisions and take actions.
  • An article by James Wilson and Paul Daugherty published on Harvard Business Review stated that companies could not realize the full potential of AI if they only use AI to replace workers for performing tasks, such as equipment maintenance, because human-machine collaborative intelligence could be optimized through business process transformation to transform operation and workforce, among others. Workers should learn how to delegate tasks to machines, combine human and machine intelligence to achieve a better outcome than working alone, teach machines to gain new skills, and train machines to "work well within AI-enhanced processes."
  • A study by Fabio Massimo Zanzotto at the University of Rome Tor Vergata argued that Human-in-the-loop Artificial Intelligence (HitAI) is a fairer model of the AI systems by allowing to repay skilled or unskilled workers for extracting their knowledge from their daily work.
  • Regarding the role of humans in the process of developing AI models, InData Labs' Valeryia Shchutskaya stated that data scientists recommend the 80/19/1 approach which sees 80% of the work to be completed by machine, the human is in charge of 19% of the work, and the remaining 1% leave for randomness. Some typical use cases of the HITL concept is the development of chatbots for customer service, self-driving cars, and the recognition of traffic signs in photos, all of which requires human to recognize the content and train the machine learning models.