Evaluating expert curation in a baby milestone tracking app

Ayelet Ben-Sasson, Eli Ben-Sasson, Kayla Jacobs, Elisheva Rotman Argaman, Eden Saig

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Early childhood developmental screening is critical for timely detection and intervention. babyTRACKS1 is a free, live, interactive developmental tracking mobile app with over 3,000 children’s diaries. Parents write or select short milestone texts, like “began taking first steps”, to record their babies’ developmental achievements, and receive crowd-based percentiles to evaluate development and catch potential delays. Currently, an expert-based Curated Crowd Intelligence (CCI) process manually groups incoming novel parent-authored milestone texts according to their similarity to existing milestones in the database (for example, “starting to walk”), or determining that the milestone represents a new developmental concept not seen before in another child’s diary. CCI cannot scale well, however, and babyTRACKS is mature enough, with a rich enough database of existing milestone texts, to now consider machine learning tools to replace or assist the human curators. Three new studies explore (1) the usefulness of automation, by analyzing the human cost of CCI and how the work is currently broken down; (2) the validity of automation, by testing the inter-rater reliability of curators; and (3) the value of automation, by appraising the “real world” clinical value of milestones when assessing child development. We conclude that automation can indeed be appropriate and helpful for a large percentage, though not all, of CCI work. We further establish realistic upper bounds for algorithm performance; confirm that the babyTRACKS milestones dataset is valid for training and testing purposes; and verify that it represents clinically meaningful developmental information.

Original languageEnglish
Title of host publicationCHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450359702
DOIs
StatePublished - 2 May 2019
Event2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 - Glasgow, United Kingdom
Duration: 4 May 20199 May 2019

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2019 CHI Conference on Human Factors in Computing Systems, CHI 2019
Country/TerritoryUnited Kingdom
CityGlasgow
Period4/05/199/05/19

Bibliographical note

Funding Information:
We thank the babyTRACKS team members for their contributions to the system design and maintenance: Gal Agmon, Moriah Anochi, Tal Bussel, Shir Har-Noy, Rotem Malinovitch, Daniel Moran, and Naama Tzur. Research was funded by the European Research Council (under grant agreement 240258), the Israeli Science Foundation (grants 1501/14 and 1435/18), the US-Israel Binational Science Foundation (grant 2014-359) and the Hiroshi Fujiware Cyber Security Research Center at Technion.

Publisher Copyright:
© 2019 Association for Computing Machinery.

Keywords

  • Crowd wisdom
  • Curated crowd intelligence
  • Early childhood development

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Evaluating expert curation in a baby milestone tracking app'. Together they form a unique fingerprint.

Cite this