ErrCorrection-ICUDL.ppt

Error Detection and Correction in Metadata Nilu Prahallad, Zhenkun Zhou, Ting Zhang and Vamshi Ambati Carnegie Mellon University, USA and Zheijiang University, China Agenda Typical errors in Metadata Title Language Subject Other fields Correction Strategies Future Research directions Learning from Example  Universal Digital Library Large scale digital collections and archive - first of its kind 1.46 Million Books 21 different languages Large scale distributed collaboration - first of its kind Four countries - USA, China, Egypt, India 35 scanning locations 3000 people (or more…) What has kept us busy for last 1 year? We reached 1 M books at our last meeting in EGYPT Aggregating and Cleaning the metadata took us 1 complete year Metadata is the most important component in a Library, more so in a Digital Library Humans works in strange ways that computers don’t YET What is metadata? Information to identify a book Title, Author, Year, Language, Subject, Publisher, Copyright Dublincore standard Strcutural metadata - METS standard Why do we have problems in Metadata? Cataloguing in libraries by professionals is accurate but expensive $100 per book? At ULIB we want to get things done on a large scale but economically We are not limited by our visions, but our funds To Err is Human Nature of the Problems Data Entry problems Genuine confusion Careless entry Data Normalization Multiple languages and Standards Although not a problem, absolutely necessary for multilingual access What are the solutions on table? Manual effort Reliable but expensive and time consuming Original born digital metadata records Not all books have them, coordinating to get these is time-consuming Complete Automatic, Unsupervised Not reliable, more good than harm? Semi-supervised techniques Manual 20% , Automatic 80% We think we know how to work in such a scenario Going Semi-Automatic Computers are really good at Anomaly Detection We identify and perform automatic correction for most confident reco