Human-centered Machine Learning (2018 Spring)

Course staff

Insturctor: Chenhao Tan contact
Office hours: 3:30-4:30 on Monday, 4:30-5:30 on Wednesday, or by appointment (ECES 118A)

Logistics

• Location and time: ECES 112, 1:00-2:15pm on Mondays and Wednesdays
• Syllabus (Must READ if you are taking the course)
• Piazza

Prerequisites

Mathematical background: Please finish these questions and submit by the second week of the semester. This does not contribute to your final scores, but it helps both yourself and the instructors to know your mathematical background.

Programming: You are expected to know how to program and finish a final project.

Course evaluation

• proposals (2 times, 20%)
• course presentation (10%)
• final project (50%)
• peer feedback (15%)
• participation (5%)

Schedule

• Week 1:

  • Jan 17, Introduction
    Additional reading:
    • Human-centered Machine Learning: a Machine-in-the-loop Approach, Chenhao Tan.

• Week 2: You are not so Smart

  • Jan 22, Human Decisions and Machine Predictions. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan. Quarterly Journal of Economics, 2018.
  • Jan 24, The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter. Chenhao Tan, Lillian Lee, Bo Pang. In Proceedings of ACL, 2014.
    Additional reading:
    • Judgment under Uncertainty: Heuristics and Biases. Amos Tversky and Daniel Kahneman, Science, 1974.
    • Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey, Journal of Experimental Psychology: General, 2014.
    • Podcast: You are not so smart
    • Thinking, Fast and Slow. Daniel Kahneman. 2011.

• Week 3: Open the Black Box (Interpretable Machine Learning)

  • Jan 29, Why should I trust you?: Explaining the Predictions of Any Classifier. Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. In Proceedings of KDD, 2016.
  • Jan 31, Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Been Kim, Rajiv Khanna, Oluwasanmi Koyejo. In Proceedings of NIPS, 2016.
    Additional reading:
    • DeepXplore: Automated Whitebox Testing of Deep Learning Systems, Pei, Cao, Yang and Jana. In Proceedings of SOSP, 2017.
    • Interactive and Interpretable Machine Learning Models for Human Machine Collaboration, Been Kim, PhD thesis.
    • The Mythos of Model Interpretability, Zachary C. Lipton.
    • Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola. In Proceedings of EMNLP, 2016.
    • Learning Explanatory Rules from Noisy Data. Richard Evans, Edward Grefenstette. Journal of Artificial Intelligence Research, 2018.
    • Network Dissection: Quantifying Interpretability of Deep Visual Representations. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. In Proceedings of CVPR 2017.

• Week 4: Human-in-the-loop Machine Learning

  • Feb 5, Interactive Machine Learning. Jerry Alan Fails, Dan R. Olsen, Jr. In Proceedings of IUI, 2003.
  • Feb 7, Learning with Humans in the Loop. Thorsten Joachims. ECML keynote talk, 2013.
    Additional reading:
    • Visual Recognition with Humans in the Loop. Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, Serge Belongie. In Proceedings of ECCV, 2010.
    • Flock: Hybrid Crowd-Machine Learning Classifiers. Justin Cheng, Michael S. Bernstein. In Proceedings of CSCW, 2015.
    • Power to the People: The Role of Humans in Interactive Machine Learning. Saleema Amershi, Maya Cakmak, William Bradley Knox, Todd Kulesza. AI Magazine, 2014.

• Week 5: Machine-in-the-loop Interactions

  • Feb 12, Principles of Mixed-Initiative User Interfaces. Eric Horvitz. In Proceedings of CHI, 1999.
  • Feb 14, Brainstorming
    Additional reading:
    • Natural Language Translation at the Intersection of AI and HCI. Spence Gree, Jeffrey Heer, Christopher D. Manning. In Queue, 2015.
    • Beyond binary choices: Integrating individual and social creativity. Gerhard Fischer, Elisa Giaccardi, Hal Eden, Masanori Sugimoto, Yunwen Ye. International Journal of Human-Computer Studies, 2005.
      First proposal due on Feb 16

• Week 6: First Proposal

  • Feb 19, presentation & discussion
  • Feb 21, presentation & discussion
    First proposal peer feedback due on Feb 23

• Week 7: Machine-in-the-loop application: Creative Writing

  • Feb 26, Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. In Proceedings of IUI, 2017.
  • Feb 28, Creative Help: A Story Writing Assistant. Melissa Roemmele, Andrew S. Gordon. In Proceedings of ICIDS, 2015.
    Additional reading:
    • Inside Jokes: Identifying Humorous Cartoon Captions. Dafna Shahaf, Eric Horvitz, Robert Mankoff. In Proceedings of KDD, 2015.
    • Hafez: an Interactive Poetry Generation System. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. In Proceedings of ACL, 2017 (Demo Track).

• Week 8: Machine-in-the-loop application: Decision Making

  • Mar 5, Assessing Human Error Against a Benchmark of Perfection. Ashton Anderson, Jon Kleinberg, Sendhil Mullainathan. In Proceedings of KDD, 2016.
  • Mar 7, Predicting the knowledge–recklessness distinction in the human brain. Iris Vilares, Michael J. Wesley, Woo-Young Ahn, Richard J. Bonnie, Morris Hoffman, Owen D. Jones, Stephen J. Morse, Gideon Yaffe, Terry Lohrenz, and P. Read Montague. PNAS, 2016.
    Additional reading:
    • Prediction Policy Problems. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, Ziad Obermeyer. American Economic Review, 2015.
    • The Tyranny of Data? The Bright and Dark Sides of Data-Driven Decision-Making for Social Good. Bruno Lepri, Jacopo Staiano, David Sangokoya, Emmanuel Letouzé, and Nuria Oliver. Transparent Data Mining for Big and Small Data, 2017.
      Second proposal due on Mar 9

• Week 9: Second Proposal

  • Mar 12, presentation & discussion
  • Mar 14, presentation & discussion
    Second proposal peer feedback due on Mar 16

• Week 10: Machine-in-the-loop application: Categorization & Education

  • Mar 19, Teaching Programming with Gamified Semantics. Ian Arawjo, Cheng-Yao Wang, Andrew Myers, Erik Andersen, François Guimbretière. In Proceedings of CHI, 2017.
  • Mar 21, Improving Human-Machine Cooperative Classification via Cognitive Theories of Similarity. Brett D. Roads, Michael C. Mozer. Cognitive Science, 2016.
    Guest instructor: Mike Mozer.
    Additional reading:
    • Faster Teaching via POMDP Planning. Anna N. Rafferty, Emma Brunskill, Thomas L. Griffiths, Patrick Shaftod. Cognitive Science, 2015.
    • Self-Directed Learning and the Sensemaking Paradox. Kirsten R. Butcher & Tamara Sumner. Human-Computer Interaction, 2011.
      Team formation & project choice due on Mar 19

• Week 11: Spring break

• Week 13: Machine-in-the-loop application: Accessibility

  • Apr 2, Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS. Shaun K. Kane, Meredith Ringel Morris. In Proceedings of DIS, 2017.
  • Apr 4, Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service. Shaomei Wu, Jeffrey Wieland, Omid Farivar, Julie Schiller. In Proceedings of CSCW, 2017. Additional reading:
    • Automatically generating tailored accessible user interfaces for ubiquitous services. Julio Abascal, Amaia Aizpurua, Idoia Cearreta, Borja Gamecho, Nestor Garay-Vitoria, Raúl Miñón. In Proceedings of ASSETS, 2011.

• Week 12: Midpoint project presentation

  • Apr 9, free time to work on your projects
  • Apr 11, presentation
    Project peer feedback due on Apr 13

• Week 14: Machine-in-the-loop application: Robotics

  • Apr 16, Human-robot proxemics: Physical and psychological distancing in human-robot interaction. Jonathan Mumm, Bilge Mutlu. In Proceedings of HRI, 2011.
  • Apr 18, Robotic Assistance in Coordination of Patient Care. Matthew Gombolay, Jessie Yang, Bradley Hayes, Nicole Seo, Samir Wadhwania, Zixi Liu, Tania Yu, Neel Shah, Toni Golen, Julie Shah. In Proceedings of RSS, 2016. Additional reading:
    • Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaboration. Stefanos Nikolaidis, Swaprava Nath, Ariel D. Procaccia, and Siddhartha Srinivasa. In Proceedings of HRI, 2017.
    • Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences. Robert Pinsler, Riad Akrour, Takayuki Osa, Jan Peters, and Gerhard Neumann. In Proceedings of ICRA, 2018.

• Week 15: Fairness, Accountability, and Transparency

  • Apr 23, Inherent Trade-Offs in the Fair Determination of Risk Scores. Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan. In Proceedings of ITCS, 2017.
  • Apr 25, Algorithmic decision making and the cost of fairness. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq. In Proceedings of KDD, 2017.
    Guest instructor: Raf Frongillo.
    Additional reading:
    • Equality of opportunity in supervised learning. Moritz Hardt, Eric Price, Nathan Srebro. In Proceedings of NIPS, 2016.
    • European Union regulations on algorithmic decision-making and a "right to explanation". Bryce Goodman and Seth Flaxman.

• Week 16: Final Project Presentation

  • Apr 30, presentation
  • May 2, presentation
    Final project report due on May 2