Overview

The goal of this project is for you to develop your skills in conducting and communicating original research. All interested students will be supported (and encouraged) by the instructor to submit this semester-long project for publication (e.g., to HCOMP, CHI, WACV, CVPR). All interested students will also be supported (and encouraged) by the instructor to turn this semester-long project into an exciting job opportunity in industry, whether via a summer internship or full-time job (e.g., Evolv Technology, Spare5, ...).

For your project, you will propose and investigate a novel idea about how to employ crowdsourced humans to help convert visual data into discoveries and innovations that can benefit society at large. This is an opportunity for you to enhance your expertise on a topic you feel most passionate about. As part of your project, you must use a crowdsourcing system to collect annotations describing images and/or videos.

Your final project will constitute 40% of your total class grade. Your project grade will be calculated as follows:

Project Pre-proposal

The project pre-proposal should establish:

• Your group: you are strongly encouraged to work with a partner (but with instructor approval can work alone)
• Research ideas: submit one PDF per group that includes members' names and five ideas for your final project

After submitting your ideas, you will need to choose a five minute time slot that works for all group members to meet with the instructor to discuss the ideas. Available time slots will be posted.

How to identify possible final project ideas? Relevant topics include (but are not limited to):

• Analysis of one or more crowdsourcing annotation collection methods discussed in class (must include empirical evaluation)
• Implementation and evaluation of a new crowdsourcing algorithm or system to improve upon approaches discussed in class; e.g., propose a machine learning algorithm or a more user-friendly/efficient HCI design to collect annotations
• A novel vision problem, supported by a new dataset and experiments demonstrating the value of the new problem
• A system that employs crowdsourced annotations to tackle a vision problem; e.g., image/video retrieval or classification system

When choosing your topic, general guidelines are to:

Feel free to review project videos from last semester for inspiration: Spring 2017 Final Projects.

Project Proposal

The project proposal should:

1. Establish the research problem and novel idea your group will tackle for your course project.
2. Identify relevant related work. You will need to conduct a literature search that goes beyond the related works covered in class.

• Title
• [Section 1] Introduction
• Paragraph 1: Explain the motivation for your work; e.g., Why anyone should care? What are the desired benefits?
• Paragraph 2: Explain why existing solutions are inadequate for the motivated problem; e.g., Is there a gap in the literature? Is there a weakness in existing approaches?
• Paragraph 3: Explain what you are proposing, what is novel/new about your idea, and why you believe this solution will be better than previous solutions; e.g., Are you asking a new question, offering a greater understanding of a research problem, establishing a new methodology to solve a problem, building a new software tool, or offering greater understanding about existing methods/tools?
• [Section 2] Related Work

• Identify 3-5 related topics. Then, for each topic, cite 3-6 related papers (must include the bibliography). Finally, for each cluster of related works, give a 1-2 sentence explanation describing the key difference(s) of your proposed idea to the cluster of prior works. One way to format each topic is as follows:

  Topic:

  • Reference 1
  • Reference 2
  • Reference 3
  • Reference 4
  • Reference 5

  Our work is different from these works because...

• Bibliography: this must be formatted correctly.

Please note that your proposed project is not a binding contract. You will continue to update and improve it as you learn more from your readings and/or feedback.

Project Outline Submission

The project outline should map out the entire project. You will be expected to:

1. Submit a detailed project outline that is 4-6 pages long (including references).
2. Demo your proposed crowdsourcing system; you should have a complete working prototype that you will use for your experiments.

For the project outline, you should submit one PDF per group. The paper should include each of the following:

• Title
• [Section 1] Introduction (improve upon the material from your proposal)
• [Section 2] Related Work (improve upon the material from your proposal)
• [Section 3] Methods - describe the implementation of your proposed idea (e.g., algorithm, system, etc.) in such a way that a 1) reader could reproduce your set-up and 2) understand why you made your design decisions; also, include a figure that illustrates your proposed idea (e.g., a screenshot of the user interface, a flowchart illustrating the steps in your larger system, etc)
• [Section 4] Experimental Design - describe the experiments you plan to conduct to substantiate that your proposed idea is novel; e.g., What is the main purpose of each experiment? What datasets will you analyze? Who will be your crowd? What evaluation metrics do you plan to use? Do you make any comparisons with alternative approaches?
• [Section 5] Experimental Results - report your expected results and what you might conclude from them. Also, include at least one placeholder figure and/or table for communicating your experimental findings. Finally, include one paragraph to explain what questions are not fully answered by your experiments as well as natural next steps for this direction of research.
• [Section 6] Conclusions - summarize in one paragraph what you expect will be the take-away point from your work
• [Section 7] Bibliography

Final Project Presentation

The final project presentation will involve:

1. Presenting a 5 minute recorded video about your project.
2. Taking 5 minutes to answer questions from the audience.

• Motivate the problem your work is designed to solve.
• (Very) briefly explain what other solutions are available and why they are not suitable.
• Demo your idea, approach, and key design decisions.
• Highlight key findings from your experiments and offer insights into what your work has taught us. Focus on finding 1-3 punchlines that explain why your work is exciting/valuable.

At the end of class, a best project award will be announced. The best paper will be determined by popular vote from the members of the class.

You also must email the video to the professor at least one hour before class begins. The video MUST be in .mp4 format.

Peer Evaluation

Final Project Submission

For your final project document, you will be expected to submit:

1. One PDF per group that is 6-8 pages long (including references).
2. All files used for your crowdsourcing system and all results from crowdsourcing (e.g., html, js, css, .input, .question, .properties, .results, etc).

• Title
• Abstract - one paragraph summary of your paper describing the motivation, problem, conducted experiments, and experimental findings
• [Section 1] Introduction (improve upon the material from your project outline)
• [Section 2] Related Work (improve upon the material from your project outline; if you have not already, you should remove the bulleted structure you used in the initial proposal and instead have a paragraph form)
• [Section 3] Methods (improve upon the material from your project outline)
• [Section 4] Experimental Design (improve upon the material from your project outline)
• [Section 5] Experimental Results (improve upon the material from your project outline)
• [Section 6] Conclusions (improve upon the material from your project outline)
• [Section 7] Bibliography

iSchool Open House Presentation