Overview

The goal of this project is for you to develop your skills in conducting and communicating original research. All interested students are encouraged to submit this semester-long project for publication (e.g., to HCOMP, CHI, WACV, CVPR). All interested students also are encouraged by the instructor to turn this semester-long project into an exciting job opportunity in industry, whether via a summer internship or full-time job.

For your project, you will propose and investigate a novel idea about how to employ crowdsourced humans to help convert visual data into discoveries and innovations that can benefit society at large. This is an opportunity for you to enhance your expertise on a topic you feel most passionate about. As part of your project, you must use a crowdsourcing system to collect annotations describing images and/or videos.

Your final project will constitute 40% of your total class grade. Your project grade will be calculated as follows:

Project Pre-proposal

The project pre-proposal should establish:

• Your group: you are strongly encouraged to work with a partner (but with instructor approval can work alone)
• Research ideas: submit one PDF per group that includes members' names and five ideas for your final project

After submitting your ideas, you will need to choose a 10 minute time slot that works for all group members to meet with the instructor to discuss the ideas. Available time slots will be posted.

How to identify possible final project ideas? Relevant topics include (but are not limited to):

• Analysis of one or more crowdsourcing annotation collection methods discussed in class (must include empirical evaluation)
• Implementation and evaluation of a new crowdsourcing algorithm or system to improve upon approaches discussed in class; e.g., propose a machine learning algorithm or a more user-friendly/efficient HCI design to collect annotations
• A novel vision problem, supported by a new dataset and experiments demonstrating the value of the new problem
• A system that employs crowdsourced annotations to tackle a vision problem; e.g., image/video retrieval or classification system

When choosing your topic, general guidelines are to:

Project Proposal

The project proposal should:

1. Establish the research problem and novel idea your group will tackle for your course project.
2. Identify relevant related work. You will need to conduct a literature search that goes beyond the related works covered in class.

• Title
• [Section 1] Introduction
• Paragraph 1: Explain the motivation for your work; e.g., Why anyone should care? What are the desired benefits?
• Paragraph 2: Explain why existing solutions are inadequate for the motivated problem; e.g., Is there a gap in the literature? Is there a weakness in existing approaches?
• Paragraph 3: Explain what you are proposing, what is novel/new about your idea, and why you believe this solution will be better than previous solutions; e.g., Are you asking a new question, offering a greater understanding of a research problem, establishing a new methodology to solve a problem, building a new software tool, or offering greater understanding about existing methods/tools?
• [Section 2] Related Work

• Identify 3-5 related topics. Then, for each topic, cite 3-6 related papers (must include the bibliography). Finally, for each cluster of related works, give a 1-2 sentence explanation describing the key difference(s) of your proposed idea to the cluster of prior works. One way to format each topic is as follows:

  Topic:

  • Reference 1
  • Reference 2
  • Reference 3
  • Reference 4
  • Reference 5

  Our work is different from these works because...

• Bibliography: this must be formatted correctly.

Please note that your proposed project is not a binding contract. You will continue to update and improve it as you learn more from your readings and/or feedback.

Project Outline Submission

The project outline should map out the entire project. You will be expected to:

1. Submit a detailed project outline that is 4-6 pages long (including references).
2. Demo your proposed crowdsourcing system; you should have a complete working prototype that you will use for your experiments.

For the project outline, you should submit one PDF per group. The paper should include each of the following:

• Title
• [Section 1] Introduction (improve upon the material from your proposal)
• [Section 2] Related Work (update the material from your proposal so this section includes a paragraph per topic instead of a bullet list per topic)
• [Section 3] Methods - describe the implementation of your proposed idea (e.g., algorithm, system, etc.) so that:
  • A reader could reproduce your set-up
  • A reader understand why you made your design decisions
  • It includes a figure illustrating your proposed idea; e.g., a flowchart illustrating the steps in your machine learning system(s)
• [Section 4] Experimental Design - describe the experiments you plan to conduct to substantiate that your proposed idea is novel. The requirement is to describe 2-3 experiments you plan to conduct and indicate the following for each experiment:
  • Main purpose: 1-3 sentence high level explanation of what is the purpose of the experiment
  • Dataset(s): 1-3 sentence description of the dataset
  • Baseline(s): describe status quo method(s) that you will use for comparison
  • Evaluation Metrics(s): which ones will you use and why?
• [Section 5] Experimental Results - for each experiment:
• Main finding(s): report your expected results and what you might conclude
• Include at least one placeholder figure and/or table for communicating your experimental findings
• Include one paragraph to explain what questions are not fully answered by your experiments as well as natural next steps for this direction of research
• [Section 6] Conclusions - summarize in one paragraph what you expect will be the take-away point from your work
• [Section 7] Bibliography

Final Project Presentation

The final project presentation will involve:

1. Presenting a 5 minute recorded video about your project.
2. Taking 5 minutes to answer questions from the audience.

• Motivate the problem your work is designed to solve.
• (Very) briefly explain what other solutions are available and why they are not suitable.
• Demo your idea, approach, and key design decisions.
• Highlight key findings from your experiments and offer insights into what your work has taught us. Focus on finding 1-3 punchlines that explain why your work is exciting/valuable.

At the end of class, a best project award will be announced. The best paper will be determined by popular vote from the members of the class.

Indicate in your submission on Canvas whether you are submitting a video in .mp4 format or a public URL for your video. If you are submitting a .mp4 video, then email that to me at danna.gurari@ischool.utexas.edu. If you are submitting a public URL, then indicate this URL in your Canvas submission.

Peer Evaluation

Final Project Submission

For the final project submission, you should submit a zip file which contains the following content:

• A PDF per group that is a complete research paper.
• Your final video presentation, either in .mp4 format or a URL link for its location on the web.
• All code for your project.

The paper should be 5-7 pages (excluding references) and include each of the following: