The goal of this project is for you to develop your skills in conducting and communicating original research. All interested students will be supported (and encouraged) by the instructor to submit this semester-long project for publication (e.g., to HCOMP, CHI, WACV, CVPR). All interested students will also be supported (and encouraged) by the instructor to turn this semester-long project into an exciting job opportunity in industry, whether via a summer internship or full-time job (e.g., Evolv Technology, Spare5, ...).
For your project, you will propose and investigate a novel idea about how to employ crowdsourced humans to help convert visual data into discoveries and innovations that can benefit society at large. This is an opportunity for you to enhance your expertise on a topic you feel most passionate about. As part of your project, you must use a crowdsourcing system to collect annotations describing images and/or videos.
Your final project will constitute 40% of your total class grade. Your project grade will be calculated as follows:
Your group: you are strongly encouraged to work with a partner (but with instructor approval can work alone)
Research ideas: submit one PDF per group that includes members' names and five ideas for your final project
After submitting your ideas, you will need to choose a five minute time slot that works for all group members to meet with the instructor to discuss the ideas. Available time slots will be posted after spring break.
How to identify possible final project ideas? Relevant topics include (but are not limited to):
Analysis of one or more crowdsourcing annotation collection methods discussed in class (must include empirical evaluation)
Implementation and evaluation of a new crowdsourcing algorithm or system to improve upon approaches discussed in class; e.g., propose a machine learning algorithm or a more user-friendly/efficient HCI design to collect annotations
A novel vision problem, supported by a new dataset and experiments demonstrating the value of the new problem
A system that employs crowdsourced annotations to tackle a vision problem; e.g., image/video retrieval or classification system
When choosing your topic, general guidelines are to:
Choose a problem you have an idea for how to solve
Choose a problem someone else cares about
Choose a problem that is not yet solved (know current literature!)
Choose a problem that you can objectively evaluate by tying it to a task
Revisit advice on how to read a research paper to evaluate your own ideas (e.g,. from the first week of assigned readings)
Project Proposal
The project proposal should:
Establish the research problem and novel idea your group will tackle for your course project.
Identify relevant related work. You will need to conduct a literature search that goes beyond the related works covered in class.
You will need to submit one PDF per group that is 1-2 pages long (excluding references). The paper should include each of the following:
Title
[Section 1] Introduction
Paragraph 1: Explain the motivation for your work; e.g., Why anyone should care? What are the desired benefits?
Paragraph 2: Explain why existing solutions are inadequate for the motivated problem; e.g., Is there a gap in the literature? Is there a weakness in existing approaches?
Paragraph 3: Explain what you are proposing, what is novel/new about your idea, and why you believe this solution will be better than previous solutions; e.g., Are you asking a new question, offering a greater understanding of a research problem, establishing a new methodology to solve a problem, building a new software tool, or offering greater understanding about existing methods/tools?
[Section 2] Related Work
Identify 3-5 related topics. Then, for each topic, cite 3-6 related papers (must include the bibliography). Finally, for each cluster of related works, give a 1-2 sentence explanation describing the key difference(s) of your proposed idea to the cluster of prior works. One way to format each topic is as follows:
Topic:
Reference 1
Reference 2
Reference 3
Reference 4
Reference 5
Our work is different from these works because...
Bibliography: this must be formatted correctly.
Please note that your proposed project is not a binding contract. You will continue to update and improve it as you learn more from your readings and/or feedback.
Project Outline Submission
The project outline should map out the entire project. You will be expected to:
Submit a detailed project outline that is 4-6 pages long (including references).
Demo your proposed crowdsourcing system; you should have a complete working prototype that you will use for your experiments.
You will need to choose a twenty minute time slot that works for all group members to meet with the instructor to demo the system and receive feedback on the paper. Available time slots will be posted before the due date.
For the project outline, you should submit one PDF per group. The paper should include each of the following:
Title
[Section 1] Introduction (improve upon the material from your proposal)
[Section 2] Related Work (improve upon the material from your proposal)
[Section 3] Methods - describe the implementation of your proposed idea (e.g., algorithm, system, etc.) in such a way that a 1) reader could reproduce your set-up and 2) understand why you made your design decisions; also, include a figure that illustrates your proposed idea (e.g., a screenshot of the user interface, a flowchart illustrating the steps in your larger system, etc)
[Section 4] Experimental Design - describe the experiments you plan to conduct to substantiate that your proposed idea is novel; e.g., What is the main purpose of each experiment? What datasets will you analyze? Who will be your crowd? What evaluation metrics do you plan to use? Do you make any comparisons with alternative approaches?
[Section 5] Experimental Results - report your expected results and what you might conclude from them. Also, include at least one placeholder figure and/or table for communicating your experimental findings. Finally, include one paragraph to explain what questions are not fully answered by your experiments as well as natural next steps for this direction of research.
[Section 6] Conclusions - summarize in one paragraph what you expect will be the take-away point from your work
[Section 7] Bibliography
Final Project Presentation
The final project presentation will involve:
Presenting a 5 minute recorded video about your project.
Taking 5 minutes to answer questions from the audience.
Your video should be no longer than 5 minutes and should do the following:
Motivate the problem your work is designed to solve.
(Very) briefly explain what other solutions are available and why they are not suitable.
Demo your idea, approach, and key design decisions.
Highlight key findings from your experiments and offer insights into what your work has taught us. Focus on finding 1-3 punchlines that explain why your work is exciting/valuable.
Please design your video for an audience who has not taken the class. In other words, your mom, dad, friend, or a potential employer should be able to watch it and understand what you did and why what you did is valuable.
At the end of class, two best project awards will be announced. The first best paper will be determined by popular vote from the members of the class (popular award). The second best paper will be awarded from a guest visitor to the class (faculty award).
Peer Evaluation
You will evaluate the presentation from every group (including your own) as the video is presented on paper surveys that will be collected at the end of class. The evaluations will not take time outside of class. The evaluations that you do for other students' projects will not affect your own grade, except that you will be penalized if you do not complete an evaluation for every group.
Final Project Submission
For your final project document, you will be expected to submit:
Your video presentation.
One PDF per group that is 6-8 pages long (including references).
All files used for your crowdsourcing system and all results from crowdsourcing (e.g., html, js, css, .input, .question, .properties, .results, etc).
For the final project report, you should submit one PDF per group that is a complete research paper (with the reading material from this course as good examples). The paper should include each of the following:
Title
Abstract - one paragraph summary of your paper describing the motivation, problem, conducted experiments, and experimental findings
[Section 1] Introduction (improve upon the material from your project outline)
[Section 2] Related Work (improve upon the material from your project outline; if you have not already, you should remove the bulleted structure you used in the initial proposal and instead have a paragraph form)
[Section 3] Methods (improve upon the material from your project outline)
[Section 4] Experimental Design (improve upon the material from your project outline)
[Section 5] Experimental Results (improve upon the material from your project outline)
[Section 6] Conclusions (improve upon the material from your project outline)