Skip to main content
Learning and Optimization for Sequential Decision Making
Learning and Optimization for Sequential Decision Making
Pages and Files
Lecture Notes and references
Add "All Pages"
Project proposals are due on March 9, 2016
Description of the project, at most 1-page
Project reports are due on May 2
Guidance on choosing a topic
You can pick any topic for the project you wish so long as there is some direct connection to making decisions in online or sequential manner. For your project, you can run an experiment, or you can think about a theoretical problem or algorithm, or you can do a blend of both. You can work individually or with a partner. Working as a pair is very much encouraged -- you are likely to learn more, have more fun, and accomplish more if working with a classmate.
Here are examples of possible types of projects:
Study one of the many related topics not covered in this course. Examples include: Active learning, Optimal stopping problem, Stochastic control, Recommendation systems. To get ideas, browse assigned or optional readings, or the chapters of our book that we did not cover, or explore the survey articles that will be provided during the course.
Run experiments with an algorithm on some datasets. Measure the error rates on separate datasets. Compare performance to another algorithm, or to a modification of the same algorithm. Or, compare performance to what is predicted by theory. For instance, here is how you can use
data sets available for supervised learning problems like classification, for example in the
UCI ML repository
, to test multi-armed bandit algorithms. To use a classification dataset to test an MAB algorithm, think of an MAB problem with actions as "pick a class", and reward 0 or 1 depending on whether it is the correct class. The aim is to maximize the total number of correct classifications.
Find an open problem, and attempt to make progress on it. A great place to look for these is the proceedings of recent COLT conferences (Conference on Learning Theory) which every year publishes short open-problem papers (some of which have monetary awards for solving). (Obviously, since these were published a while ago, it is possible that some of these will have already been solved. But that doesn't mean they can't be the starting point for a project.)
from a similar class on reinforcement learning provides some very good project ideas.
Think about what the consequences are of changing one of the sequential decision making models or changing the definition of regret. Does the problem become easier or harder? What are the general properties of the new model? Can you think of new algorithms for interesting problems in the new model?
Create an applet or other graphical program that interactively demonstrates the behavior of an algorithm that you have read about. See, for instance, this
. What insights about the algorithm did you gain by playing with your game or demo? (Be sure to keep the emphasis on the algorithm rather than, say, the graphical interface. Make sure that it is clear that you have learned something about sequential decision making.)
Build or apply an algorithm tailored to a particular application coming from some other field such as internet advertising, clinical trials, assortment optimization, dynamic pricing, recommendation systems, robot navigation, automated game playing.
Come up with your own idea! Be original and creative.
Writing a final report
The end result of your project should be a written report clearly and concisely describing what you did, what results you got and what the results mean. If you are working individually, your report should be 3-5 pages long. If you are working with a partner, the two of you together should submit a single report, which may be 4-7 pages long. The report should use 12pt font, 1-inch margins, and single spacing. The page length limits do not include figures or citations. Papers that vary from these guidelines risk receiving a grade deduction and/or some sections not being read.
Your report should follow the general format of a scholarly paper in this area. You should write your report as clearly as possible in a manner that would be understandable to a fellow student of this course. In other words, you should not assume that the reader has background beyond what has been covered in class (as well as the prerequisites of the course).
Your report should begin by describing the problem you are studying, some background (what has been done before) and the motivation for the problem, i.e., why it is worth studying. Previous work and outside sources should be cited throughout your report in a scholarly fashion following the style of academic papers in this area.
Next, you should clearly explain what you did, both from a high level, and then in more detail. For an experimental paper, you should explain the experiments in enough detail that there is a reasonable possibility that a motivated reader would be able to replicate them. You also should outline some of the theory underlying the algorithms you are studying. State your results clearly, and think about graphical tools you can use to make your results clearer (a table of numbers is usually less compelling than a graphical representation of the same data). Look through published papers for ideas. For a theoretical paper, the model and other mathematical details should be explained well enough for the results to be stated with mathematical precision and clarity.
In every case, be sure to explain the meaning of your results. Don't just give a table of results or a dry mathematical formula. Explain what the results mean, and what conclusions can be drawn from them. Again, do all this in a way that would be understandable and interesting to a fellow student of this course. What did you expect to find? What did you find instead? What are the implications? If you found something surprising, can you think of how it might be explained?
What you will be graded on:
Projects will be graded along the following dimensions:
originality and creativity
theoretical component and/or experimental design and execution
discussion and interpretation of results
writing of the final report, including clarity, completeness and conciseness
[contents of this page were derived from
this excellent guidance
from Rob Schapire on choosing projects for his course at Princeton.]
help on how to format text
Turn off "Getting Started"