Final Project

Overview

Congratulations for making it to the end of the course! We hope this final project gives you a chance to apply what you’ve learned throughout the semester to reason about an application in computing that you’re excited about and/or want to learn more about. In doing so, you will need to struggle with the process of “just getting it working”, develop and test hypotheses about the system, perform some analysis on varying architectures, etc.

We envision two different directions in which you might want to take this final project: gem5-based, in which we will refer you to the gem5 resources for direction to get started; or CCV-based, in which you may find it useful to attend/refer to the final project gear-up. If you wish to use another tool for your final project, please run it by the course staff to ensure with a plan of action so that we can ensure that it seems feasible in the timeframe remaining in the semester.

Initial abstract and hypotheses (due 4/11, 11pm)

Once you decide on the emerging application you want to explore, write a short abstract and enumerate some hypotheses you want to evaluate. We’re not looking for you to have anticipated all possible nuances of your emerging application/hypotheses, but the more initial effort you put forth here, the better and more detailed our feedback to you will be. The feedback will also be an opportunity to help you calibrate what a “sufficiently complex” project is. We’re looking for hypotheses/emerging applications that are complex enough to demonstrate understanding of the nuances/themes of the course while still controlling for enough variables to make for “good science.”

As you’re planning ahead for your project, keep in mind that a large part of the work will involve setting up experiments in gem5/on Oscar. We don’t expect you to implement the code for your emerging application from scratch – it’s fine to use a well-documented repository here. Note that, in the past, some groups have explored tweaking this code or providing compiler hints as part of a hypothesis that tests the effects of developer intervention on performance.

Abstract

Please write a 5-7 sentence paragraph describing the project. In particular, this paragraph should describe:

  1. What your project is about (~1 sentence)
  2. At a high level, what software and hardware (real or simulated) you want to use for your project (2-3 sentences). In particular, it should be clear from this part of your abstract why the tools that you’re using are relevant to your proposed project.
  3. At a high level, what you are hoping to learn in your evaluation (~1 sentence)

Hypotheses

In more detail, enumerate several (3-5) hypotheses that your evaluation would ideally uncover with respect to the themes that we’ve learned about in the course. A well-formed hypothesis should make an educated claim about an expected, measurable behavior and identify a component of the architecture that would influence that behavior. For example, “I believe that X should outperform Y in terms of throughput for application Z because Z does a which works particularly well on X and not so well on Y because <theme/concept from the course!>

Things you could test for include (but are certainly not limited to!):

  • Whether the performance of your application changes when manipulating specific aspects of the memory hierarchy (cache size, types/number of caches, associativity, virtual memory setup).
  • Whether your application would benefit from dynamic approaches for ILP (deep pipelining, OOO, speculative execution, superscalar, hardware multithreading, etc). If so, which ones, and why? If not, why not? How can you test this?
  • Whether your application would benefit from static approaches for ILP (loop unrolling, VLIW, hand-written optimizations. etc). Note that you should be confident that you can check that your application does/does not take advantage of these.
  • Whether your application would benefit from hardware designed to take advantage of DLP (SIMD instructions, GPUs).
  • Whether the above changes would apply to any configuration/workload of your application, or if the application would benefit from a heuristic approach that chooses between hardware setups.
  • Whether the choice of ISA to compile your application to affects performance, usability, extensibility, etc. of the application
  • Whether an increase in performance of the application is worth an increase in energy/power, cost, compute time, etc. For applications such as machine learning, the answer might be different for the configuration used to train the model vs. the configuration used to run the model.

At least one of your hypotheses should touch on DLP, and at least one should touch on an impact other than performance. There are many ways to think about the impact of our computing setups (resource usage, effect on the developer, effect on the user, security, waste/reuse as new hardware becomes available, etc). Feel free to come by office hours to discuss ideas!

Part of effective experiment design is choosing what aspects of the setup to change (independent variables) and what aspects to keep constant (control variable). A well-written hypothesis will align with an experimental design that has enough control variables to truly test the effect of the independent variable. If you write a hypothesis that tests the effects of dynamic ILP and static ILP and DLP, that might be a sign to split your hypotheses up into multiple pieces. Also consider the complications of testing your hypotheses: if you want to compare how your application performs when compiled to different ISAs, how can you be confident that the specific compiler is actually capable of making the optimizations that your application needs? Again, please stop by office hours if you want to talk more about hypothesis design.

Implementation and final report (due 5/5, 11pm)

In the introduction (section 1) of your final report, include your revised abstract, hypotheses, and pointers to code (either a repository or a submission of the code you used to run your experiments.) Please also include the following sections:

2: Application

Briefly summarize the application you’re using and any challenges you had setting up the code to run. Document any resources you used and any changes you made to the application code. You don’t have to write pages and pages that describe your application, but a person who is not familiar with the application (but has a strong general background in CS) should be able to understand why your hypotheses make sense based on your description of the application.

In addition, working with open source projects can be a bit of a pain sometimes, and just getting stuff up and running can be tricky. Please enumerate the steps it takes to get your project up and running here so that we can empathize with any potential struggle you may endure!

3: Experimentation environment

Explain the details of your experimentation environment, with as much detail as you deem necessary to convince us that the environment matches up with the hypotheses you are testing. For example, some questions you may consider explaining: how many CPUs do you want to use? GPUs? Which types of these devices are you using? Do you think making this decision would matter? What does your memory hierarchy look like?

Finally, once your environment is together, please describe any scaffolding and/or run scripts that you put together to be able to perform your evaluation. This may include scripts to run stuff on CCV, in gem5, etc. or to directly call components from the open source project.

4: Evaluation

Now that your environment is together, time to test your hypotheses! For each hypothesis, describe the experiment(s) you ran to test it. A good experiment is one that controls for as many variables in the architecture as possible, and leaves only the thing that you want to test as variable (this is easier in a simulator than on real hardware).

Afterwards, summarize your evaluation. What result did you find? Did it confirm your hypothesis? Deny it? Leave results inconclusive? Are you convinced that the experiment truly evaluates your hypothesis, or are there sources of noise and uncertainty? Please try to justify your explanation in however much detail you find reasonable, but note that we don’t expect more than a paragraph.

Part 5: Summary

You worked hard on the project, so use this section to brag about what you learned

  1. What trade-offs did you explore?
  2. Do you feel like your hypotheses and experiments allow you to make a definitive architectural recommendation for your application? Why or why not?
  3. What else would you like to have explored, given more time? How would this have changed your view of your application and/or computer architecture?

Feel free to include any other thoughts that help highlight the insights you’ve gained from doing the work.