Nbgrader notes

From Earlham CS Department
Jump to navigation Jump to search

Notes

  • There is a user nbgrader (password is on Google Drive) on tools.cs.earlham.edu that must be used for all setup, assignment creation, grading, etc.
  • nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
    • This documentation is extremely good, and should generally be referred to first.
  • Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
  • Making assignments
  • Keyboard input does not work with auto-grading, grading generally?
  • Keyboard input may work with manual grading now
  • Notes to students (in each assignment)
  • Make sure your solution is named the same as the assignment.
  • Don’t use spaces in file names (ever). Avoid using special characters aside from -
  • Find our tools for working with nbgrader here.
  • At the finish of every semester, make a copy of the cs128 directory and name it as follows: sem_year_cs128 (with the corresponding semester as fall or spring) and move it to the cs128Archives directory.

Please also read:

Beginning of Term Notes: Setting up a New Class

  1. At least a week before the term starts, send the admins (admin [at] cs [dot] earlham [dot] edu) the list of student names and email addresses so the admins can set up accounts for them.
  2. Log in as nbgrader. From within the terminal, use nbgrader quickstart classname to create a new class, where classname is the name of the class (something like cs128-f21). Folders for cs128 classes are called cs128-f20 or cs128-s21, where f and s are "fall" and "spring" and the two digits are the year. This command will create a new folder for the class, plus a source folders and config file within it.
  3. Copy the contents of the source directory from last term’s course into the new directory's source directory (or whatever material from previous courses that you plan to use.) The source directory is where assignments are stored before they are released to students.
  4. The config file at ~nbgrader/.jupyter/nbgrader_config.py should be edited to update the two references to the old class so that they refer to the new class.
  5. You may need to restart nbgrader’s server at this point. From the jupyter interface, select “Control Panel,” then “Stop My Server.” After a moment, you will be able to select “Start Server.”
  6. Navigate to /srv/nbgrader/exchange. Move the previous term’s directory into the old/ directory. This will prevent students from ever seeing the old labs (and possibly getting them by mistake, rather than getting the new labs.) -- This step may now be unnecessary, but it shouldn’t hurt.
  7. Update the student name list in the file student-names-file. This file lives in nbgrader’s home directory. student-names-file-sample contains an example of what this should look like. If you have a .csv downloaded from The Heart, tools in addUsers.ipynb can be used to convert produce a student-names-file using:
    1. make_student_names_file(“csv_file_from_the_heart.csv”, “student-names-file”)
  8. You shouldn’t need to do this now, but you can use other tools in addUsers.ipynb to add and remove individual students from the database. For this to work, note that addUsers.ipynb must be inside the course directory for the class; if it is run from nbgrader’s home directory, this will not work. This is useful if a student joins late and you just need to add one student. Dave believes that this also works to change the student’s name.
  9. Currently, the student-names-file file is used to build up the student db. You can import names from the list using nbgrader db student import student-names-file The content in the student-names-file should follow the style required by nbgrader (the first line needs to be as follows, and the last line needs to be empty). If you generated student-names-file using the instructions in the previous steps, it should already look like this.
    1. Line 1: id,last_name,first_name, email
    2. Line 2: smaizaz18,Aizaz,Shah, smaizaz18@earlham.edu
  10. (Optional) You can use nbgrader db student list to check if the students has been added successfully
  11. (Optional) You can use nbgrader db student list --course-dir=”~/<course-directory>” to check the student list for a specified course.
  12. This step and the remaining steps may no longer be necessary. Copy the two feedback files, which have names like feedback-summer19.sh, and return_feedback-summer19.py, into the course directory for the new course.
  13. Update the names of those two files to refer to the correct term.
  14. Edit the feedback shell script with the name of the feedback .py file.
  15. Edit the feedback .py file by changing students = ["barbeda",...] to a list of the students in the class.

Managing Assignments

Preparing an Assignment

  1. Assignments can be found in the source directory within the course directory.
  2. Assignments should be updated before release - once an assignment is released, further changes to the source copy will not be reflected in the version that students get.

Releasing an Assignment

  1. Log in to the Jupyter web interface at tools.cs.earlham.edu as nbgrader.
  2. In Jupyter, navigate to the “Formgrader” tab.
  3. Choose an assignment you would like to release. Any assignment in the source directory should be visible here. If it's not, something has gone wrong during the setup process.
  4. “Generate” => “Release”
    1. Both of these should pop up log output boxes. Neither log output should contain errors, just information.

Student Work

  1. Once an assignment has been released, students can access it.
  2. When they log into Jupyter, they should have access to an Assignments tab. They can select an assignment to fetch from that tab. This places the assignment in a new directory in their home directory. They can complete their work in the workbook in that directory.
  3. When they are done, they can submit their work in the Assignments tab. They can submit work multiple times, but generally students just submit the work when it’s done.

Collecting And Grading Student Work

  1. Once students have submitted their work, you can Collect submissions using the “Collect” button in “Formgrader” tab. You can collect submissions as many times as necessary.
  2. There are several ways to Autograde the submission.
    1. Method A) In the terminal, run nbgrader autograde <Lab_Name>. To autograde one student’s submission, use nbgrader autograde <Lab-name> --student <student-id>
    2. Method B) From within the formgrader, after the lab has been collected, click the number of submissions for that lab, then click Autograde for each student you want it to autograde. This is useful if a student turns in work late.
  3. Once Autograding is complete, manually grade any parts of the assignments that can’t be autograded. Feedback is also added to student work at this time, and this is where partial credit can be awarded. In the formgrader, choose “Manual Grading” on the left. Then, select the labs and assignments you wish to manually grade. You can also adjust scores produced by the autograding system, in order to award partial credit. (Or even full credit, if the student’s work is correct but was mislabeled by the autograding system.)

Releasing Feedback

  1. After coursework has been graded, you can generate and release feedback using the appropriate buttons in the Manage Assignments screen within the formgrader. Students can then fetch the feedback in a process similar to how they fetch assignments.

Alternate method

Feedback can also be generated and distributed using the files feedback-summer19.sh and return_feedback-summer19.py (The names of these will change each term.) These must be located in the directory for the course.

To generate and return feedback for all students in the current course of lab <Lab-Name> $ ./feedback-summer19 <Lab-Name> feedback-summer19: this file read the student list from the file student-names-file, and generate feedback file for each student, then it will run the return_feedback-summer19.py return_feedback-summer19.py: this file will copy the feedback folder to the students’ home directory.

To return feedback for a specified student: $ nbgrader feedback --student <student-id> --assignment <LAB_NAME> --force $ sudo python return_feedback-f19.py <LAB_NAME> --student <student-id>

Make sure the lab-names.txt file in the root nbgrader directory is up to date (i.e. the names of the labs are correct). They need to be correct to use MOSS.

Upgrading nbgrader

  • Grab latest release of nbgrader from github along with further install information.
  • Copy source code zip to username@home.cs.earlham.edu
    • scp source-code.zip username@home.cs.earlham.edu:~/
  • login as nbgrader on jupyter and then become root.
    • sudo su -
  • If going through a shell, ssh tools from home.
  • Backup exsiting nbgrader install
    • /mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader /nbgrader-timestamp/nbgrader
  • Uninstall current version of nbgrader.
    • Ideally conda remove nbgrader should work, if you get NoPackagesFoundError try uninstalling with pip
    • pip uninstall nbgrader
  • Install new nbgrader with conda and specify the anaconda python 3.5 environment
    • conda install -c conda-forge nbgrader -n py35
  • Restart nbgrader user server from Jupyter control panel.
    • May take a couple minutes for jupyter come back online
  • Ability to switch between courses in assignment list extension #563
    • Needs to be added manually until it gets merged into nbgrader master.

Multiple Simultaneous Graders

Using multiple graders works as follows: One user should log in as nbgrader and run nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like: 

c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]

In the nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.

To tell if formgrade is running, do a ps auxw | grep formgrade | grep -v

Then check to see if there’s one running from: 

/mnt/lovelace/software/anaconda/envs/py35/bin/python
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade

If there is, then somebody already has the formgrade session going, and you can just to to the web address.

Digging into nbgrader

Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).

The main nbgrader source code directory is here: 

/mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader/

There are other files located here as well: 

/mnt/lovelace/software/anaconda/pkgs/nbgrader-0.3.3-py35_0/lib/python3.5/site-packages/nbgrader/.

If you’re specifically looking for the assignment list extension, those are in nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.

If you get a message that looks like this: 

Traceback (most recent call last):
     File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
       return_value = func(*args, **kwargs)
...
...
AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'

Conda may have gone out of sync. Try: 

CONDA_SSL_VERIFY=false conda update pyopenssl

Authoring Worksheets

  • If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
  • Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.

Removing Students

In a python notebook, run the remove_student(student_id) command. It will delete the student along with the student's submissions. Don't forget to import Gradebook from nbgrader.api and to create a gradebook object:

gb = Gradebook("sqlite:///gradebook.db")

gb.remove_student("syweiss15") <ref>https://nbgrader.readthedocs.io/en/stable/api/gradebook.html</ref>

Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Nbgrader_notes&oldid=17991"