Nbgrader notes

Notes

• There is a user nbgrader (password is on Google Drive) on tools.cs.earlham.edu that must be used for all setup, assignment creation, grading, etc.
• nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
• Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
• Making assignments
• Keyboard input does not work with auto-grading, grading generally?
• Keyboard input may work with manual grading now
• Notes to students (in each assignment)
• Make sure your solution is named the same as the assignment.
• Don’t use spaces in file names (ever). Avoid using special characters aside from -
• Find our tools for working with nbgrader here.
• At the finish of every semester, make a copy of the cs128 directory and name it as follows: sem_year_cs128 (with the corresponding semester as fall or spring) and move it to the cs128Archives directory.

Upgrading nbgrader

• Grab latest release of nbgrader from github along with further install information.
• Copy source code zip to username@home.cs.earlham.edu
  • scp source-code.zip username@home.cs.earlham.edu:~/
• login as nbgrader on jupyter and then become root.
  • sudo su -
• If going through a shell, ssh tools from home.
• Backup exsiting nbgrader install
  • /mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader /nbgrader-timestamp/nbgrader
• Uninstall current version of nbgrader.
  • Ideally conda remove nbgrader should work, if you get NoPackagesFoundError try uninstalling with pip
  • pip uninstall nbgrader
• Install new nbgrader with conda and specify the anaconda python 3.5 environment
  • conda install -c conda-forge nbgrader -n py35
• Restart nbgrader user server from Jupyter control panel.
  • May take a couple minutes for jupyter come back online
• Ability to switch between courses in assignment list extension #563
  • Needs to be added manually until it gets merged into nbgrader master.

First Time Setup for a Class

Set up file space for the class

• Create a directory for your class in ~nbgrader, e.g. mkdir pcs290
• cd to that directory and create a config file. A template file appropriately called nbgrader_config_template.py exists in nbgraders home directory, and you can copy and adapt that one for the course: cp ../nbgrader/nbgrader_config_template.py pcs290/nbgrader_config.py . ; $EDITOR nbgrader_config.py
• Alternatively you can use nbgrader’s tool to generate a default one and then figure-out all the bits that need to be changed/added for our local setup. nbgrader --generate-config ; $EDITOR nbgrader_config.py
• Create a source directory within the new course directory, e.g. ~nbgrader/pcs290/source All of the assignment directories should be created in this directory (either from the command line or from the Jupyter/iPython interface).
  • N.B. This is also the directory that you should run all of the commands to autograde, collect, etc. assignments for this class.

Add students to the class

The config file now requires a list of all students and assignments for a class -- see the test config file for an example of how to do this.

• Add the users to the database for the class. nbgrader uses a SQLite database to hold all of the students and assignments. This database is called gradebook.db and it's in ~nbgrader/pcs290 (or whatever your class directory is). Each class has it’s own database.
• Create a file that lists the first name, last name, and email address for the students in the class (cut the three columns from an XLS export from TheHeart).
• use generate with the add.student lines active
• There's a script in ~nbgrader/cs128 called addUsers.ipynb that will connect to the database and add students. You must copy that script.
• Activate nbgrader for each student in the class
• Using the list of users from above and generate.. with the activate line active
• Create a file called return_feedback.py in the course directory (copy the one found in the test folder and modify it if necessary).
• A student can run /home/nbgrader/set-me-up if the assignments tab isn't showing up for them. They'll need to restart their server after that.

Creating an Assignment

• In ~nbgrader/cs128/source create a directory for the assignment you are going to make. In that directory you can begin creating notebooks for those assignments.
• If you log into the notebook server at jupyter.cs.earlham.edu as nbgrader you will be able to create assignments. All assignments should be placed in the source/ directory of the cs128/ directory. There are assignments and then are notebooks that can go with those assignments.
• To create a notebook: select New --> Python 3. In the Cell Toolbar: dropdown list, select Create Assignment. Make sure you rename the notebook to something appropriate for the assignment. You can create multiple problems in the notebook, give them an id, set the points, and then set the grading type. If you want to manually grade them you'll set it to Manually Grade. There's an autograde option, and also where you can set the solutions to the problems but for that information go to nbgrader.readthedocs.org. This file is just an introduction to nbgrader.
• Don’t use the RawNbConvert cell type.
• Save the notebook and exit.

Assigning Assignments

• Note that if you are assigning an assignment with manually graded cells to check beforehand that everything is running smoothly with the point system on the formgrade. In the past there have been issues with this so you may want to double check that it is working correctly.
• Now we will assign the assignment to the students. Running nbgrader assign (not in the source directory, in ~/cs128) will look in the source/ directory, which is where you should've just saved the assignment you created. The assignment must be in the database before you can assign it (this means in must be in the config file), but there's a --create option with nbgrader that will create it as we assign. Run:

# this needs to be done any time you start a new session
cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

nbgrader assign --create ASSIGNMENT-NAME

• If you get an error like No assignment called '%s' exists in the config you need to add the assignment to the config file.
• If you get an error like No assignment called 'Lab-02-Exp' exists in the config you need to re-run the export CONFIGPROXY_AUTH_TOKEN... line from a few lines up.
• That command will put the assignment in the release/ directory. To release that assignment to students, type: nbgrader release ASSIGNMENT-NAME
• The assignment should've been release to the students. Once they log into the notebook server, if they have the extension installed, they will be able to click the Assignments tab, and fetch the assignments that are released. They can download and submit them as well.
• N.b. - If you want to make a change to an assignment after releasing it, you can overwrite what’s already released with --force. However, a student who has already fetched the assignment will not have what they fetched altered, and there is not a way to remove the assignment through the Jupityr interface. Throught the terminal, they can do a rm -r ASSIGNMENT-NAME as normal, and after doing that, they can re-fetch the assignment through the Jupityr interface or with nbgrader fetch ASSIGNMENT-NAME --course "COURSE-NAME"

Temporary Status Quo 10.16.17

If you get an error that says TypeError: 'Assignment' object is not subscriptable: - Assign the assignment in the terminal, using the --no-db flag and the --create flag, like this:

  nbgrader assign Lab7-Files-F17 --create --no-db

Collecting and grading an assignment

This is almost all entirely out of date since the Fall 2017 update

Basic Steps: When doing these steps, make sure you are in the course's directory and not the source directory. For cs128, you want to be in the cs128 directory.

1. Run the export

export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TO KEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterh ub/jupyterhub_config.py)

2. Collect the assingments nbgrader collect ASSIGNMENT_NAME

3. Autograde nbgrader autograde ASSIGNMENT_NAME

4. Formgrade nbgrader formgrade Opend browser to https://jupyter.cs.earlham.edu/hub/nbgrader/cs128 In the upper right hand corner, you can change the view to view students or assignments.

• Before you can manually grade the assignments by running nbgrader formgrade, you have to run nbgrader autograde even if the assignments were set to be manually graded (don't ask me why, I don't know right now). Now if you run nbgrader formgrade again and go to the web browser, you should be able to see all of the assignments and students, and submissions. You click on an assignment to grade it.
• nbgrader formgrade is the command used to manual grade notebooks. This runs in the browser and connects with JupyterHub. You will be able to see all of the students and assignments release and whether or not they have submitted anything. Run nbgrader formgrade and then direct your browser to jupyter.cs.earlham.edu/hub/nbgrader/cs128. You will see assignments and students.
• collect submitted assignments: cd “course” (ie cs128); nbgrader collect LabName

• This populates the ‘submitted’ folder

• If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to go back and run

  cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

4. autograde the submissions: cd “course”; nbgrader autograde LabName # this can take a while

• If there are problems with the student’s submission (e.g. spaces in filenames, extra notebooks), they will manifest themselves here with a long error message. Use mv or rm depending on the nature of the problem.
• Just keep doing the up arrow, return, fix something, up arrow, return, fix something loop until autograde does not return any errors.
• If you can’t find a specific user file reference in the error message see b. Above... autograde populates the ‘autograded’ folder

5. manually grade the submissions: cd “course”; nbgrader formgrade then opon your $BROWSER to jupyter.cs.earlham.edu/hub/nbgrader/cs128

• In the upper right hand corner is a drop-down that will let you sort by student or by submission.
• If you receive “address in use” ps auxw | grep formgrade and kill -9 ...

• If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to

  export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

• If you get errno 13 permission denied:

  chown -R nbgrader:nbgrader /run/user/11040 (each time you login? seems to be owned by the underlying UID:GID)

6. OR, alternatively, run ../formgrade.sh from within a given course directory like cs128, cs256, etc.
7. ssh home.cs.earlham.edu
8. ssh tools.cs.earlham.edu
9. sudo su - nbgrader
10. cd <course>

11. ../formgrade.sh

12. In general, if you ever want to do one of these operations for only one student (because they were late or something, you can use --student [studentid]

13. If an assignment is missing from formgrade (this can happen if a student does an update), you can collect the assignments again, and then force the autograder to grade the new one with nbgrader autograde [assignment-name] --student [studentid] --force

Arguments: ("submitted/aabdul15/Lab11-1/Charlie's solution.ipynb",)


write the header row - “email, lab10-2, lab11-0, lab11-1, lab11-2” 
read cs128/submitted 
for each user in alpha order by username (may be easier to order in spreadsheet)
    check for 10.2 lab
    check for 11.0 lab 
    …
    write email, 0|1, … 

If you need to collect late assignments from students:

nbgrader collect FilesFinally --student adomi14 --course cs128
nbgrader autograde "WhereIsThat?" --student adomi14 --course cs128 --force

Returning feedback to students

If you've set up your feedback script as described in the "Beginning of Term Notes" section, you can just use that, and you're done.

• Generate feedback: cd "course"; nbgrader feedback LabName
• Return feedback: sudo python return_feedback.py LabName

To enable the nbgrader assignment list extension for the students:

Each student should check that they have a .jupyter directory, and an nbconfig directory underneath that. After those directories are created, the student should run:

nbgrader extension install 
nbgrader extension activate

They may have to stop their server and restart for the changes to take effect. (Using the Control Panel in the upper-right hand corner of a Jupityr window.)




Temporary Status Quo - 10.06.17

Currently, it is possible to return feedback to students correctly, but it requires some fiddling. Dave thinks you may have to copy the gradebook file **back** into the cs128 directory; from there, you can run the feedback return script, and it appears to work. Check that it's actually working, however, before you tell students to look. If you do it wrong, feedback is in the feedback directory, but is either not returned to students or they get a version that hasn't been manually graded. You may have to edit the script to add --force to the nbgrader feedback command if you do it wrong the first time.

Multiple Simultaneous Graders

Using multiple graders works as follows: One user should log in as nbgrader and run nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like:

c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]

In the nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.

To tell if formgrade is running, do a ps auxw | grep formgrade | grep -v

Then check to see if there’s one running from:

/mnt/lovelace/software/anaconda/envs/py35/bin/python
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade

If there is, then somebody already has the formgrade session going, and you can just to to the web address.

Digging into nbgrader

Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).

The main nbgrader source code directory is here:

/mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader/

There are other files located here as well:

/mnt/lovelace/software/anaconda/pkgs/nbgrader-0.3.3-py35_0/lib/python3.5/site-packages/nbgrader/.

If you’re specifically looking for the assignment list extension, those are in nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.

If you get a message that looks like this:

Traceback (most recent call last):
     File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
       return_value = func(*args, **kwargs)
...
...
AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'

Conda may have gone out of sync. Try:

CONDA_SSL_VERIFY=false conda update pyopenssl

Authoring Worksheets

• If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
• Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.

Removing Students

In a python notebook, run the remove_student(student_id) command. It will delete the student along with the student's submissions. Don't forget to import Gradebook from nbgrader.api and to create a gradebook object:

gb = Gradebook("sqlite:///gradebook.db")

gb.remove_student("syweiss15") <ref>https://nbgrader.readthedocs.io/en/stable/api/gradebook.html</ref>




Beginning of Term Notes

At the beginning of each term:

• Using the addUsers.ipynb to add all of the students to your class.
• Make new file(s) akin to return_feedback-f17-314.py, changing the student IDs to the ones for your class(es)
• Make new file(s) akin to feedbackf17, changing the student IDs to the ones for your class(es) and changing the room numbers and references to the above file as appropriate.