Nbgrader notes

Notes

• There is a user nbgrader (password is on Google Drive) on tools.cs.earlham.edu that must be used for all setup, assignment creation, grading, etc.
• nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
• Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
• Making assignments
• Keyboard input does not work with auto-grading, grading generally?
• Keyboard input may work with manual grading now
• Notes to students (in each assignment)
• Make sure your solution is named the same as the assignment.
• Don’t use spaces in file names (ever)

First Time Setup for a Class

• Create a directory for your class in ~nbgrader
• mkdir pcs290
• cd to that directory and create a config file. You can either make a copy of an existing one that’s close to what you need or use nbgrader’s tool to generate a default one and then figure-out all the bits that need to be changed/added for our local setup.

cd pcs290

cp ../test/nbgrader_config.py . ; $EDITOR nbgrader_config.py

nbgrader --generate-config ; $EDITOR nbgrader_config.py

The config file now requires a list of all students and assignments for a class -- see the test config file for an example of how to do this.

• Create a source directory within the new course directory, e.g. ~nbgrader/pcs290/source All of the assignment directories should be created in this directory (either from the command line or from the Jupyter/iPython interface).
• N.B. This is also the directory that you should run all of the commands to autograde, collect, etc. assignments for this class.
• Add the users to the database for the class. nbgrader uses a SQLite database to hold all of the students and assignments. This database is called gradebook.db and it's in ~nbgrader/pcs290 (or whatever your class directory is). Each class has it’s own database.
• Create a file that lists the first name, last name, and email address for the students in the class (cut the three columns from an XLS export from TheHeart).
• use generate with the add.student lines active
• There's a script in ~nbgrader/cs128 called addUsers.ipynb that will connect to the database and add students. You must copy that script.
• Activate nbgrader for each student in the class
• Using the list of users from above and generate.. with the activate line active
• Create a file called return_feedback.py in the course directory (copy the one found in the test folder and modify it if necessary).
• A student can run /home/nbgrader/set-me-up if the assignments tab isn't showing up for them. They'll need to restart their server after that.

Creating an Assignment

• In ~nbgrader/cs128/source create a directory for the assignment you are going to make. In that directory you can begin creating notebooks for those assignments.
• If you log into the notebook server at jupyter.cs.earlham.edu as nbgrader you will be able to create assignments. All assignments should be placed in the source/ directory of the cs128/ directory. There are assignments and then are notebooks that can go with those assignments.
• To create a notebook: select New --> Python 3. In the Cell Toolbar: dropdown list, select Create Assignment. Make sure you rename the notebook to something appropriate for the assignment. You can create multiple problems in the notebook, give them an id, set the points, and then set the grading type. If you want to manually grade them you'll set it to Manually Grade. There's an autograde option, and also where you can set the solutions to the problems but for that information go to nbgrader.readthedocs.org. This file is just an introduction to nbgrader.
• Don’t use the RawNbConvert cell type.
• Save the notebook and exit.
• Now we will assign the assignment to the students. Running nbgrader assign (not in the source directory, in ~/cs128) will look in the source/ directory, which is where you should've just saved the assignment you created. The assignment must be in the database before you can assign it, but there's a --create option with nbgrader that will create it as we assign. Run:

# this needs to be done any time you start a new session
cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

nbgrader assign --create ASSIGNMENT-NAME

• If you get an error like No assignment called '%s' exists in the config you need to add the assignment to the config file.
• If you get an error like No assignment called 'Lab-02-Exp' exists in the config you need to re-run the export CONFIGPROXY_AUTH_TOKEN... line from a few lines up.
• That command will put the assignment in the release/ directory. To release that assignment to students, type: nbgrader release ASSIGNMENT-NAME
• The assignment should've been release to the students. Once they log into the notebook server, if they have the extension installed, they will be able to click the Assignments tab, and fetch the assignments that are released. They can download and submit them as well.
• N.b. - If you want to make a change to an assignment after releasing it, you can overwrite what’s already released with --force. However, a student who has already fetched the assignment will not have what they fetched altered, and there is not a way to remove the assignment through the Jupityr interface. Throught the terminal, they can do a rm -r ASSIGNMENT-NAME as normal, and after doing that, they can re-fetch the assignment through the Jupityr interface or with nbgrader fetch ASSIGNMENT-NAME --course "COURSE-NAME"

Collecting and grading an assignment

1. Before you can manually grade the assignments by running nbgrader formgrade, you have to run nbgrader autograde even if the assignments were set to be manually graded (don't ask me why, I don't know right now). Now if you run nbgrader formgrade again and go to the web browser, you should be able to see all of the assignments and students, and submissions. You click on an assignment to grade it.
2. nbgrader formgrade is the command used to manual grade notebooks. This runs in the browser and connects with JupyterHub. You will be able to see all of the students and assignments release and whether or not they have submitted anything. Run nbgrader formgrade and then direct your browser to jupyter.cs.earlham.edu/hub/nbgrader/cs128. You will see assignments and students.
3. collect submitted assignments: cd “course” (ie cs128); nbgrader collect LabName

• This populates the ‘submitted’ folder

• If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to go back and run

  cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

4. autograde the submissions: cd “course”; nbgrader autograde LabName # this can take a while

• If there are problems with the student’s submission (e.g. spaces in filenames, extra notebooks), they will manifest themselves here with a long error message. Use mv or rm depending on the nature of the problem.
• Just keep doing the up arrow, return, fix something, up arrow, return, fix something loop until autograde does not return any errors.
• If you can’t find a specific user file reference in the error message see b. Above... autograde populates the ‘autograded’ folder

5. manually grade the submissions: cd “course”; nbgrader formgrade then opon your $BROWSER to jupyter.cs.earlham.edu/hub/nbgrader/cs128

• In the upper right hand corner is a drop-down that will let you sort by student or by submission.
• If you receive “address in use” ps auxw | grep formgrade and kill -9 ...

• If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to

  export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)

• If you get errno 13 permission denied:

  chown -R nbgrader:nbgrader /run/user/11040 (each time you login? seems to be owned by the underlying UID:GID)

6. OR, alternatively, run ../formgrade.sh from within a given course directory like cs128, cs256, etc.
7. ssh home.cs.earlham.edu
8. ssh tools.cs.earlham.edu
9. sudo su - nbgrader
10. cd <course>

11. ../formgrade.sh

12. In general, if you ever want to do one of these operations for only one student (because they were late or something, you can use --student [studentid]

13. If an assignment is missing from formgrade (this can happen if a student does an update), you can collect the assignments again, and then force the autograder to grade the new one with nbgrader autograde [assignment-name] --student [studentid] --force

Arguments: ("submitted/aabdul15/Lab11-1/Charlie's solution.ipynb",)


write the header row - “email, lab10-2, lab11-0, lab11-1, lab11-2” 
read cs128/submitted 
for each user in alpha order by username (may be easier to order in spreadsheet)
    check for 10.2 lab
    check for 11.0 lab 
    …
    write email, 0|1, … 

If you need to collect late assignments from students:

nbgrader collect FilesFinally --student adomi14 --course cs128
nbgrader autograde "WhereIsThat?" --student adomi14 --course cs128 --force

Returning feedback to students

• Generate feedback: cd "course"; nbgrader feedback LabName
• Return feedback: sudo python return_feedback.py LabName

People with extra credit labs in 11-1 (Lab11-1-2, -3, etc. Were these fixed by repeated calls to autograde? One per extra cell per notebook?

To enable the nbgrader assignment list extension for the students:

Each student should check that they have a .jupyter directory, and an nbconfig directory underneath that. After those directories are created, the student should run:

nbgrader extension install 
nbgrader extension activate

They may have to stop their server and restart for the changes to take effect. (Using the Control Panel in the upper-right hand corner of a Jupityr window.)

Grading notes go here.

Multiple Simultaneous Graders

Using multiple graders works as follows: One user should log in as nbgrader and run nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like:

c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]

In the nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.

To tell if formgrade is running, do a ps auxw | grep formgrade | grep -v

Then check to see if there’s one running from:

/mnt/lovelace/software/anaconda/envs/py35/bin/python
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade

If there is, then somebody already has the formgrade session going, and you can just to to the web address.

Digging into nbgrader

Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).

The main nbgrader source code directory is here:

/mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader/

There are other files located here as well:

/mnt/lovelace/software/anaconda/pkgs/nbgrader-0.3.3-py35_0/lib/python3.5/site-packages/nbgrader/.

If you’re specifically looking for the assignment list extension, those are in nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.

If you get a message that looks like this:

Traceback (most recent call last):
     File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
       return_value = func(*args, **kwargs)
...
...
AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'

Conda may have gone out of sync. Try:

CONDA_SSL_VERIFY=false conda update pyopenssl

Authoring Worksheets

• If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
• Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.