- There is a user nbgrader (password is on Google Drive) on
tools.cs.earlham.eduthat must be used for all setup, assignment creation, grading, etc.
- nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
- Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
- Making assignments
- Keyboard input does not work with auto-grading, grading generally?
- Keyboard input may work with manual grading now
- Notes to students (in each assignment)
- Make sure your solution is named the same as the assignment.
- Don’t use spaces in file names (ever)
First Time Setup for a Class
- Create a directory for your class in
cdto that directory and create a config file. You can either make a copy of an existing one that’s close to what you need or use nbgrader’s tool to generate a default one and then figure-out all the bits that need to be changed/added for our local setup.
cd pcs290 cp ../test/nbgrader_config.py . ; $EDITOR nbgrader_config.py nbgrader --generate-config ; $EDITOR nbgrader_config.py
The config file now requires a list of all students and assignments for a class -- see the test config file for an example of how to do this.
- Create a source directory within the new course directory, e.g.
~nbgrader/pcs290/sourceAll of the assignment directories should be created in this directory (either from the command line or from the Jupyter/iPython interface).
- N.B. This is also the directory that you should run all of the commands to autograde, collect, etc. assignments for this class.
- Add the users to the database for the class. nbgrader uses a SQLite database to hold all of the students and assignments. This database is called
gradebook.dband it's in
~nbgrader/pcs290(or whatever your class directory is). Each class has it’s own database.
- Create a file that lists the first name, last name, and email address for the students in the class (cut the three columns from an XLS export from TheHeart).
- use generate with the add.student lines active
- There's a script in
addUsers.ipynbthat will connect to the database and add students. You must copy that script.
- Activate nbgrader for each student in the class
- Using the list of users from above and generate.. with the activate line active
- Create a file called
return_feedback.pyin the course directory (copy the one found in the test folder and modify it if necessary).
Creating an Assignment
~nbgrader/cs128/sourcecreate a directory for the assignment you are going to make. In that directory you can begin creating notebooks for those assignments.
- If you log into the notebook server at jupyter.cs.earlham.edu as nbgrader you will be able to create assignments. All assignments should be placed in the
source/directory of the
cs128/directory. There are assignments and then are notebooks that can go with those assignments.
- To create a notebook: select
New --> Python 3. In the Cell Toolbar: dropdown list, select Create Assignment. Make sure you rename the notebook to something appropriate for the assignment. You can create multiple problems in the notebook, give them an id, set the points, and then set the grading type. If you want to manually grade them you'll set it to Manually Grade. There's an autograde option, and also where you can set the solutions to the problems but for that information go to nbgrader.readthedocs.org. This file is just an introduction to nbgrader.
- Don’t use the
- Save the notebook and exit.
- Now we will assign the assignment to the students. Running nbgrader assign (not in the source directory, in
~/cs128) will look in the
source/directory, which is where you should've just saved the assignment you created. The assignment must be in the database before you can assign it, but there's a
--createoption with nbgrader that will create it as we assign. Run:
# this needs to be done any time you start a new session cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py) nbgrader assign --create ASSIGNMENT-NAME
- If you get an error like
No assignment called '%s' exists in the configyou need to add the assignment to the config file.
- If you get an error like
No assignment called 'Lab-02-Exp' exists in the configyou need to re-run the
export CONFIGPROXY_AUTH_TOKEN...line from a few lines up.
- That command will put the assignment in the
release/directory. To release that assignment to students, type:
nbgrader release ASSIGNMENT-NAME
- The assignment should've been release to the students. Once they log into the notebook server, if they have the extension installed, they will be able to click the Assignments tab, and fetch the assignments that are released. They can download and submit them as well.
- N.b. - If you want to make a change to an assignment after releasing it, you can overwrite what’s already released with
--force. However, a student who has already fetched the assignment will not have what they fetched altered, and there is not a way to remove the assignment through the Jupityr interface. Throught the terminal, they can do a
rm -r ASSIGNMENT-NAMEas normal, and after doing that, they can re-fetch the assignment through the Jupityr interface or with
nbgrader fetch ASSIGNMENT-NAME --course "COURSE-NAME"
Collecting and grading an assignment
- Before you can manually grade the assignments by running
nbgrader formgrade, you have to run
nbgrader autogradeeven if the assignments were set to be manually graded (don't ask me why, I don't know right now). Now if you run nbgrader formgrade again and go to the web browser, you should be able to see all of the assignments and students, and submissions. You click on an assignment to grade it.
- nbgrader formgrade is the command used to manual grade notebooks. This runs in the browser and connects with JupyterHub. You will be able to see all of the students and assignments release and whether or not they have submitted anything. Run nbgrader formgrade and then direct your browser to jupyter.cs.earlham.edu/hub/nbgrader/cs128. You will see assignments and students.
- collect submitted assignments:
cd “course” (ie cs128); nbgrader collect LabName
- This populates the ‘submitted’ folder
If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to go back and run
cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)
- autograde the submissions:
cd “course”; nbgrader autograde LabName# this can take a while
- If there are problems with the student’s submission (e.g. spaces in filenames, extra notebooks), they will manifest themselves here with a long error message. Use mv or rm depending on the nature of the problem.
- Just keep doing the up arrow, return, fix something, up arrow, return, fix something loop until autograde does not return any errors.
- If you can’t find a specific user file reference in the error message see b. Above... autograde populates the ‘autograded’ folder
- manually grade the submissions:
cd “course”; nbgrader formgradethen opon your $BROWSER to jupyter.cs.earlham.edu/hub/nbgrader/cs128
- In the upper right hand corner is a drop-down that will let you sort by student or by submission.
- If you receive “address in use” ps auxw | grep formgrade and kill -9 ...
If you get a bunch of errors and warnings about
JPY_API_TOKEN, you need to
export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)
If you get
errno 13 permission denied:
chown -R nbgrader:nbgrader /run/user/11040 (each time you login? seems to be owned by the underlying UID:GID)
- OR, alternatively, run
../formgrade.shfrom within a given course directory like cs128, cs256, etc.
sudo su - nbgrader
In general, if you ever want to do one of these operations for only one student (because they were late or something, you can use
If an assignment is missing from formgrade (this can happen if a student does an update), you can collect the assignments again, and then force the autograder to grade the new one with
nbgrader autograde [assignment-name] --student [studentid] --force
Arguments: ("submitted/aabdul15/Lab11-1/Charlie's solution.ipynb",) write the header row - “email, lab10-2, lab11-0, lab11-1, lab11-2” read cs128/submitted for each user in alpha order by username (may be easier to order in spreadsheet) check for 10.2 lab check for 11.0 lab … write email, 0|1, …
- If you need to collect late assignments from students:
nbgrader collect FilesFinally --student adomi14 --course cs128 nbgrader autograde "WhereIsThat?" --student adomi14 --course cs128 --force
Returning feedback to students
- Generate feedback:
cd "course"; nbgrader feedback LabName
- Return feedback:
sudo python return_feedback.py LabName
People with extra credit labs in 11-1 (Lab11-1-2, -3, etc. Were these fixed by repeated calls to autograde? One per extra cell per notebook?
To enable the nbgrader assignment list extension for the students:
Each student should check that they have a
.jupyter directory, and an nbconfig directory underneath that. After those directories are created, the student should run:
nbgrader extension install nbgrader extension activate
They may have to stop their server and restart for the changes to take effect. (Using the Control Panel in the upper-right hand corner of a Jupityr window.)
Grading notes go here.
Multiple Simultaneous Graders
Using multiple graders works as follows: One user should log in as nbgrader and run
nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like:
c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]
nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.
To tell if formgrade is running, do a
ps auxw | grep formgrade | grep -v
Then check to see if there’s one running from:
/mnt/lovelace/software/anaconda/envs/py35/bin/python /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade
If there is, then somebody already has the formgrade session going, and you can just to to the web address.
Digging into nbgrader
Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).
Some of the actual code is in
There is also code in
If you’re specifically looking for the assignment list extension, those are in
nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.
If you get a message that looks like this:
Traceback (most recent call last): File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler return_value = func(*args, **kwargs) ... ... AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'
Conda may have gone out of sync. Try:
CONDA_SSL_VERIFY=false conda update pyopenssl
- If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
- Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.