- 1 Notes
- 2 Upgrading nbgrader
- 3 Creating an Assignment
- 4 Assigning Assignments
- 5 Collecting and grading an assignment
- 6 Returning feedback to students
- 7 Multiple Simultaneous Graders
- 8 Digging into nbgrader
- 9 Authoring Worksheets
- 10 Removing Students
- 11 Beginning of Term Notes: Setting up a New Class
- 12 Managing Assignments
- There is a user nbgrader (password is on Google Drive) on
tools.cs.earlham.eduthat must be used for all setup, assignment creation, grading, etc.
- nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
- This documentation is extremely good, and should generally be referred to first.
- Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
- Making assignments
- Keyboard input does not work with auto-grading, grading generally?
- Keyboard input may work with manual grading now
- Notes to students (in each assignment)
- Make sure your solution is named the same as the assignment.
- Don’t use spaces in file names (ever). Avoid using special characters aside from -
- Find our tools for working with nbgrader here.
- At the finish of every semester, make a copy of the cs128 directory and name it as follows:
sem_year_cs128(with the corresponding semester as fall or spring) and move it to the
Please also read:
- Grab latest release of
nbgraderfrom github along with further install information.
- Copy source code zip to
scp source-code.zip firstname.lastname@example.org:~/
- login as
jupyterand then become
sudo su -
- If going through a shell,
- Backup exsiting
- Uninstall current version of nbgrader.
conda remove nbgradershould work, if you get
NoPackagesFoundErrortry uninstalling with
pip uninstall nbgrader
- Install new
condaand specify the
anaconda python 3.5environment
conda install -c conda-forge nbgrader -n py35
nbgraderuser server from Jupyter control panel.
- May take a couple minutes for
jupytercome back online
- May take a couple minutes for
- Ability to switch between courses in assignment list extension #563
- Needs to be added manually until it gets merged into
- Needs to be added manually until it gets merged into
Creating an Assignment
~nbgrader/cs128/sourcecreate a directory for the assignment you are going to make. In that directory you can begin creating notebooks for those assignments.
- If you log into the notebook server at jupyter.cs.earlham.edu as nbgrader you will be able to create assignments. All assignments should be placed in the
source/directory of the
cs128/directory. There are assignments and then are notebooks that can go with those assignments.
- To create a notebook: select
New --> Python 3. In the Cell Toolbar: dropdown list, select Create Assignment. Make sure you rename the notebook to something appropriate for the assignment. You can create multiple problems in the notebook, give them an id, set the points, and then set the grading type. If you want to manually grade them you'll set it to Manually Grade. There's an autograde option, and also where you can set the solutions to the problems but for that information go to nbgrader.readthedocs.org. This file is just an introduction to nbgrader.
- Don’t use the
- Save the notebook and exit.
- Note that if you are assigning an assignment with manually graded cells to check beforehand that everything is running smoothly with the point system on the formgrade. In the past there have been issues with this so you may want to double check that it is working correctly.
- Now we will assign the assignment to the students. Running nbgrader assign (not in the source directory, in
~/cs128) will look in the
source/directory, which is where you should've just saved the assignment you created. The assignment must be in the database before you can assign it (this means in must be in the config file), but there's a
--createoption with nbgrader that will create it as we assign. Run:
# this needs to be done any time you start a new session cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py) nbgrader assign --create ASSIGNMENT-NAME
- If you get an error like
No assignment called '%s' exists in the configyou need to add the assignment to the config file.
- If you get an error like
No assignment called 'Lab-02-Exp' exists in the configyou need to re-run the
export CONFIGPROXY_AUTH_TOKEN...line from a few lines up.
- That command will put the assignment in the
release/directory. To release that assignment to students, type:
nbgrader release ASSIGNMENT-NAME
- The assignment should've been release to the students. Once they log into the notebook server, if they have the extension installed, they will be able to click the Assignments tab, and fetch the assignments that are released. They can download and submit them as well.
- N.b. - If you want to make a change to an assignment after releasing it, you can overwrite what’s already released with
--force. However, a student who has already fetched the assignment will not have what they fetched altered, and there is not a way to remove the assignment through the Jupityr interface. Throught the terminal, they can do a
rm -r ASSIGNMENT-NAMEas normal, and after doing that, they can re-fetch the assignment through the Jupityr interface or with
nbgrader fetch ASSIGNMENT-NAME --course "COURSE-NAME"
Temporary Status Quo 10.16.17
If you get an error that says TypeError: 'Assignment' object is not subscriptable: - Assign the assignment in the terminal, using the --no-db flag and the --create flag, like this:
nbgrader assign Lab7-Files-F17 --create --no-db
Collecting and grading an assignment
This is almost all entirely out of date since the Fall 2017 update
When doing these steps, make sure you are in the course's directory and not the source directory. For cs128, you want to be in the
- Run the export
export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TO
KEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterh
- Collect the assingments
nbgrader collect ASSIGNMENT_NAME
nbgrader autograde ASSIGNMENT_NAME
nbgrader formgradeOpend browser to
https://jupyter.cs.earlham.edu/hub/nbgrader/cs128In the upper right hand corner, you can change the view to view students or assignments.
- Before you can manually grade the assignments by running
nbgrader formgrade, you have to run
nbgrader autogradeeven if the assignments were set to be manually graded (don't ask me why, I don't know right now). Now if you run nbgrader formgrade again and go to the web browser, you should be able to see all of the assignments and students, and submissions. You click on an assignment to grade it.
- nbgrader formgrade is the command used to manual grade notebooks. This runs in the browser and connects with JupyterHub. You will be able to see all of the students and assignments release and whether or not they have submitted anything. Run nbgrader formgrade and then direct your browser to jupyter.cs.earlham.edu/hub/nbgrader/cs128. You will see assignments and students.
- collect submitted assignments:
cd “course” (ie cs128); nbgrader collect LabName
- This populates the ‘submitted’ folder
If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to go back and run
cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)
- autograde the submissions:
cd “course”; nbgrader autograde LabName# this can take a while
- If there are problems with the student’s submission (e.g. spaces in filenames, extra notebooks), they will manifest themselves here with a long error message. Use mv or rm depending on the nature of the problem.
- Just keep doing the up arrow, return, fix something, up arrow, return, fix something loop until autograde does not return any errors.
- If you can’t find a specific user file reference in the error message see b. Above... autograde populates the ‘autograded’ folder
- manually grade the submissions:
cd “course”; nbgrader formgradethen opon your $BROWSER to jupyter.cs.earlham.edu/hub/nbgrader/cs128
- In the upper right hand corner is a drop-down that will let you sort by student or by submission.
- If you receive “address in use” ps auxw | grep formgrade and kill -9 ...
If you get a bunch of errors and warnings about
JPY_API_TOKEN, you need to
export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)
If you get
errno 13 permission denied:
chown -R nbgrader:nbgrader /run/user/11040 (each time you login? seems to be owned by the underlying UID:GID)
- OR, alternatively, run
../formgrade.shfrom within a given course directory like cs128, cs256, etc.
sudo su - nbgrader
In general, if you ever want to do one of these operations for only one student (because they were late or something, you can use
If an assignment is missing from formgrade (this can happen if a student does an update), you can collect the assignments again, and then force the autograder to grade the new one with
nbgrader autograde [assignment-name] --student [studentid] --force
Arguments: ("submitted/aabdul15/Lab11-1/Charlie's solution.ipynb",) write the header row - “email, lab10-2, lab11-0, lab11-1, lab11-2” read cs128/submitted for each user in alpha order by username (may be easier to order in spreadsheet) check for 10.2 lab check for 11.0 lab … write email, 0|1, …
- If you need to collect late assignments from students:
nbgrader collect FilesFinally --student adomi14 --course cs128 nbgrader autograde "WhereIsThat?" --student adomi14 --course cs128 --force
Returning feedback to students
If you've set up your feedback script as described in the "Beginning of Term Notes" section, you can just use that, and you're done.
- Generate feedback:
cd "course"; nbgrader feedback LabName
- Return feedback:
To enable the nbgrader assignment list extension for the students:
Each student should check that they have a
.jupyter directory, and an nbconfig directory underneath that. After those directories are created, the student should run:
nbgrader extension install nbgrader extension activate
They may have to stop their server and restart for the changes to take effect. (Using the Control Panel in the upper-right hand corner of a Jupyter window.)
Multiple Simultaneous Graders
Using multiple graders works as follows: One user should log in as nbgrader and run
nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like:
c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]
nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.
To tell if formgrade is running, do a
ps auxw | grep formgrade | grep -v
Then check to see if there’s one running from:
/mnt/lovelace/software/anaconda/envs/py35/bin/python /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade
If there is, then somebody already has the formgrade session going, and you can just to to the web address.
Digging into nbgrader
Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).
The main nbgrader source code directory is here:
There are other files located here as well:
If you’re specifically looking for the assignment list extension, those are in
nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.
If you get a message that looks like this:
Traceback (most recent call last): File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler return_value = func(*args, **kwargs) ... ... AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'
Conda may have gone out of sync. Try:
CONDA_SSL_VERIFY=false conda update pyopenssl
- If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
- Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.
In a python notebook, run the
remove_student(student_id) command. It will delete the student along with the student's submissions.
Don't forget to
import Gradebook from nbgrader.api and to create a gradebook object:
gb = Gradebook("sqlite:///gradebook.db")
Beginning of Term Notes: Setting up a New Class
- Log in as nbgrader. From within the terminal, use
nbgrade quickstart classnameto create a new class, where classname is the name of the class. Folders for cs128 classes are called cs128-f20 or cs128-s21, where f and s are "fall" and "spring" and the two digits are the year. This command will create a new folder for the class, plus a source folders and config file within it.
- Copy the contents of the source directory from last term’s course into the new directory's source directory (or whatever material from previous courses that you plan to use.) The source directory is where assignments are stored before they are released to students.
- The config file at
~nbgrader/.jupyter/nbgrader_config.pyshould be edited to update the two references to the old class so that they refer to the new class.
- You may need to restart nbgrader’s server at this point. From the jupyter interface, select “Control Panel,” then “Stop My Server.” After a moment, you will be able to select “Start Server.”
- Navigate to
/srv/nbgrader/exchange. Move the previous term’s directory into the old/ directory. This will prevent students from ever seeing the old labs (and possibly getting them by mistake, rather than getting the new labs.) -- This step may now be unnecessary, but it shouldn’t hurt.
- Update the student name list in the file student-names-file. This file lives in nbgrader’s home directory.
student-names-file-samplecontains an example of what this should look like. If you have a .csv downloaded from The Heart, tools in
addUsers.ipynbcan be used to convert produce a
- You shouldn’t need to do this now, but you can use other tools in
addUsers.ipynbto add and remove individual students from the database. For this to work, note that
addUsers.ipynbmust be inside the course directory for the class; if it is run from nbgrader’s home directory, this will not work. This is useful if a student joins late and you just need to add one student. Dave believes that this also works to change the student’s name.
- Currently, the
student-names-filefile is used to build up the student db. You can import names from the list using
nbgrader db student import student-names-fileThe content in the
student-names-fileshould follow the style required by nbgrader (the first line needs to be as follows, and the last line needs to be empty). If you generated student-names-file using the instructions in the previous steps, it should already look like this.
- Line 1:
- Line 2:
- Line 1:
- (Optional) You can use
nbgrader db student listto check if the students has been added successfully
- (Optional) You can use
nbgrader db student list --course-dir=”~/<course-directory>”to check the student list for a specified course.
- This step and the remaining steps may no longer be necessary. Copy the two feedback files, which have names like feedback-summer19.sh, and return_feedback-summer19.py, into the course directory for the new course.
- Update the names of those two files to refer to the correct term.
- Edit the feedback shell script with the name of the feedback .py file.
- Edit the feedback .py file by changing
students = ["barbeda",...]to a list of the students in the class.
Releasing an Assignment
- Log in to the Jupyter web interface at tools.cs.earlham.edu as nbgrader.
- In Jupyter, navigate to the “Formgrader” tab.
- Choose an assignment you would like to release. Any assignment in the source directory should be visible here. If it's not, something has gone wrong during the setup process.
- “Generate” => “Release”
- Both of these should pop up log output boxes. Neither log output should contain errors, just information.
- Once an assignment has been released, students can access it.
- When they log into Jupyter, they should have access to an Assignments tab. They can select an assignment to fetch from that tab. This places the assignment in a new directory in their home directory. They can complete their work in the workbook in that directory.
- When they are done, they can submit their work in the Assignments tab. They can submit work multiple times, but generally students just submit the work when it’s done.
Collecting And Grading Student Work
- Once students have submitted their work, you can Collect submissions using the “Collect” button in “Formgrader” tab. You can collect submissions as many times as necessary.
- There are several ways to Autograde the submission.
- Method A) In the terminal, run
nbgrader autograde <Lab_Name>. To autograde one student’s submission, use nbgrader
autograde <Lab-name> --student <student-id>
- Method B) From within the formgrader, after the lab has been collected, click the number of submissions for that lab, then click Autograde for each student you want it to autograde. This is useful if a student turns in work late.
- Method A) In the terminal, run
- Once Autograding is complete, manually grade any parts of the assignments that can’t be autograded. Feedback is also added to student work at this time, and this is where partial credit can be awarded. In the formgrader, choose “Manual Grading” on the left. Then, select the labs and assignments you wish to manually grade. You can also adjust scores produced by the autograding system, in order to award partial credit. (Or even full credit, if the student’s work is correct but was mislabeled by the autograding system.)
- After coursework has been graded, you can generate and release feedback using the appropriate buttons in the Manage Assignments screen within the formgrader. Students can then fetch the feedback in a process similar to how they fetch assignments.
Feedback can also be generated and distributed using the files feedback-summer19.sh and return_feedback-summer19.py (The names of these will change each term.) These must be located in the directory for the course.
To generate and return feedback for all students in the current course of lab <Lab-Name> $ ./feedback-summer19 <Lab-Name> feedback-summer19: this file read the student list from the file student-names-file, and generate feedback file for each student, then it will run the return_feedback-summer19.py return_feedback-summer19.py: this file will copy the feedback folder to the students’ home directory.
To return feedback for a specified student: $ nbgrader feedback --student <student-id> --assignment <LAB_NAME> --force $ sudo python return_feedback-f19.py <LAB_NAME> --student <student-id>
Make sure the lab-names.txt file in the root nbgrader directory is up to date (i.e. the names of the labs are correct). They need to be correct to use MOSS.