Difference between revisions of "Nbgrader notes"

From Earlham CS Department
Jump to navigation Jump to search
(→‎First Time Setup for a Class)
(Igor added a bullet point to the beginning of the list of things to do in the beginning of the term: send the list of student names and email addresses to the admins)
 
(42 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
* There is a user nbgrader (password is on Google Drive) on <code>tools.cs.earlham.edu</code> that must be used for all setup, assignment creation, grading, etc.
 
* There is a user nbgrader (password is on Google Drive) on <code>tools.cs.earlham.edu</code> that must be used for all setup, assignment creation, grading, etc.
 
* nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
 
* nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
 +
** This documentation is extremely good, and should generally be referred to first.
 
* Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
 
* Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
 
* Making assignments
 
* Making assignments
Line 12: Line 13:
 
* Find our tools for working with nbgrader [[nbgrader_notes/tools|here]].
 
* Find our tools for working with nbgrader [[nbgrader_notes/tools|here]].
 
*At the finish of every semester, make a copy of the cs128 directory and name it as follows: <code>sem_year_cs128</code> (with the corresponding semester as fall or spring) and move it to the <code>cs128Archives</code> directory.
 
*At the finish of every semester, make a copy of the cs128 directory and name it as follows: <code>sem_year_cs128</code> (with the corresponding semester as fall or spring) and move it to the <code>cs128Archives</code> directory.
 +
 +
Please also read:
 +
* [[Nbgrader notes/Grading]]
 +
* [[Nbgrader notes/tools]]
 +
* [[Nbgrader notes/Weekly setup]]
 +
 +
== Beginning of Term Notes: Setting up a New Class ==
 +
 +
# At least a week before the term starts, send the admins (admin [at] cs [dot] earlham [dot] edu) the list of student names and email addresses so the admins can set up accounts for them.
 +
# Log in as nbgrader. From within the terminal, use <code>nbgrader quickstart classname</code> to create a new class, where classname is the name of the class (something like cs128-f21). Folders for cs128 classes are called cs128-f20 or cs128-s21, where f and s are "fall" and "spring" and the two digits are the year. This command will create a new folder for the class, plus a source folders and config file within it.
 +
# Copy the contents of the source directory from last term’s course into the new directory's source directory (or whatever material from previous courses that you plan to use.) The source directory is where assignments are stored before they are released to students.
 +
# The config file at <code>~nbgrader/.jupyter/nbgrader_config.py</code> should be edited to update the two references to the old class so that they refer to the new class.
 +
# You may need to restart nbgrader’s server at this point. From the jupyter interface, select “Control Panel,” then “Stop My Server.” After a moment, you will be able to select “Start Server.”
 +
# Navigate to <code>/srv/nbgrader/exchange</code>. Move the previous term’s directory into the old/ directory. This will prevent students from ever seeing the old labs (and possibly getting them by mistake, rather than getting the new labs.) -- This step may now be unnecessary, but it shouldn’t hurt.
 +
# Update the student name list in the file student-names-file. This file lives in nbgrader’s home directory. <code>student-names-file-sample</code> contains an example of what this should look like. If you have a .csv downloaded from The Heart, tools in <code>addUsers.ipynb</code> can be used to convert produce a <code>student-names-file</code> using:
 +
## <code>make_student_names_file(“csv_file_from_the_heart.csv”, “student-names-file”)</code>
 +
# You shouldn’t need to do this now, but you can use other tools in <code>addUsers.ipynb</code> to add and remove individual students from the database. For this to work, note that <code>addUsers.ipynb</code> must be inside the course directory for the class; if it is run from nbgrader’s home directory, this will not work. This is useful if a student joins late and you just need to add one student. Dave believes that this also works to change the student’s name.
 +
# Currently, the <code>student-names-file</code> file is used to build up the student db. You can import names from the list using <code>nbgrader db student import student-names-file</code> The content in the <code>student-names-file</code> should follow the style required by nbgrader (the first line needs to be as follows, and the last line needs to be empty). If you generated student-names-file using the instructions in the previous steps, it should already look like this.
 +
## Line 1: <code>id,last_name,first_name, email</code>
 +
## Line 2: <code>smaizaz18,Aizaz,Shah, smaizaz18@earlham.edu</code>
 +
# (Optional) You can use <code>nbgrader db student list</code> to check if the students has been added successfully
 +
# (Optional) You can use <code>nbgrader db student list --course-dir=”~/<course-directory>”</code> to check the student list for a specified course.
 +
# ''This step and the remaining steps may no longer be necessary.'' Copy the two feedback files, which have names like feedback-summer19.sh, and return_feedback-summer19.py, into the course directory for the new course.
 +
# Update the names of those two files to refer to the correct term.
 +
# Edit the feedback shell script with the name of the feedback .py file.
 +
# Edit the feedback .py file by changing <code>students = ["barbeda",...]</code> to a list of the students in the class.
 +
 +
== Managing Assignments ==
 +
=== Preparing an Assignment ===
 +
# Assignments can be found in the source directory within the course directory.
 +
# Assignments should be updated before release - once an assignment is released, further changes to the source copy will not be reflected in the version that students get.
 +
 +
=== Releasing an Assignment ===
 +
# Log in to the Jupyter web interface at tools.cs.earlham.edu as nbgrader.
 +
# In Jupyter, navigate to the “Formgrader” tab.
 +
# Choose an assignment you would like to release. Any assignment in the source directory should be visible here. If it's not, something has gone wrong during the setup process.
 +
# “Generate” => “Release”
 +
## Both of these should pop up log output boxes. Neither log output should contain errors, just information.
 +
 +
===Student Work===
 +
# Once an assignment has been released, students can access it.
 +
# When they log into Jupyter, they should have access to an Assignments tab. They can select an assignment to fetch from that tab. This places the assignment in a new directory in their home directory. They can complete their work in the workbook in that directory.
 +
# When they are done, they can submit their work in the Assignments tab. They can submit work multiple times, but generally students just submit the work when it’s done.
 +
===Collecting And Grading Student Work===
 +
# Once students have submitted their work, you can Collect submissions using the “Collect” button in “Formgrader” tab. You can collect submissions as many times as necessary.
 +
# There are several ways to Autograde the submission.
 +
## Method A) In the terminal, run <code>nbgrader autograde <Lab_Name></code>. To autograde one student’s submission, use nbgrader <code>autograde <Lab-name> --student <student-id></code>
 +
## Method B) From within the formgrader, after the lab has been collected, click the number of submissions for that lab, then click Autograde for each student you want it to autograde. This is useful if a student turns in work late.
 +
# Once Autograding is complete, manually grade any parts of the assignments that can’t be autograded. Feedback is also added to student work at this time, and this is where partial credit can be awarded. In the formgrader, choose “Manual Grading” on the left. Then, select the labs and assignments you wish to manually grade. You can also adjust scores produced by the autograding system, in order to award partial credit. (Or even full credit, if the student’s work is correct but was mislabeled by the autograding system.)
 +
 +
===Releasing Feedback===
 +
# After coursework has been graded, you can generate and release feedback using the appropriate buttons in the Manage Assignments screen within the formgrader. Students can then fetch the feedback in a process similar to how they fetch assignments.
 +
 +
'''Alternate method'''
 +
 +
Feedback can also be generated and distributed using the files feedback-summer19.sh and return_feedback-summer19.py (The names of these will change each term.) These must be located in the directory for the course.
 +
 +
To generate and return feedback for all students in the current course of lab <Lab-Name>
 +
$ ./feedback-summer19 <Lab-Name>
 +
feedback-summer19: this file read the student list from the file student-names-file, and generate feedback file for each student, then it will run the return_feedback-summer19.py
 +
return_feedback-summer19.py: this file will copy the feedback folder to the students’ home directory.
 +
 +
To return feedback for a specified student:
 +
$ nbgrader feedback --student <student-id> --assignment <LAB_NAME> --force
 +
$ sudo python return_feedback-f19.py <LAB_NAME>  --student <student-id>
 +
 +
Make sure the lab-names.txt file in the root nbgrader directory is up to date (i.e. the names of the labs are correct). They need to be correct to use MOSS.
  
 
== Upgrading nbgrader ==
 
== Upgrading nbgrader ==
Line 32: Line 100:
 
* [https://github.com/jupyter/nbgrader/pull/563/files Ability to switch between courses in assignment list extension #563]
 
* [https://github.com/jupyter/nbgrader/pull/563/files Ability to switch between courses in assignment list extension #563]
 
** Needs to be added manually until it gets merged into <code>nbgrader master</code>.
 
** Needs to be added manually until it gets merged into <code>nbgrader master</code>.
−
 
−
== First Time Setup for a Class ==
 
−
 
−
* Create a directory for your class in <code>~nbgrader</code>
 
−
* <code>mkdir pcs290</code>
 
−
* <code>cd</code> to that directory and create a config file. A template file appropriately called <code>nbgrader_config_template.py</code> exists in nbgraders home directory, and you can adapt that one for the course. Alternatively you can use nbgrader’s tool to generate a default one and then figure-out all the bits that need to be changed/added for our local setup.
 
−
 
−
<pre>cd pcs290
 
−
 
−
cp ../test/nbgrader_config.py . ; $EDITOR nbgrader_config.py
 
−
 
−
nbgrader --generate-config ; $EDITOR nbgrader_config.py</pre>
 
−
The config file now requires a list of all students and assignments for a class -- see the test config file for an example of how to do this.
 
−
 
−
* Create a source directory within the new course directory, e.g. <code>~nbgrader/pcs290/source</code> All of the assignment directories should be created in this directory (either from the command line or from the Jupyter/iPython interface).
 
−
* N.B. This is also the directory that you should run all of the commands to autograde, collect, etc. assignments for this class.
 
−
* Add the users to the database for the class. nbgrader uses a SQLite database to hold all of the students and assignments. This database is called <code>gradebook.db</code> and it's in <code>~nbgrader/pcs290</code> (or whatever your class directory is). Each class has it’s own database.
 
−
* Create a file that lists the first name, last name, and email address for the students in the class (cut the three columns from an XLS export from TheHeart).
 
−
* use generate with the add.student lines active
 
−
* There's a script in <code>~nbgrader/cs128</code> called <code>addUsers.ipynb</code> that will connect to the database and add students. You must copy that script.
 
−
* Activate nbgrader for each student in the class
 
−
* Using the list of users from above and generate.. with the activate line active
 
−
* Create a file called <code>return_feedback.py</code> in the course directory (copy the one found in the test folder and modify it if necessary).
 
−
* A student can run /home/nbgrader/set-me-up if the assignments tab isn't showing up for them. They'll need to restart their server after that.
 
−
 
−
== Creating an Assignment ==
 
−
 
−
* In <code>~nbgrader/cs128/source</code> create a directory for the assignment you are going to make. In that directory you can begin creating notebooks for those assignments.
 
−
* If you log into the notebook server at jupyter.cs.earlham.edu as nbgrader you will be able to create assignments. All assignments should be placed in the <code>source/</code> directory of the <code>cs128/</code> directory. There are assignments and then are notebooks that can go with those assignments.
 
−
* To create a notebook: select <code>New --&gt; Python 3</code>. In the Cell Toolbar: dropdown list, select Create Assignment. Make sure you rename the notebook to something appropriate for the assignment. You can create multiple problems in the notebook, give them an id, set the points, and then set the grading type. If you want to manually grade them you'll set it to Manually Grade. There's an autograde option, and also where you can set the solutions to the problems but for that information go to nbgrader.readthedocs.org. This file is just an introduction to nbgrader.
 
−
* Don’t use the <code>RawNbConvert</code> cell type.
 
−
* Save the notebook and exit.
 
−
== Assigning Assignments==
 
−
 
−
* Note that if you are assigning an assignment with manually graded cells to check beforehand that everything is running smoothly with the point system on the formgrade. In the past there have been issues with this so you may want to double check that it is working correctly.
 
−
* Now we will assign the assignment to the students. Running nbgrader assign (not in the source directory, in <code>~/cs128</code>) will look in the <code>source/</code> directory, which is where you should've just saved the assignment you created. The assignment must be in the database before you can assign it (this means in must be in the config file),  but there's a <code>--create</code> option with nbgrader that will create it as we assign. Run:
 
−
 
−
<pre># this needs to be done any time you start a new session
 
−
cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)
 
−
 
−
nbgrader assign --create ASSIGNMENT-NAME</pre>
 
−
* If you get an error like <code>No assignment called '%s' exists in the  config</code> you need to add the assignment to the config file.
 
−
* If you get an error like <code>No assignment called 'Lab-02-Exp' exists in  the config</code> you need to re-run the <code>export CONFIGPROXY_AUTH_TOKEN...</code> line from a few lines up.
 
−
* That command will put the assignment in the <code>release/</code> directory. To release that assignment to students, type: <code>nbgrader release  ASSIGNMENT-NAME</code>
 
−
* The assignment should've been release to the students. Once they log into the notebook server, if they have the extension installed, they will be able to click the Assignments tab, and fetch the assignments that are released. They can download and submit them as well.
 
−
* N.b. - If you want to make a change to an assignment after releasing it, you can overwrite what’s already released with <code>--force</code>. However, a student who has already fetched the assignment will not have what they fetched altered, and there is not a way to remove the assignment through the Jupityr interface. Throught the terminal, they can do a <code>rm -r ASSIGNMENT-NAME</code> as normal, and after doing that, they can re-fetch the assignment through the Jupityr interface or with <code>nbgrader fetch ASSIGNMENT-NAME --course &quot;COURSE-NAME&quot;</code>
 
−
 
−
=== Temporary Status Quo 10.16.17 ===
 
−
 
−
If you get an error that says TypeError: 'Assignment' object is not subscriptable:
 
−
- Assign the assignment in the terminal, using the --no-db flag and the --create flag, like this:
 
−
 
 
−
  nbgrader assign Lab7-Files-F17 --create --no-db
 
−
 
−
== Collecting and grading an assignment ==
 
−
 
−
'''This is almost all entirely out of date since the Fall 2017 update'''
 
−
 
−
Basic Steps:
 
−
When doing these steps, make sure you are in the course's directory and not the source directory. For cs128, you want to be in the <code>cs128</code> directory.
 
−
# Run the export
 
−
<code>export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TO
 
−
KEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterh
 
−
ub/jupyterhub_config.py) </code>
 
−
 
−
<ol start="2" style="list-style-type: decimal;">  <li>Collect the assingments
 
−
<code> nbgrader collect ASSIGNMENT_NAME </code></li></ol>
 
−
 
−
<ol start="3" style="list-style-type: decimal;">  <li>Autograde
 
−
 
−
<code>nbgrader autograde ASSIGNMENT_NAME</code></li></ol>
 
−
 
−
<ol start="4" style="list-style-type: decimal;">  <li>Formgrade
 
−
 
−
<code> nbgrader formgrade</code>
 
−
 
−
Opend browser to <code>https://jupyter.cs.earlham.edu/hub/nbgrader/cs128</code>
 
−
 
−
In the upper right hand corner, you can change the view to view students or assignments.</li></ol>
 
−
 
−
* Before you can manually grade the assignments by running <code>nbgrader  formgrade</code>, you have to run <code>nbgrader autograde</code> even if the assignments were set to be manually graded (don't ask me why, I don't know right now). Now if you run nbgrader formgrade again and go to the web browser, you should be able to see all of the assignments and students, and submissions. You click on an assignment to grade it.
 
−
* nbgrader formgrade is the command used to manual grade notebooks. This runs in the browser and connects with JupyterHub. You will be able to see all of the students and assignments release and whether or not they have submitted anything. Run nbgrader formgrade and then direct your browser to jupyter.cs.earlham.edu/hub/nbgrader/cs128. You will see assignments and students.
 
−
* collect submitted assignments: <code>cd “course” (ie cs128); nbgrader collect LabName</code>
 
−
 
−
<ul>
 
−
<li>This populates the ‘submitted’ folder</li>
 
−
<li><p>If you get a bunch of errors and warnings about JPY_API_TOKEN, you need to go back and run</p>
 
−
<pre>cd “course”; export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)</pre></li></ul>
 
−
 
−
<ol start="4" style="list-style-type: decimal;">
 
−
<li>autograde the submissions: <code>cd “course”; nbgrader autograde LabName</code> # this can take a while</li></ol>
 
−
 
−
* If there are problems with the student’s submission (e.g. spaces in filenames, extra notebooks), they will manifest themselves here with a long error message. Use mv or rm depending on the nature of the problem.
 
−
* Just keep doing the up arrow, return, fix something, up arrow, return, fix something loop until autograde does not return any errors.
 
−
* If you can’t find a specific user file reference in the error message see b. Above... autograde populates the ‘autograded’ folder
 
−
 
−
<ol start="5" style="list-style-type: decimal;">
 
−
<li>manually grade the submissions: <code>cd “course”; nbgrader formgrade</code> then opon your $BROWSER to jupyter.cs.earlham.edu/hub/nbgrader/cs128</li></ol>
 
−
 
−
<ul>
 
−
<li>In the upper right hand corner is a drop-down that will let you sort by student or by submission.</li>
 
−
<li>If you receive “address in use” ps auxw | grep formgrade and kill -9 ...</li>
 
−
<li><p>If you get a bunch of errors and warnings about <code>JPY_API_TOKEN</code>, you need to</p>
 
−
<pre>export CONFIGPROXY_AUTH_TOKEN=036536a1e95a4d4d83907648238eaa8e; export JPY_API_TOKEN=$(jupyterhub token --db=sqlite:////etc/jupyterhub/jupyterhub.sqlite -f /etc/jupyterhub/jupyterhub_config.py)</pre></li>
 
−
<li><p>If you get <code>errno 13 permission denied</code>:</p>
 
−
<pre>chown -R nbgrader:nbgrader /run/user/11040 (each time you login? seems to be owned by the underlying UID:GID)</pre></li></ul>
 
−
 
−
<ol start="6" style="list-style-type: decimal;">
 
−
<li>OR, alternatively, run <code>../formgrade.sh</code> from within a given course directory like cs128, cs256, etc.</li>
 
−
<li><code>ssh home.cs.earlham.edu</code></li>
 
−
<li><code>ssh tools.cs.earlham.edu</code></li>
 
−
<li><code>sudo su - nbgrader</code></li>
 
−
<li><code>cd &lt;course&gt;</code></li>
 
−
<li><p><code>../formgrade.sh</code></p></li>
 
−
<li><p>In general, if you ever want to do one of these operations for only one student (because they were late or something, you can use <code>--student [studentid]</code></p></li>
 
−
<li><p>If an assignment is missing from formgrade (this can happen if a student does an update), you can collect the assignments again, and then force the autograder to grade the new one with <code>nbgrader autograde [assignment-name] --student [studentid] --force</code></p></li></ol>
 
−
 
−
<pre>Arguments: (&quot;submitted/aabdul15/Lab11-1/Charlie's solution.ipynb&quot;,)
 
−
 
−
 
−
write the header row - “email, lab10-2, lab11-0, lab11-1, lab11-2”
 
−
read cs128/submitted
 
−
for each user in alpha order by username (may be easier to order in spreadsheet)
 
−
    check for 10.2 lab
 
−
    check for 11.0 lab
 
−
    …
 
−
    write email, 0|1, … </pre>
 
−
 
−
<ul> If you need to collect late assignments from students:
 
−
<pre>
 
−
nbgrader collect FilesFinally --student adomi14 --course cs128
 
−
nbgrader autograde "WhereIsThat?" --student adomi14 --course cs128 --force
 
−
</pre>
 
−
 
−
== Returning feedback to students ==
 
−
 
−
If you've set up your feedback script as described in the "Beginning of Term Notes" section, you can just use that, and you're done.
 
−
 
−
* Generate feedback: <code>cd &quot;course&quot;; nbgrader feedback LabName</code>
 
−
* Return feedback: <code>sudo python return_feedback.py LabName</code>
 
−
 
−
To enable the nbgrader assignment list extension for the students:
 
−
 
−
Each student should check that they have a <code>.jupyter</code> directory, and an nbconfig directory underneath that. After those directories are created, the student should run:
 
−
 
−
<pre>nbgrader extension install
 
−
nbgrader extension activate</pre>
 
−
They may have to stop their server and restart for the changes to take effect. (Using the Control Panel in the upper-right hand corner of a Jupityr window.)
 
−
 
−
 
−
=== Temporary Status Quo - 10.06.17 ===
 
−
 
−
Currently, it is possible to return feedback to students correctly, but it requires some fiddling. Dave thinks you may have to copy the gradebook file **back** into the cs128 directory; from there, you can run the feedback return script, and it appears to work. Check that it's actually working, however, before you tell students to look. If you do it wrong, feedback is in the feedback directory, but is either not returned to students or they get a version that hasn't been manually graded. You may have to edit the script to add --force to the nbgrader feedback command if you do it wrong the first time.
 
  
 
== Multiple Simultaneous Graders ==
 
== Multiple Simultaneous Graders ==
Line 239: Line 154:
  
 
<code>gb.remove_student("syweiss15")</code>  <ref>https://nbgrader.readthedocs.io/en/stable/api/gradebook.html</ref>
 
<code>gb.remove_student("syweiss15")</code>  <ref>https://nbgrader.readthedocs.io/en/stable/api/gradebook.html</ref>
−
 
−
 
−
== Beginning of Term Notes ==
 
−
 
−
At the beginning of each term:
 
−
 
−
* Using the addUsers.ipynb to add all of the students to your class.
 
−
* Make new file(s) akin to return_feedback-f17-314.py, changing the student IDs to the ones for your class(es)
 
−
* Make new file(s) akin to feedbackf17, changing the student IDs to the ones for your class(es) and changing the room numbers and references to the above file as appropriate.
 

Latest revision as of 19:43, 28 January 2022

Notes

  • There is a user nbgrader (password is on Google Drive) on tools.cs.earlham.edu that must be used for all setup, assignment creation, grading, etc.
  • nbgrader doc: http://nbgrader.readthedocs.org http://nbgrader.readthedocs.org/en/stable/api/gradebook.html
    • This documentation is extremely good, and should generally be referred to first.
  • Good resource: https://github.com/CSSIP-AIR/Big-Data-Workbooks/blob/master/manage_assignments.ipynb -(and interesting big data assignments)
  • Making assignments
  • Keyboard input does not work with auto-grading, grading generally?
  • Keyboard input may work with manual grading now
  • Notes to students (in each assignment)
  • Make sure your solution is named the same as the assignment.
  • Don’t use spaces in file names (ever). Avoid using special characters aside from -
  • Find our tools for working with nbgrader here.
  • At the finish of every semester, make a copy of the cs128 directory and name it as follows: sem_year_cs128 (with the corresponding semester as fall or spring) and move it to the cs128Archives directory.

Please also read:

Beginning of Term Notes: Setting up a New Class

  1. At least a week before the term starts, send the admins (admin [at] cs [dot] earlham [dot] edu) the list of student names and email addresses so the admins can set up accounts for them.
  2. Log in as nbgrader. From within the terminal, use nbgrader quickstart classname to create a new class, where classname is the name of the class (something like cs128-f21). Folders for cs128 classes are called cs128-f20 or cs128-s21, where f and s are "fall" and "spring" and the two digits are the year. This command will create a new folder for the class, plus a source folders and config file within it.
  3. Copy the contents of the source directory from last term’s course into the new directory's source directory (or whatever material from previous courses that you plan to use.) The source directory is where assignments are stored before they are released to students.
  4. The config file at ~nbgrader/.jupyter/nbgrader_config.py should be edited to update the two references to the old class so that they refer to the new class.
  5. You may need to restart nbgrader’s server at this point. From the jupyter interface, select “Control Panel,” then “Stop My Server.” After a moment, you will be able to select “Start Server.”
  6. Navigate to /srv/nbgrader/exchange. Move the previous term’s directory into the old/ directory. This will prevent students from ever seeing the old labs (and possibly getting them by mistake, rather than getting the new labs.) -- This step may now be unnecessary, but it shouldn’t hurt.
  7. Update the student name list in the file student-names-file. This file lives in nbgrader’s home directory. student-names-file-sample contains an example of what this should look like. If you have a .csv downloaded from The Heart, tools in addUsers.ipynb can be used to convert produce a student-names-file using:
    1. make_student_names_file(“csv_file_from_the_heart.csv”, “student-names-file”)
  8. You shouldn’t need to do this now, but you can use other tools in addUsers.ipynb to add and remove individual students from the database. For this to work, note that addUsers.ipynb must be inside the course directory for the class; if it is run from nbgrader’s home directory, this will not work. This is useful if a student joins late and you just need to add one student. Dave believes that this also works to change the student’s name.
  9. Currently, the student-names-file file is used to build up the student db. You can import names from the list using nbgrader db student import student-names-file The content in the student-names-file should follow the style required by nbgrader (the first line needs to be as follows, and the last line needs to be empty). If you generated student-names-file using the instructions in the previous steps, it should already look like this.
    1. Line 1: id,last_name,first_name, email
    2. Line 2: smaizaz18,Aizaz,Shah, smaizaz18@earlham.edu
  10. (Optional) You can use nbgrader db student list to check if the students has been added successfully
  11. (Optional) You can use nbgrader db student list --course-dir=”~/<course-directory>” to check the student list for a specified course.
  12. This step and the remaining steps may no longer be necessary. Copy the two feedback files, which have names like feedback-summer19.sh, and return_feedback-summer19.py, into the course directory for the new course.
  13. Update the names of those two files to refer to the correct term.
  14. Edit the feedback shell script with the name of the feedback .py file.
  15. Edit the feedback .py file by changing students = ["barbeda",...] to a list of the students in the class.

Managing Assignments

Preparing an Assignment

  1. Assignments can be found in the source directory within the course directory.
  2. Assignments should be updated before release - once an assignment is released, further changes to the source copy will not be reflected in the version that students get.

Releasing an Assignment

  1. Log in to the Jupyter web interface at tools.cs.earlham.edu as nbgrader.
  2. In Jupyter, navigate to the “Formgrader” tab.
  3. Choose an assignment you would like to release. Any assignment in the source directory should be visible here. If it's not, something has gone wrong during the setup process.
  4. “Generate” => “Release”
    1. Both of these should pop up log output boxes. Neither log output should contain errors, just information.

Student Work

  1. Once an assignment has been released, students can access it.
  2. When they log into Jupyter, they should have access to an Assignments tab. They can select an assignment to fetch from that tab. This places the assignment in a new directory in their home directory. They can complete their work in the workbook in that directory.
  3. When they are done, they can submit their work in the Assignments tab. They can submit work multiple times, but generally students just submit the work when it’s done.

Collecting And Grading Student Work

  1. Once students have submitted their work, you can Collect submissions using the “Collect” button in “Formgrader” tab. You can collect submissions as many times as necessary.
  2. There are several ways to Autograde the submission.
    1. Method A) In the terminal, run nbgrader autograde <Lab_Name>. To autograde one student’s submission, use nbgrader autograde <Lab-name> --student <student-id>
    2. Method B) From within the formgrader, after the lab has been collected, click the number of submissions for that lab, then click Autograde for each student you want it to autograde. This is useful if a student turns in work late.
  3. Once Autograding is complete, manually grade any parts of the assignments that can’t be autograded. Feedback is also added to student work at this time, and this is where partial credit can be awarded. In the formgrader, choose “Manual Grading” on the left. Then, select the labs and assignments you wish to manually grade. You can also adjust scores produced by the autograding system, in order to award partial credit. (Or even full credit, if the student’s work is correct but was mislabeled by the autograding system.)

Releasing Feedback

  1. After coursework has been graded, you can generate and release feedback using the appropriate buttons in the Manage Assignments screen within the formgrader. Students can then fetch the feedback in a process similar to how they fetch assignments.

Alternate method

Feedback can also be generated and distributed using the files feedback-summer19.sh and return_feedback-summer19.py (The names of these will change each term.) These must be located in the directory for the course.

To generate and return feedback for all students in the current course of lab <Lab-Name> $ ./feedback-summer19 <Lab-Name> feedback-summer19: this file read the student list from the file student-names-file, and generate feedback file for each student, then it will run the return_feedback-summer19.py return_feedback-summer19.py: this file will copy the feedback folder to the students’ home directory.

To return feedback for a specified student: $ nbgrader feedback --student <student-id> --assignment <LAB_NAME> --force $ sudo python return_feedback-f19.py <LAB_NAME> --student <student-id>

Make sure the lab-names.txt file in the root nbgrader directory is up to date (i.e. the names of the labs are correct). They need to be correct to use MOSS.

Upgrading nbgrader

  • Grab latest release of nbgrader from github along with further install information.
  • Copy source code zip to username@home.cs.earlham.edu
    • scp source-code.zip username@home.cs.earlham.edu:~/
  • login as nbgrader on jupyter and then become root.
    • sudo su -
  • If going through a shell, ssh tools from home.
  • Backup exsiting nbgrader install
    • /mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader /nbgrader-timestamp/nbgrader
  • Uninstall current version of nbgrader.
    • Ideally conda remove nbgrader should work, if you get NoPackagesFoundError try uninstalling with pip
    • pip uninstall nbgrader
  • Install new nbgrader with conda and specify the anaconda python 3.5 environment
    • conda install -c conda-forge nbgrader -n py35
  • Restart nbgrader user server from Jupyter control panel.
    • May take a couple minutes for jupyter come back online
  • Ability to switch between courses in assignment list extension #563
    • Needs to be added manually until it gets merged into nbgrader master.

Multiple Simultaneous Graders

Using multiple graders works as follows: One user should log in as nbgrader and run nbgrader formgrade. Once that has been run, as long as the formgrade session is up, any number of users can log in as themselves and grade by going to the jupyter.cs.earlham.edu/hub/nbgrader/cs128 link. Additional accounts that are allowed to grade can be added with line like: 

c.HubAuth.graders = ["nbgrader", "charliep", "barbeda", "kmmuter11"]

In the nbgrader_config.py file. Dave is currently uncertain what happens if multiple people attempt to grade while logged in as the same username, but this appears to work. It’s also unclear how conflicts are resolved, so it’s probably best if people avoid editing grades haphazardly.

To tell if formgrade is running, do a ps auxw | grep formgrade | grep -v

Then check to see if there’s one running from: 

/mnt/lovelace/software/anaconda/envs/py35/bin/python
/mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader formgrade

If there is, then somebody already has the formgrade session going, and you can just to to the web address.

Digging into nbgrader

Digging into nbgrader’s code should not be necessary for normal operation. which nbgrader will point you to /mnt/lovelace/software/anaconda/envs/py35/bin/nbgrader, (which isn’t a directory - just a file).

The main nbgrader source code directory is here: 

/mnt/lovelace/software/anaconda/envs/py35/lib/python3.5/site-packages/nbgrader/

There are other files located here as well: 

/mnt/lovelace/software/anaconda/pkgs/nbgrader-0.3.3-py35_0/lib/python3.5/site-packages/nbgrader/.

If you’re specifically looking for the assignment list extension, those are in nbextensions/static/assignment_list in either of those two code directories. There’s overlap in what code is in each and I (Dave) am not familiar enough with Anaconda to get their relationship.

If you get a message that looks like this: 

Traceback (most recent call last):
     File "/mnt/lovelace/software/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
       return_value = func(*args, **kwargs)
...
...
AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'

Conda may have gone out of sync. Try: 

CONDA_SSL_VERIFY=false conda update pyopenssl

Authoring Worksheets

  • If you put each of the tests in its own cell, then each one will be a separate validate check. This makes it so that the autograde doesn’t fail completely if they miss part of it.
  • Because autograder tests care about what order the arguments to a function appear in, the instructions for an exercise should suggest an order.

Removing Students

In a python notebook, run the remove_student(student_id) command. It will delete the student along with the student's submissions. Don't forget to import Gradebook from nbgrader.api and to create a gradebook object:

gb = Gradebook("sqlite:///gradebook.db")

gb.remove_student("syweiss15") <ref>https://nbgrader.readthedocs.io/en/stable/api/gradebook.html</ref>

Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Nbgrader_notes&oldid=17991"