Frequently asked questions

Can I use nbgrader for purely manually graded assignments (i.e., without autograding)?

Yes, absolutely! Mark all the cells where students write their answers as “Manually graded answer” cells and then during grading run nbgrader autograde and the formgrader as normal. If you don’t want to even execute the notebooks, you can pass the --no-execute flag to nbgrader autograde.

Can I assign my own cell ids rather than using the autogenerated ones?

Yes, by default nbgrader will assign your cells an id like cell-8dc50d93bc80e053, but you can alter this to be something more descriptive if you like. You must use unique names for each graded cell.

Can I hide the test cells in a nbgrader assignment?

Yes, as of version 0.5.0 of nbgrader you will be able to hide tests in “Autograder tests” cells (see “Autograder tests” cells with hidden tests).

How does nbgrader ensure that students do not change the tests?

Please see the documentation on “Read-only” cells.

Does nbgrader support parallel autograding of assignments?

Not yet, though it is on the todo list (see #174). PRs welcome!

Does nbgrader protect against infinite loops?

Yes. nbgrader will stop executing a cell after a certain period of time. This timeout is customizable through the ExecutePreprocessor.timeout configuration option. See Configuration options.

Does nbgrader protect against unsafe code?

Not directly, but you can use a kernel inside a Docker container to do secure grading. Please see Grading in a docker container for details.

How does nbgrader handle late assignments?

By default nbgrader won’t explicitly assign late penalties, but it will compute how late each submission is. If you wish to customize this default behavior see adding customization plugins.

For this to work, you must include a duedate for the assignment and then a timestamp.txt file in the folder for each submission with a single line containing a timestamp (e.g. 2015-02-02 14:58:23.948203 America/Los_Angeles). Then, when you run nbgrader autograde, nbgrader will record these timestamps into the database. You can access the timestamps through the API, like so:

from nbgrader.api import Gradebook
with Gradebook("sqlite:///gradebook.db") gb:
    assignment = gb.find_assignment("ps1")
    for submission in assignment.submissions:
        print("Submission from '{}' is {} seconds late".format(
            submission.student_id, submission.total_seconds_late))

Note that if you use the release_assignment/fetch_assignment/submit/collect commands (see Exchanging assignment files), the timestamp.txt files will be included automatically.

Do I have to use sqlite for the nbgrader database?

No, and in fact, if you have multiple people grading accessing the formgrader at the same time we strongly encourage you not to use sqlite because it is not threadsafe. Postgres is also supported, and anything else that works with SQLAlchemy is likely to work (e.g. MySQL), though only sqlite and Postgres have been tested. If you want to use another SQL-based database and find that it doesn’t work for some reason, please open an issue!

Does nbgrader work with non-Python kernels?

Yes, though it hasn’t been extensively tested with other kernels and it is likely there are some edge cases where things do not work quite right. One thing in particular that you will need to do is customize how the student version is produced – for example, you will need to change the delimiters for the solution regions to use the appropriate comment marks for your language.

If you run into any issues using nbgrader with other kernels, please open an issue!

How do I get out grade information from the database?

nbgrader offers a fairly rich API for interfacing with the database. Please see Getting information from the database for more details.

Can I use the “Assignment List” extension with multiple classes?

New in version 0.6.0.

Yes! To use the “Assignment List” extension with multiple courses, you will want to set the following config options in your students’ nbgrader_config.py files:

from nbgrader.auth JupyterHubAuthPlugin
c = get_config()
c.Exchange.path_includes_course = True
c.Authenticator.plugin_class = JupyterHubAuthPlugin

The first option (Exchange.path_includes_course) will tell the transfer apps (i.e. nbgrader fetch_assignment, nbgrader submit, and nbgrader list) to assume that the paths for assignments include the course name, such as ./course101/ps1 rather than just ./ps1 (which is the default). Then, when using the “Assignment List” extension, students will be able to switch between different classes.

The second option (Authenticator.plugin_class) tells nbgrader to ask JupyterHub which courses the student is enrolled in. It does require a bit more setup than simply editing the nbgrader_config.py file, however. Please see JupyterHub Authentication for details.

Is nbgrader compatible with Windows/Mac/Linux?

Linux and Mac

nbgrader is fully compatible with Linux and also with Mac (with the exception of JupyterHub integration, as JupyterHub does not run on Mac).

Windows

The core pieces of nbgrader will also work on Windows: the “Create Assignment” extension, nbgrader generate_assignment, nbgrader autograde, nbgrader formgrade, nbgrader generate_feedback, nbgrader validate, and nbgrader export.

However, the parts of nbgrader corresponding to file management (the “Assignment List” extension, nbgrader release_assignment, nbgrader fetch_assignment, nbgrader submit, nbgrader collect, nbgrader list, nbgrader release_feedback, nbgrader fetch_feedback) will not work under Windows.

What happens if I do some manual grading, and then rerun the autograder?

If you rerun the autograder, nbgrader will never overwrite any manual grades or comments that you have added, and manual grades always take precedence over autogrades.

However, if you have given a manual grade, then rerun the autograder, and the autograder produces a grade as well, then it will mark that problem as “needing manual grade”. This functionality is primarily to aid you in grading in the scenarios where you want to grade a newer version of the student’s submission—for example, if you gave them a chance to revise it. In this hypothetical scenario, a student might have not completed a problem, leading you to originally assign it a low partial credit score. But then they turn in a newer version, which you run through the autograder and which attains full credit. Since the manual grade always takes precedence over the autograde, the student would still receive the low score unless you updated your grade: hence the motivation for marking it as needing to be manually graded (again).

Do students have to install anything on their own computers to use nbgrader?

No, nbgrader only needs to be installed for the instructor. However, students may optionally install the Validate extension to verify that their submission passes all the test cases.

What should I do if validation or grading of a notebook fails with a “Timeout waiting for execute reply” error?

This occurs because the validator or autograder is taking too long to validate or autograde your notebook. This can be fixed by adding the following line to nbgrader_config.py:

# increase timeout to 60 seconds
c.ExecutePreprocessor.timeout = 60

Can tests be only temporarily hidden, so that students can reveal them?

No, the tests are either present in the student version of the notebook or they are not. However, there exist extensions such as https://github.com/kirbs-/hide_code which can assist in hiding code cells.

Can I modify the notebook behavior during autograding or validation?

Yes, when running the autograder or validator, nbgrader sets the NBGRADER_EXECUTION environment variable, either to 'autograde' or 'validate'. You can check whether this variable is set and then modify the logic in the notebook accordingly. For example, you could use this to skip executing some cells during autograding only.