Problem setter's handbook


This document is dedicated to professionals and students alike eager to learn how to create programming problems (also called “programming assignments” or “programming challenges”) for the Problems module of the Sphere Engine service. 10 years of cumulative experience has enabled us to build a powerful tool that can be used to create high-quality content for educational, recruitment, and competitive programming purposes.

At Sphere Engine we focus on the essence of the topic at hand, meaning that we avoid any references to our systems' user interface. If you already have experience in creating programming problems for Online Judge systems you can go straight to the Problems Editor where you can find guidelines that will help you to learn about the user interface.

We will start with the naming conventions used in this handbook. This glossary provides references for all chapters, so please consult it if something is unclear.

Dictionary

Online Judge
Online Judges allow for automated evaluation of solutions to programming problems. Prominent examples of such systems include Sphere Engine Problems and Spoj (created by Sphere Research Labs).
Programming problem (or simply “problem”)
A challenge designed to be solved by a computer program, which can be defined in the form of a fictional story based on a common real-life scenario or as a purely mathematical statement.

It is a fundamental element of the Sphere Engine Problems service, consisting of:

  • description - text explanation of the nature of the problem along with input/output specification
  • test cases - data used for correctness verification (details below)
  • master judge - component responsible for setting the final result and score (details below)
Problem setter (also “author”, “tutor”)
The person who creates programming problems.
Problem solver (also “user”, “student”)
The person (usually programmer, developer) who tries to solve a programming problem.
Submission (also “solution”, “user's program”)
The source code, potentially a correct solution to the set programming problem.
Submission verdict
Short information - an answer - regarding the result of the submission execution. This verdict can be one of the following:
  1. accepted
  2. wrong answer
  3. time limit exceeded
  4. run-time error
  5. compile error
Test case
The set of data (usually a pair consisting of the input file and the model output file) used for verification of the user's submission. It contains additional components used in the verification process:
  • time limit - the maximum time allowed for the submission executed against the data from the input file
  • test case judge - the program that analyses the output generated by the user's solution (usually comparing the user's output data with the model output file)
Master judge
The program responsible for summarizing the results of individual test cases.