All interactions with the Exercism website are handled automatically. Analyzers have the single responsibility of taking a solution and returning a status and any messages.
two-fer
).analysis.json
file to the output directory.The analysis.json
file should be structured as followed:
{
"summary": "This solution looks good but has a few points to address",
"comments": [
{
"comment": "ruby.general.some_parameterised_message",
"params": { "foo": "param1", "bar": "param2" },
"type": "essential"
},
{
"comment": "ruby.general.some_unparameterised_message",
"params": {},
"type": "actionable"
},
{
"comment": "ruby.general.some_unparameterised_message"
},
"ruby.general.some_unparameterised_message"
]
}
summary
(optional)The summary field is a text (not markdown) field that summarizes the output. It might say something like "Your solution is nearly there - there's just two small changes you can make." or "The code works great, but there's a little bit of linting that needs doing.". This summary is rendered on the website above the comments.
comments
Comments are keys into website-copy/automated-comments/
, e.g. ruby.general.explicit_return -> automated-comments/ruby/general/explicit_return.md
.
Then can be structured either as single pointer strings (e.g. the last example above) or using a JSON Object to specify the follow keys:
comment
The pointer-string to a file in website-copy
.
params
(optional)A JSON Object containing any params that should be interpolated during rendering.
For example, in the markdown file, you could write Try %{variable_name} += 1 instead
, and then set params
to { "variable_name": "foo"}
in order to substitute %{variable_name}
for the actual variable that the student used.
When using parameterized files, ensure to escape all uses of %
by placing anther %
in front of it.
e.g. Try aim aim for 100%% of the tests passing
.
type
(optional)The following type
s are valid:
essential
: We will soft-block students until they have addressed this commentactionable
: Any comment that gives a specific instruction to a user to improve their solutioninformative
: Comments that give information, but do not necessarily expect students to use it. For example, in Ruby, if someone uses String Concatenation in TwoFer, we also tell them about String Formatting, but don't suggest that it is a better option.celebratory
: Comments that tell users they've done something right, either as a general comment on the solution, or on a technique.Comments without a type field default to informative
.
Currently in the website, we soft-block on essential comments, encourage students to complete actionable comments before marking as complete on Practice Exercises (but not Concept Exercises), but don't suggest any action on informative
or celebratory
.
However, in the future we may choose to add emojis or indicators to other types, or group them separately.
The contents of stdout
and stderr
from each run will be persisted into files that can be viewed later.
You may write an analysis.out
file that contains debugging information you want to later view.
NOTE: Everything below this line is currently changing.
Exercism is responsible for the display and communication of comments. The analyzer's job is purely to provide functional comments. Please follow these guidelines:
Comments should be actionable. The user should understand the action they need to undertake.
While friendly, they should not try and pretend to be a human and should not contain greetings, etc.
The solution should not act like the start of a discussion.
A good comment would be "This solution may be improved by doing XYZ".
A bad comment would be "Hello. Have you thought about doing XYZ?".
The goal of a language track on Exercism is to give people a way to achieve a high level of fluency at a low level of proficiency. We're aiming for fluency in the syntax, idioms, and the standard library of the language. You can read more about the goal of exercism here.
In the following paragraphs, keywords such as MUST, SHOULD, MAY are to be interpreted as in RFC2119; given that we recognize the following four output states and their restrictions:
approve
: MUST be an approvable solution, MAY be with comment.disapprove
: MUST be with commentrefer_to_mentor
: MAY be with commentPer RFC2119, if MUST is used, it is a guarantee that the rule is always so, and does not need to be guarded for. For example, MUST be without comment means that the website could crash if an analyzer sends a comment anyway. SHOULD indicates any consumer of the output must still guard against unwanted behavior.
A solution is approvable if it either matches the (set of) optimal solution(s) of that exercise, or if it shows understanding of the teaching goals of that exercise and every core exercise that came before. This means that an analyzer MAY approve a solution but still provide a set of comments as tips relating to the exercise or further improvements.
In this document, de facto is defined as follows:
In other words, if a nearly all developers (non-hobbyists) who write code in a certain language have established certain rules, these rules are a de facto standard and become idiomatic use. Example: Ruby uses 2 space indentation.
Some rules are language features, even if they are not documented well. These
language features are part of "idiomatic rules" and not stylistic choices.
Example: Ruby's MRI treats variables named _
differently.
Finally there are rules that are pure preferences, even though they might be
adopted by large bodies such as organizations and corporations. These rules
are usually part of competing standards. Exercism does not favor one over
another. Example: TypeScript has a linter tslint
(or eslint
+ plugin)
which is maintained by a company that is not Microsoft. It competes with other
linters such as xo
. Most of the rules are not language features or idiomatic
rules, and therefore stylistic choices.
For feedback generated by automated mentoring:
disapprove
when someone isn't using one of the
subjects the exercise is supposed to teach,disapprove
MUST have at least one comment,disapprove
SHOULD steer a student towards an approve
,refer_to_mentor
in a newer iteration, after a
disapprove
, given they follow that pathway.refer_to_mentor
MAY have one or more comments which will be given to
the mentor,It is currently undecided as to whether there is a minimum or maximum amount of :speech_balloon: comments. However, each comment SHOULD be aiding the student towards an approvable solution. Our recommendation is currently to aim for one to three comments per iteration.
Given the above, and to re-iterate that we focus on language fluency:
In general, if it's a language feature that will be caught by a compiler or
parser or based on official rules (which means there is a dependency on
correctness in tools), you should disapprove
, preferably linking
to the official rules.
- :-1: disapprove if there are official guidelines on naming conventions
- :speech_balloon: leave a comment if there is something noteworthy
- :nobell: if its a _stylistic preference, and there are competing standards, do not remark at all. Since there are competing standards, they're all preferences.
- :speechballoon: if it's a _stylistic preference, and there is one clear standard, comment on it. These rules enforce idiomatic code.
- :question: If it's a stylistic preference, and there is no clear standard, but most to all non-hobbyist have adopted the same style, this might be idiomatic. Comment at your discretion.
_
is treated differently,
_
for a variable name, but then uses it.constants
only if they start with a Capital Letter,
snake_case
for a class
namecAsInG
and name-ing
,
snake_case
is to be
expected by most IDEs and highlighting on exercism in code blocks._
).
*
means private
, which is not the case in JavaScript.golint
)gofmt
)The same rules apply as above. In general, if linting
and a specific format is
not part of the official language, and/or not integral to the language: