sUTL language reference

This document defines sUTL, the language implemented by the Python and browser runners on this site. It provides one coherent reference where older documentation and implementations sometimes disagree.

Evidence policy

When sources disagree, use this order:

1. Explicit language documentation and examples whose intended result is stated.
2. Python 2 implementation and tests (sUTL_py, version 1.0.6), which saw extensive production use in 2017–2018.
3. Studio examples and the Haxe-generated JavaScript evaluator used to run them.
4. The earlier handwritten JavaScript implementation.
5. The old sUTL-spec README where later evidence did not supersede it.

Agreement among independent sources is stronger than any source alone. An implementation result is not made normative merely because it is reproducible. Open questions are recorded below and represented as provisional or disputed cases in the conformance corpus.

Data model

sUTL operates on MLSNBN values. These are the values that round-trip through JSON:

• map (JSON object with string keys)
• list (JSON array)
• string
• number
• boolean
• null

A transform is itself an MLSNBN value. There is no textual parser beyond JSON: evaluation dispatches on the shape and reserved keys of the transform.

Evaluation context

evaluate(source, transform, library) retains five pieces of context:

• source: the root input, selected by $
• parent scope: the caller’s scope, selected by @
• current builtin scope: the scope after builtin arguments are added, selected by ^
• library: named transforms, selected by *
• root transform: the original transform, selected by ~
• builtins: host-provided primitive operations

The public evaluate call begins with the source as both root source and local scope.

Transform forms and precedence

Maps with reserved keys are recognized before ordinary map transforms:

1. {"!": ...} — evaluate transform
2. {"!!": ...} — alternate evaluate transform
3. {"&": ...} — builtin or library call
4. {"'": ...} — quote transform
5. {":": ...} — colon/literal transform
6. any other map — map transform

Lists and strings can also use compact builtin syntax. All remaining scalar values are literal transforms.

Literal and structural transforms

Numbers, booleans, and null evaluate to themselves. A normal map or list evaluates each child transform and preserves the enclosing shape:

{
  "name": "^$.user.name",
  "active": true
}

Strings are literals unless they match compact builtin/path syntax.

Evaluate transform: !

{
  "!": "<transform producing another transform>",
  "argument": "<transform>",
  "*": {
    "local-library-name": "<transform>"
  }
}

The value of ! is evaluated to obtain a new transform. Other map members are evaluated to form its local scope. Optional * constructs a replacement local library. The constructed transform is then evaluated in that context.

This is sUTL’s principal metaprogramming and function-call mechanism.

Alternate evaluate transform: !!

The later Studio dialect separates the scope delta from the transform:

{
  "!!": {"'": "<sub-transform>"},
  "s": {"argument": "<transform>"},
  "*": {
    "local-library-name": "<transform>"
  }
}

The value of !! is evaluated to obtain the next transform. s is evaluated separately and merged into the current scope (or replaces a non-map scope). Optional * supplies a local library. The next transform is then evaluated in that context.

This form is documented on the original site and works in the handwritten JavaScript evaluator. The Studio/HaxeJS implementation contains the form but has a scope-merge defect, while the production Python 2 port omits it. Those implementation gaps do not override the explicit language documentation.

Builtin or library call: &

{
  "&": "builtin-name",
  "argument": "<transform>"
}

Arguments are the other map members. A matching library transform can be invoked using the same form; builtins may also be overridden through the library.

Quote transform: '

{"'": value} suppresses one level of normal evaluation while recursively processing the quoted structure. Inside a quote, {"''": transform} escapes back into normal evaluation.

{
  "'": {
    "unevaluated": "^$.x",
    "evaluated": {"''": "^$.x"}
  }
}

Colon transform: :

{":": value} returns value without recursively evaluating it. It is a compact whole-value literal form and is commonly used while constructing a transform for later evaluation.

List transforms and flattening

A normal list evaluates each item. A list beginning with the exact string "&&" evaluates the remaining items and flattens one list level:

["&&", [1, 2], 3, [4, 5]]

evaluates to [1, 2, 3, 4, 5].

Compact builtin syntax

List form

["&+", 1, 2, 3]

The first character controls result selection:

• &name returns the full builtin result.
• ^name returns the first item of a list result, or null.

Arguments are reduced left-to-right for binary builtins.

The logical-and builtin is named &&, so its compact list spelling is "&&&". This is distinct from the exact flatten marker "&&".

String form

A compact string is split on dots and processed like list form:

"^$.person.name"

is equivalent to a head-selecting call to path builtin $ with the successive path components "person" and "name".

Numeric path components are converted to integer indexes.

Paths

The sUTL syntax uses ^ for a head result and & for a list result, followed by one of these selectors:

• $ — root source
• @ — parent/caller scope
• ^ — current builtin scope
• * — library
• ~ — root transform
• % — explicitly supplied value

Examples:

• ^$ — root source
• ^$.person.name — first selected name
• &$.people.*.name — all names immediately under people
• &$.**.name — all recursively selected names
• ^*.map — the library transform stored as map

Path components:

• a string selects a map member
• an integer selects a list item
• * selects every immediate map value or list item
• ** recursively selects descendants

Missing selections produce an empty list in & form and null in ^ form. Path traversal is deliberately forgiving: an inapplicable component discards that branch rather than raising an error.

Raw pathing supplies its own starting value:

["^%", {"person": {"name": "Fred"}}, "person", "name"]

evaluates to "Fred".

The older spec’s #/## notation describes an earlier dialect and is not part of sUTL.

Builtins

The production Python implementation provides:

• arithmetic: +, -, x, /
• comparison: =, !=, <, >, <=, >=
• logical: &&, ||, !
• control: if
• collections: zip, removekeys, len, keys, values, makemap, reduce, quicksort, head, tail
• conversion/string: split, trim, pos, string, number, boolean, lower, upper
• paths: $, @, ^, *, ~, %

keys and values use sorted keys in the later Python implementation. This makes cross-host results deterministic and is adopted by the working dialect.

reduce exposes item, accum, and zero-based ix in the local scope.

Logical &&, ||, and ! return booleans after applying sUTL truthiness. The production Python implementation accidentally leaked Python’s operand-returning and/or behavior; the explicit site reference and both JavaScript implementations agree on boolean results.

Declarations, distributions, and libraries

A declaration wraps a transform:

{
  "name": "addone_example_org",
  "language": "sUTL",
  "transform-t": ["&+", "^@.value", 1],
  "requires": ["dependency"],
  "test-t": null
}

A distribution is an ordered list of declarations. compilelib recursively matches each required name against declaration-name prefixes. The transform is stored in the resulting library under the required name, not necessarily its full declared name.

test-t, when enabled, is evaluated with the declaration’s transform-t as source. A falsey result passes; a truthy result describes failure.

The working runner reports a missing requirement as a compile failure instead of silently constructing an incomplete library. Historical implementations were inconsistent here; explicit failure is the sUTL safety rule.

Known disagreements and unresolved semantics

These are not silently canonized:

• Division: Python 2 performs integer division for two integers; JavaScript uses floating-point division. The working runner uses mathematical/JSON division (5 / 2 is 2.5) and records the Python 2 result as historical.
• Eagerness: the Python structured-builtin path evaluates argument members before dispatch, even for if; quoting is therefore needed to delay complex branches. Whether this was wholly intentional remains open.
• ! implementation order: historical evaluators differ internally in whether the next transform or scope delta is evaluated first. !! makes the intended separation explicit; side-effect freedom limits most observable differences, but metaprogramming edge cases need more tests.
• Invalid operands: prose often specifies null, while host exceptions and fallback values differ between implementations. Cases must be adjudicated individually.
• zip length: Python’s builtin truncates to the shortest list, while the old spec says shorter lists are padded with null. Production behavior is retained only as historical evidence. The guide uses zip only with equal-length lists and makes no unequal-length guarantee.
• Library builtin overrides: the Python implementation contains a likely string-formatting defect in constructing _override_<name>. The runner uses the evidently intended spelling but marks override behavior provisional.
• Map ordering: only operations that explicitly sort keys are ordered. General map member order is not semantically meaningful.

Observed implementation differences

The executable corpus currently produces these historical differences:

• The handwritten JavaScript implementation predates deterministic sorted keys/values and the string/number conversion builtins.
• The production Python 2 implementation does not implement the documented Studio-era !! form.
• Studio/HaxeJS implements !! but merges the evaluated next transform into scope instead of the evaluated s delta, producing null in the corpus’s scope-delta case. The handwritten JavaScript implementation matches the site documentation.
• Studio/HaxeJS returns null for an unparseable numeric string, whereas the site reference and Python tests require 0; this is treated as a Haxe bug.
• Both JavaScript implementations allow host-language coercion in ordering comparisons (for example "2" < 10). The site reference explicitly requires numeric operands and false otherwise, so host coercion is treated as a bug.
• Neither JavaScript implementation exposes Python’s direct zip builtin in the same form. The older spec instead describes zip as a core-library transform, and also disagrees about truncation versus null padding.
• Python returns operands from logical &&/||; both JavaScript implementations and the site reference return booleans. Boolean results are therefore canonical.
• Python 2 integer division differs from both JavaScript implementations. The working dialect uses a JSON-number result and floating division.

Conformance rule

A normative conformance case needs either:

• explicit documentation plus at least one matching implementation/example, or
• agreement between two independent implementations and no contradictory documentation.

Cases supported only by one implementation are provisional. Known disagreements are represented as disputed observations, not pass/fail language requirements.

For teaching, prefer the explicit !! form when constructing higher-order transforms. The compact ! form remains part of the language for compatibility with production transforms, but its historical evaluation-order differences should not be used as a programming technique.