kartiandClaude Opus 5
4147de0282
The bridge becomes a runnable taskset. The crux is that these are continuous control environments and the agent emits text: the model issues a control decision that is held for N steps, and the hold length is measured rather than guessed. At one decision every 32 steps the scripted baseline returns -2.807 and reaches the waypoint 0% of the time, against 5.484 and 100% at every frame. Open loop — one decision for the whole flight — returns -3.764 against an inaction floor of -4.161. A bird that sets a course and leaves does no better than one that does nothing, which is why crow-nav carries the strongest rule-4 claim of the five spatial environments. The budget bites from both ends: 8 turns reach the goal 44% of the time, 12 reach 88%, 14 reach 100%. 32 episodes against Qwen3.8-27B: mean 0.4603, no malformed replies in 412, every episode replaying to an identical checksum, and 16 of 32 spending all 16 turns. The gate fires 43.75% of the time and separates the four scenarios cleanly — whatever is wrong with the four dead data-environment gates, it is not that binary gates cannot discriminate. The oracle comes from the worker's own op. Nothing recomputes in Python a number that came out of TypeScript, least of all the reward denominator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
26 lines
1.0 KiB
TOML
26 lines
1.0 KiB
TOML
# tera-crow-nav — fly a crow to a waypoint on sixteen replies.
|
|
# See environments/tera_spatial for what it measures and where these two integers
|
|
# came from (`measure_hold.py`, not judgement).
|
|
#
|
|
# uv run --project environments/tera_spatial eval @ configs/tera_crow_nav.toml \
|
|
# --model brain-qwen38-dspark \
|
|
# --client.base-url http://100.127.247.67:8001/v1 \
|
|
# --client.api-key-var SPARK_API_KEY --no-push --no-rich -c 8
|
|
#
|
|
# `node` >= 22.18 must be on PATH: the environment is TypeScript and is executed,
|
|
# not translated.
|
|
num_tasks = 16
|
|
num_rollouts = 2
|
|
|
|
[env.taskset]
|
|
id = "tera-crow-nav"
|
|
num_tasks = 16
|
|
max_turns = 16 # replies. Sixteen is where the ceiling first becomes reachable.
|
|
max_hold = 12 # steps one reply is held for. 16 x 12 = 192 against a 300-step cap,
|
|
# so what binds is the turn budget and never the simulator's own.
|
|
default_hold = 6 # what an unreadable reply costs. Never zero.
|
|
seed_base = 115
|
|
|
|
[env.agent.harness]
|
|
id = "null" # one completion per turn, no tools; the env programs the loop
|