{ "run_id": "b5a9f595342b438a8aa85a12a095d533", "timestamp": "2026-07-28T06:24:42+00:00", "target_id": "qwen3.6-35b-a3b-nvfp4@your-node", "target": { "id": "qwen3.6-35b-a3b-nvfp4@your-node", "display_name": "Qwen3.6-35B-A3B (NVFP4)", "serving": { "engine": "vllm", "model_name": "brain", "base_url": "http://your-node:8001/v1", "service": "spark", "max_model_len": 65536, "flags": { "enforce_eager": true, "gpu_memory_utilization": 0.55, "kv_cache_dtype": "fp8", "max_num_batched_tokens": 4096, "moe_backend": "flashinfer_b12x", "reasoning_parser": "qwen3", "tool_call_parser": "qwen3_coder", "enable_auto_tool_choice": true } }, "host": "your-node", "family": "qwen3.6", "params_b": 35, "active_params_b": 3, "quant": "nvfp4", "checkpoint": null, "tier": "production", "aliases": [ "brain", "spark" ], "notes": "Live behind the `brain` and `local-moe` aliases. This is the model the bench exists to interrogate: it was deployed without measurement. FLAGS OF CONCERN: --enforce-eager disables CUDA graphs (costs throughput) and --gpu-memory-utilization 0.55 leaves ~45% of unified memory unused, capping KV cache and therefore concurrency. Both are worth an A/B once the baseline is recorded.\n", "slug": "qwen3.6-35b-a3b-nvfp4__your-node", "inspect_model": "openai-api/spark/brain" }, "host": { "id": "your-node", "ssh": "your-node", "hardware": "NVIDIA GB10 Grace Blackwell (DGX Spark)", "memory_gb": 121, "memory_bandwidth_gbs": 273, "notes": "Unified LPDDR5X. Memory bandwidth is the binding constraint on decode throughput, not compute -- large dense models are bandwidth-starved here while MoE models with small active-param counts do comparatively well.\n" }, "quality": [ { "task": "ifeval", "tier": "reference", "dataset_version": "2", "n_samples": 100, "metrics": { "prompt_strict_acc": 0.93, "prompt_strict_stderr": 0.025643, "prompt_loose_acc": 0.92, "prompt_loose_stderr": 0.027266, "inst_strict_acc": 0.95092, "inst_strict_stderr": 0.018685, "inst_loose_acc": 0.944785, "inst_loose_stderr": 0.019466, "final_acc": 0.936426, "final_stderr": 0.031066, "accuracy": 0.936426, "_limit": 100.0 }, "samples": [ { "sample_id": "13", "score": 0.0, "passed": false, "excerpt": "```json\n{\n \"park_name\": \"Brooklyn Prospect Park\",\n \"location\": \"Brooklyn, New York City\",\n \"area\": \"364 acres (core parkland)\",\n \"designers\": [\n \"Frederick Law Olmsted\",\n \"Calvert Vaux\"\n ],\n \"planning_and_conception\": {\n \"era\": \"1850s\",\n \"origin\": \"Brooklyn civic leaders and reformers proposed a large public 'pleasing ground' to improve public health, provide recreational space, ... [+3234 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": false, "inst_level_strict": 0, "prompt_level_loose": false, "inst_level_loose": 0, "num_instructions": 1 } }, { "sample_id": "102", "score": 1.0, "passed": true, "excerpt": "\"Are you absolutely sure the dress code isn't strictly formal?\" I asked, adjusting the heavy sequins on my midnight-blue ball gown.\n\n\"Relax, you look stunning,\" Mark said, pulling up the drawstring on his faded gray sweats. \"It's a charity rooftop soiree, not a state dinner. 'Elegant ease' is the actual vibe.\"\n\n\"Elegant ease,\" I echoed, lifting the voluminous tulle skirt to avoid tripping over the... [+742 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 1, "prompt_level_loose": true, "inst_level_loose": 1, "num_instructions": 1 } }, { "sample_id": "122", "score": 1.0, "passed": true, "excerpt": "\"\ud83d\ude80 From Dial-Up to Doxxing: How the Internet Actually Changed Your World \ud83c\udf10\u2728\n\nOkay, let's get real for a sec. If you've ever spent three hours doom-scrolling, argued about a meme in the comments, or tried to video call your friend across the room and failed, you're living inside the most chaotic, incredible machine ever built: the Internet. But did you know it didn't start as a place for memes and ... [+4004 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 1, "prompt_level_loose": true, "inst_level_loose": 1, "num_instructions": 1 } }, { "sample_id": "127", "score": 1.0, "passed": true, "excerpt": "**Navigating Your Parenting Plan: Assessing Time and Building a Sustainable Schedule**\n\nYou may have to meet with a helper to work out a parenting plan. The first would be to estimate how much time you have every day for parenting, and determine whether that is enough to support your child's growth and well-being. This initial assessment is the cornerstone of creating a realistic and effective fra... [+2333 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 2, "prompt_level_loose": true, "inst_level_loose": 2, "num_instructions": 2 } }, { "sample_id": "136", "score": 1.0, "passed": true, "excerpt": "* A nucleus sits at the atom's center.\n* Protons carry a positive electrical charge.\n* Neutrons remain electrically neutral.\n* Strong forces bind them tightly together.\n* Their combined mass defines atomic weight.\n* Proton count sets the element's identity.\n* Neutron count creates different isotopes.\n* This dense cluster holds most mass.\n* Together, they comprise the nuclear core.", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 2, "prompt_level_loose": true, "inst_level_loose": 2, "num_instructions": 2 } }, { "sample_id": "142", "score": 1.0, "passed": true, "excerpt": "\"Why do morphologists appreciate feline collaborators? Because a cat naturally demonstrates the derivational process: a single knock establishes the lexical root, while a second knock properly affixes the suffix, providing a perfect real-world example of how morphology builds complex words from minimal meaningful units.\"", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 3, "prompt_level_loose": true, "inst_level_loose": 3, "num_instructions": 3 } }, { "sample_id": "143", "score": 1.0, "passed": true, "excerpt": "Here is the step-by-step calculation:\n\n1. **Identify the initial height:** The colt starts at 5 feet tall.\n2. **Convert the monthly growth to feet:** 6 inches is equal to 0.5 feet (since 12 inches = 1 foot).\n3. **Calculate the total number of months:** 3 years \u00d7 12 months/year = 36 months.\n4. **Calculate total growth over the period:** 36 months \u00d7 0.5 feet/month = 18 feet of growth.\n5. **Add the g... [+349 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 2, "prompt_level_loose": true, "inst_level_loose": 2, "num_instructions": 2 } }, { "sample_id": "152", "score": 1.0, "passed": true, "excerpt": "the concept of a navigable freeway represents a fundamental shift in how we perceive and interact with the infrastructure that has dominated transportation for nearly a century. historically, freeways have been defined by their capacity to move high volumes of vehicles rapidly, yet they are often plagued by congestion, unpredictability, and a sense of chaotic navigation. when we apply the term nav... [+8740 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 2, "prompt_level_loose": true, "inst_level_loose": 2, "num_instructions": 2 } }, { "sample_id": "1000", "score": 1.0, "passed": true, "excerpt": "# *Early Life*\nRaymond III was born near the year 1140. His father held the official title Count of Tripoli. His mother carried valuable Armenian heritage. Young Raymond inherited his father authority after a violent assassination in 1152. The local nobility recognized him as ruler despite his youth. He spent his formative years learning statecraft and military command. The Frankish territories re... [+2153 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 3, "prompt_level_loose": true, "inst_level_loose": 3, "num_instructions": 3 } }, { "sample_id": "1001", "score": 1.0, "passed": true, "excerpt": "HEED WELL THE PATH ACROSS THE JAPANESE ISLES\nDAY ONE: ARRIVAL IN THE CITY OF TOKYO\nWHERE ELECTRIC TORCHES WAKE THE SLEEPING NIGHT\nSEEK ASAKUSA SHRINE UPON THE MORN\nAND TASTE THE MARINATED FISH FROM STREET\nDAY TWO: ASCEND TO CROWN OF FUJI\nCLIMB HIGH WHERE MIST DO WRAP THE SACRED PEAK\nGAZE DOWN UPON THE PROVINCES GREEN AND WIDE\nTHEN REST THY BONES WITHIN THE THERMAL SPRING\nDAY THREE: RIDE THE SILVER... [+916 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 1, "prompt_level_loose": true, "inst_level_loose": 1, "num_instructions": 1 } }, { "sample_id": "1005", "score": 1.0, "passed": true, "excerpt": "**[Full Name]** \n[City, State] | [Phone Number] | [Email Address] | [LinkedIn or Portfolio URL] \n\n**OBJECTIVE** \nMotivated and detail-oriented high school graduate seeking an entry-level position as a [Target Job Title] to contribute strong organizational, communication, and [Skill 1] abilities. Eager to apply academic knowledge, volunteer experience, and a strong work ethic to support [Company... [+2625 chars]", "metadata": { "scorer": "instruction_following", "epoch": 1, "prompt_level_strict": true, "inst_level_strict": 1, "prompt_level_loose": true, "inst_level_loose": 1, "num_instructions": 1 } }, { "sample_id": "1012", "score": 1.0, "passed": true, "excerpt": "Write an email to my boss telling him that I am quitting. The email must contain a title wrapped in double angular brackets, i.e. <