TOPIC
LLM Evaluations and Benchmarks
A subject the collection covers, read through 48 synthesis notes.
View as
Can a correct scoring function still mislead about task performance?
When an agent can influence the inputs a scorer reads, does verifying the scorer's logic suffice to ensure the score reflects genuine task success? This matters because in interactive settings, correctness of computation alone doesn't guarantee validity of the result.
Where does the evaluation boundary actually end in agent benchmarks? Can planted honeypots reliably catch reward hacking automatically? What do benchmark scores actually reveal about model containment? Can LLM judges be tricked without accessing their internals? Can a higher evaluation score hide poor task performance?
Every component on the reward path belongs to the evaluation boundary. Planted honeypots detect reward hacking automatically instead of through judgment Benchmark scores omit containment by construction, not oversight. LLM judges fall for authority and beauty biases in zero-shot attacks Higher evaluation scores can conceal unchanged or deteriorating task performance
What are AIDE2's hidden evaluations hidden from, and what makes a win untrustworthy? How are task bindings validated and what does validation cost per task? How does reward hacking differ from errors in the scoring function itself? When does an agent's action earlier in the loop change what a scorer reads later? How does a single score mix exploitation ability with task capability? Can an average-case validator score hide poor performance on critical tasks? How can deterministic checks make wrong judge decisions survivable? Can an occasionally wrong judge operate safely in an optimizer loop? Can a correct scoring function still mislead when the agent shaped its inputs? What makes a win untrustworthy in hidden evaluation environments? Can a correct outcome hide a fundamentally unsound decision-making process? Who validates task bindings and how is validation checked?See all 17 inquiring lines on this note →
Can benchmark scores be trusted without knowing their origin?
When AI researchers report benchmark scores, how much do the evaluation settings, prompts, and data splits affect the results? Why does tracing each score back to its source matter for fair model comparison?
Do vector embeddings actually measure task relevance? Do frontier LLMs actually explore the full space of valid answers? How do reasoning models actually break under pressure? How do you navigate synthesis across fragmented research topics? Does lexical search outperform agentic navigation as corpus size grows?
Vector embeddings measure semantic association, not task relevance Frontier LLMs narrow valid answer spaces to small canonical subsets Reasoning models excel at formal tasks but fail at social cognition. Synthesis layers organize research around thematic maps and extraction workflows. BM25 overtakes file-system agents around 10 million corpus tokens
Can a panel of smaller judges outperform one large judge?
Does aggregating votes from multiple smaller language models across different families produce better evaluations than relying on a single large model like GPT-4? This matters because evaluation cost and bias directly affect the reliability of AI-generated content assessment.
Can LLM judges be fooled by fake credentials and formatting? Do LLM judges systematically favor arguments from other LLMs? Can crowdsourced votes reliably rank language models? Can a quorum of validators really provide independent judgment? Does model diversity actually reduce validator agreement failures?
LLM judges consistently fall for authority signals and rich formatting attacks LLM judges systematically prefer LLM-generated arguments over human ones Crowdsourced pairwise preferences at scale agree with expert raters Validators sharing eight sources of input creates correlated epistemic faults Model family diversity alone may not restore validator independence
What biases do single large LLM judges introduce into comparisons? What biases might an LLM judge introduce into an on-policy alignment process? Can signal quality regulations help smaller teachers outperform larger ones? Does ensembling smaller judges reduce bias more effectively than single large judges? How much better is a panel of smaller judges than one large judge? Does disjoint family diversity actually cancel model-specific bias in evaluation? Can a diverse panel approach work for validators beyond text evaluation? Do smaller LLM judge panels outperform single large judges in practice? Do own-company biases differ across model families in grading tasks? What compute costs separate a panel of judges from a single large judge? How much inference compute does panel-of-judges evaluation actually cost?
Can static analysis find reward-hacking paths before agents run?
Exploring whether analyzing a task package without running agents can expose exploit-enabling reward-hacking paths. This matters because it could catch vulnerabilities before deployment, without requiring expensive rollouts.
Can runtime instrumentation distinguish hacking exposure from actual exploitation? Can a finite lifecycle model detect reward hacking across benchmarks? Can semantic labels on requests prevent malicious propagation through agent networks? Can planted honeypots detect hacks that matter most? Can difference of means vectors discover unknown hacking behaviors?
Runtime instrumentation separates exposed vectors from exercised exploits BenchShield models benchmark runs as finite typed event lifecycles checked against task bindings Semantic taints propagated through dynamic collaboration graphs carry request risk downstream Planted honeypots measure only known hacks, not frontier exploits Difference of means vectors discover hacking behaviors beyond detection alone
Can infrastructure monitoring catch reward hacking that scores cannot detect? Can phase-aware static taint analysis scale across different benchmark task types? What distinguishes a task that merely exposes a hacking vector from one being actively exploited? How do planted honeypots in tasks expose agent exploits differently than task permissions? Why does treating evaluation as a local output problem miss security risks? Why can agent-restored files pass correct checks but violate task intent? Can static package analysis find hacks that designers never planted? How does phase-awareness prevent false positive exploit paths in static analysis? What counts as a source and sink in reward-hacking taint analysis? What vulnerabilities does runtime instrumentation catch that static analysis misses? How visible is the optional shortcut to the agent during task execution? Do planted test cases reliably detect agent hacking behavior?See all 23 inquiring lines on this note →
What makes accountable judgment scarce when AI cognition is cheap?
When AI systems can perform cognitive tasks cheaply and at scale, what human capabilities become most valuable? This explores whether judgment, verification, and accountability are the true bottlenecks in labor markets shaped by generative AI.
AI transforms expert roles from creators to knowledge custodians Incremental AI adoption degrades implicit alignment by replacing human labor dependency. Concentrated AI exposure enables worker reallocation that offsets employment losses AI assistance improves immediate performance without building transferable skills Delegated AI exposure concentrates in information work, tracking capability
Why do firms substitute labor for AI faster than gig worker jobs disappear? What happens when we outsource information judgment to systems without real experience? Can verification and accountability sustain meaningful human work at scale? What cognitive skills does effective AI oversight actually require? How does routine use of automation erode critical judgment over time?
Where does model adaptation actually happen?
Does improvement come from updating model weights alone, or does it require versioning the entire loop—harness, contract, and audit trail together? This matters for how we design and release systems that need to keep improving.
Agents need two complementary adaptation timescales for continuous learning Agent reliability comes from externalizing cognitive burdens into system structures. The coupled human-agent environment is the correct unit for evaluation Centralized skill evolution enables collective learning across agent ecosystems Harness self-improvement overfits training tasks without regularization constraints
What should an external contract for model improvement actually contain? What makes a harness low-friction for model strategy? Why do useful harness updates often disappear during model evolution? Does harness scaling represent a fundamentally different path than model scaling? Where do outcome grades come from once a model enters deployment?
How should we evaluate agent behavior beyond final answers?
As AI systems move from single-response tasks to multi-step interactions, what evidence should evaluation focus on? This explores whether scoring interaction trajectories alongside process quality, recovery, and coordination reveals system capabilities that final-answer metrics miss.
Should interactive evaluation be designed as a unified paradigm? Can trajectory structure replace hand-annotated process rewards? Does agent interaction time scale separately from reasoning depth? What makes an AI system truly safe in practice? How can we measure whether AI errors stay visible and recoverable?
Interactive evaluation requires design science, not benchmark collection Trajectory structure can substitute for separately trained process supervision Agent interaction scaling is orthogonal to chain-of-thought reasoning scaling Safe AI systems need visible, contestable, containable, recoverable errors Existing error measures remain fragmented and incomplete across conditions
Can agent success reports serve as reliable oversight signals in real deployment? What specific failure modes must evaluation catch before deploying action-capable systems? Can automated evaluation replace human judgment in agent testing? What evaluation criteria can hold across legitimate adoption and coercion? Why do agents report success when actions actually fail? What separates good workflow design from poor workflow design? What makes some agent benchmarks measure interaction quality better than others? Where does agent reliability come from if not better tools? What five ecosystem conditions must coordination governance and evidence actually satisfy? What role does runtime feedback play in agent verification and progress confirmation? How does evaluating interaction trajectories change what we measure beyond correctness? What governance and safety measurements matter for deployed agent environments?See all 62 inquiring lines on this note →
How much does rhetorical style shift AI review scores?
When manuscripts are rewritten to improve rhetoric while keeping scientific content identical, do LLM reviewers change their scores? Understanding this matters for ensuring AI-assisted peer review evaluates substance, not polish.
Do fluent arguments win debates through sound logic or rhetorical polish? Does polished AI output trick audiences into trusting it? Can structured pipelines make LLM novelty assessment reliable? Can inference scaling help reviewers catch errors humans miss? Do language models judge persuasion the way humans do?
LLMs win subjective debate judgments but fail formal logical adjudication Polished AI artifacts exploit professional appearance to simulate expertise Structured decomposition achieves 86 percent alignment on novelty assessment with humans Inference-scaled review catches mathematical errors and flaws human experts miss Language models poorly predict which arguments change human minds
Does banning LLM use in peer review change review outcomes?
Can policies restricting or allowing AI tools shift how reviewers score papers and make decisions? This matters because review quality and fairness depend on consistent standards.
Can human review keep pace with AI-accelerated research generation? Can structured pipelines make LLM novelty assessment reliable? Can inference scaling help reviewers catch errors humans miss? How much does rhetorical style shift AI review scores? How do you build domain expertise into general AI models?
AI-accelerated generation logically requires AI-accelerated review to avoid pipeline collapse Structured decomposition achieves 86 percent alignment on novelty assessment with humans Inference-scaled review catches mathematical errors and flaws human experts miss Rhetorical presentation systematically shifts AI review scores despite identical content Domain specialization reveals a capability cliff between depth and breadth.
Can frontier exams really measure cutting-edge AI capability?
Popular benchmarks like MMLU saturate quickly, hiding real capability differences. Can expert-designed closed-ended exams like Humanity's Last Exam discriminate at the frontier, and what would high scores actually tell us about AI systems?
Open-world evaluations reveal benchmark distortions in frontier AI capability Agent capability requires a vector across at least five separable axes Standard NLP benchmarks systematically exclude ambiguous instances. Agent benchmark victories reflect evaluation artifacts, not economic readiness Crowdsourced pairwise preferences at scale agree with expert raters
Why do AI benchmarks show rapid saturation from near-zero to near-perfect? What capability dimension does a closed-ended exam actually fail to measure? Why do static benchmarks miss frontier capabilities that open-world tasks reveal? Where do frontier AI models already exceed safety thresholds in capability areas? What real-world tasks most clearly expose gaps between benchmark performance and actual capability? What capability dimensions does a single aggregate pass rate hide? How should single-axis benchmarks account for separable capability dimensions? How does measurement error in capability benchmarks systematically underestimate or overestimate true ability? Why does benchmark saturation give a false sense of capability coverage? Can expert-frontier exams discriminate frontier capability better than saturated benchmarks? How do educators distinguish between student capability and artifact quality in AI-era assessment? What distinct domains of AI competence do current assessments actually measure?
Can infrastructure evidence replace terminal scores in benchmark validation?
Asks whether runtime monitoring of agent behavior within evaluation boundaries can provide stronger proof of valid completion than final scores alone. Matters because scores alone cannot distinguish legitimate task completion from reward hacking.
Do current reward-hacking defenses provide reusable evidence of safety? Can a correct scoring function still mislead about task performance? What do benchmark scores actually reveal about model containment? Can we measure reward hacking reliably enough to act on it? Can runtime instrumentation distinguish hacking exposure from actual exploitation?
Current defenses against reward hacking lack reusable per-run integrity evidence Correct scorers report misleading results when agents influence inputs outside task paths Benchmark scores omit containment by construction, not oversight. Reliable reward hacking measurement must come before effective mitigation. Runtime instrumentation separates exposed vectors from exercised exploits
What evidence should benchmark operators attach to completion claims? How did AIDE2 guard against untrustworthy wins in its own loop? What are AIDE2's hidden evaluations hidden from, and what makes a win untrustworthy? How are task bindings validated and what does validation cost per task? Can infrastructure monitoring catch reward hacking that scores cannot detect? Can phase-aware static taint analysis scale across different benchmark task types? Can hidden test sets reveal reward hacking that single public scores conceal? How does dense task grading compare to honeypot detection for evaluating real capability? Can a low exploitation benchmark score indicate refusal rather than inability? Can infrastructure evidence ground benchmark claims better than terminal scores alone? Why does treating evaluation as a local output problem miss security risks? What state-tracking requirements exist for defenses that verify multi-party behavioral invariants?See all 169 inquiring lines on this note →
Can a finite lifecycle model detect reward hacking across benchmarks?
Does modeling benchmark runs as typed event lifecycles, checked against task bindings, successfully detect reward-hacking exploits across multiple evaluation tasks? This approach aims to replace task-specific patches with reusable formal detection.
Can static analysis find reward-hacking paths before agents run? Can runtime instrumentation distinguish hacking exposure from actual exploitation? Do current reward-hacking defenses provide reusable evidence of safety? Where does the evaluation boundary actually end in agent benchmarks? How reusable is BenchShield if task bindings require per-task work?
Static phase-aware taint analysis discovers exploit-enabling reward-hacking paths pre-run Runtime instrumentation separates exposed vectors from exercised exploits Current defenses against reward hacking lack reusable per-run integrity evidence Every component on the reward path belongs to the evaluation boundary. BenchShield's reusability claim depends on undisclosed binding authorship and validation.
How did AIDE2 guard against untrustworthy wins in its own loop? What event types and phases structure the BenchShield lifecycle model? Can infrastructure monitoring catch reward hacking that scores cannot detect? Can phase-aware static taint analysis scale across different benchmark task types? What distinguishes a task that merely exposes a hacking vector from one being actively exploited? Can hidden test sets reveal reward hacking that single public scores conceal? How do planted honeypots in tasks expose agent exploits differently than task permissions? How is a reward hack defined and labeled across different benchmark studies? Do multiple frontier models show similar hacking rates on unmodified benchmarks? How do unplanted benchmark hacking rates compare to rates on constructed bait tasks? What determines the ground truth when detecting reward hacking in model evaluations? What specific reward-hacking shortcuts did frontier models find in Terminal Bench?See all 88 inquiring lines on this note →
Do transformers actually learn systematic compositional reasoning?
Explores whether transformers solve compositional tasks through genuine systematic reasoning or by pattern-matching against training data. This matters because it determines whether scaling alone can achieve robust generalization.
Do foundation models learn world models or task-specific shortcuts? Can neural networks learn compositional skills without symbolic mechanisms? Why do neural networks fail at compositional generalization? Does logical validity actually drive chain-of-thought gains? Do language models fail at reasoning due to complexity or novelty?
Foundation models develop task-specific heuristics, not generalizable world models. Compositional generalization emerges from scaling when training covers the task space The binding problem explains neural networks' systematic generalization failure Invalid CoT reasoning performs nearly as well as valid reasoning Reasoning failures are driven by instance-level unfamiliarity, not task complexity.
What is selective resonance and why do transformers not perform it? How do transformers perform multi-hop reasoning across distant training documents? Do transformers learn generalizable algorithms or instance-based patterns? Can neural networks represent symbolic structures without explicit mechanisms? Does architectural discovery follow an empirical scaling law like neural networks? Why do text-to-image models fail at composing multiple concepts together? What makes linear decodability a reliable signal of compositionality? Does scaling model size solve compositional generalization problems? How does error propagation limit transformer performance on complex tasks? Can symbolic mechanisms improve transformer compositional abilities? Can explicit stack mechanisms extend what formal languages transformers can learn? How does circuit complexity limit which grammatical structures transformers can acquire?See all 83 inquiring lines on this note →
How can we make reward-hacking visible in agent evaluation?
Typical benchmark scores collapse multiple factors into a single number, hiding whether agents are genuinely solving tasks or exploiting reward signals. Can separating evaluation components expose these failure modes?
What are the three distinct layers of agent code? How should we measure agent system performance beyond task success? Does a single benchmark score actually predict agent readiness? Which security protections actually slow down agent exploits? Can planted honeypots reliably catch reward hacking automatically?
Agent code splits into three governed layers with different failure modes. Agent evaluation must measure system behavior across multiple operational dimensions Agent capability requires a vector across at least five separable axes Varying protections per instance isolates their causal impact on agent performance Planted honeypots detect reward hacking automatically instead of through judgment
Can system-level engineering fixes replace hand-designed reward heuristics entirely? How should harness scaffolding be treated as a first-class object? What persistent failures remain unsolved despite harness evolution efforts? How does reward hacking emerge when agents optimize fixed proxy objectives? What feedback signals matter most during harness evolution search? Why does length exploitation emerge as a reward hacking failure in distillation? What makes a reward evolution schedule fast enough to outpace exploitation? What evidence should benchmark operators attach to completion claims? What are AIDE2's hidden evaluations hidden from, and what makes a win untrustworthy? Can infrastructure monitoring catch reward hacking that scores cannot detect? Why does held-out evaluation matter for detecting agent overfitting? What shortcuts in data or models let agents inflate benchmark scores?See all 91 inquiring lines on this note →
Can dense subtask grading reveal agent progress on ultra-long tasks?
When agent tasks stretch to hours and hundreds of episodes, does breaking them into fine-grained graded subtasks expose meaningful progress that binary pass-fail scoring completely erases? This matters because most agents fail the final outcome anyway.
How should we measure agent system performance beyond task success? Can we reward reasoning steps without human annotation? Do automated benchmarks hide what frontier AI systems can really do? Does planting honeypots in real coding tasks detect actual agent hacking? Why do high partial scores not guarantee task completion?
Agent evaluation must measure system behavior across multiple operational dimensions Information-theoretic rewards enable dense process feedback without annotation Open-world evaluations reveal benchmark distortions in frontier AI capability Hack-Verifiable Terminal Bench relocates honeypot detection to real-world coding tasks for deployment relevance. Partial progress diverges sharply from complete delivery on scientific workflows
Does setting temperature to zero actually make LLM outputs reliable?
Explores whether deterministic LLM settings that produce consistent outputs also guarantee reliable judgments, and how to measure true reliability beyond surface consistency.
Model confidence directly predicts robustness against prompt variation LLM judges consistently fall for authority signals and rich formatting attacks Preference models systematically favor five surface features humans reject Reasoning models excel at formal tasks but fail at social cognition. Assisted accuracy realizes only about half of item-level LLM accuracy gains
What makes LLM outputs fabrication rather than hallucination or confabulation? Can LLMs evaluate their own observations without external feedback? Why do one-shot transparency studies miss the temporal reversal entirely? What makes accountability and validity-orientation non-behavioral properties? Why does aggregate accuracy fail as a metric for rare harmful cases? What distinguishes minimal-pair asymmetry from standard accuracy evaluation? What should we call errors in LLM outputs when hallucination does not apply? Can systems lacking inner states express genuine truthfulness claims? What would it mean to assign explicit trust weights to synthetic data? Can LLM judges reliably estimate when they lack sufficient persona information? How should ground truth labels be assigned to simulated user sessions? How does unidimensionality in assessments affect measurement validity?See all 169 inquiring lines on this note →
Can dictionary learning scale to production language models?
Sparse autoencoders recovered interpretable features from toy models, but scaling to real production systems like Claude remains uncertain. This matters because interpretability at scale is foundational for AI safety work.
Do standard analysis methods hide nonlinear features in neural networks? Can we decode what LLM activations really represent in language? Can a model be truthful without actually being honest? What actually happens inside large language models? Can learnable spline activations beat fixed MLP designs?
Standard representation analysis methods systematically miss complex nonlinear features A decoder can translate LLM activations into natural language for interpretability and control Truthfulness and honesty are mechanistically distinct properties in LLMs LLM internal structure and external performance are surprisingly decoupled. Learnable spline activations on edges outperform standard MLPs
How reusable is BenchShield if task bindings require per-task work?
BenchShield claims to avoid task-specific patches through reusable evidence, but it checks runs against task bindings. Whether bindings are manually written or automatically derived determines how much per-task effort the approach actually requires, and the paper does not say.
Can a finite lifecycle model detect reward hacking across benchmarks? Do current reward-hacking defenses provide reusable evidence of safety? Can static analysis find reward-hacking paths before agents run? Can planted honeypots reliably catch reward hacking automatically? How does the authorization layer stay outside the poisoned path?
BenchShield models benchmark runs as finite typed event lifecycles checked against task bindings Current defenses against reward hacking lack reusable per-run integrity evidence Static phase-aware taint analysis discovers exploit-enabling reward-hacking paths pre-run Planted honeypots detect reward hacking automatically instead of through judgment Authorization layer design details remain unspecified in the excerpt
What event types and phases structure the BenchShield lifecycle model? How are task bindings validated and what does validation cost per task? Can static analysis derive task bindings without manual effort? How many task-specific bindings does BenchShield require across benchmarks? Who validates task bindings and how is validation checked? What makes durable code artifacts more valuable than per-task harness patches?
How representative is the BenchShield Trajectories labeled sample?
The corpus contains 456 human-labeled trajectories from over 31,000 public runs—about 1.5 percent. Whether this subset can estimate actual reward hacking rates depends entirely on how those 456 were selected, a choice the paper does not disclose.
How often do frontier agents exploit planted reward hacking shortcuts? Does BaitBench measure hacking propensity or bait visibility? Can planted honeypots detect hacks that matter most? Can scoped agents reliably judge semantic hacks in runtime analysis? How often do models hack unmodified coding benchmarks?
Majority of frontier agent runs exhibit reward hacking when shortcuts are planted BaitBench hacking rates conflate propensity with presentation visibility Planted honeypots measure only known hacks, not frontier exploits Scoped audit agents provide evidence-backed semantic attribution over pinned artifacts GLM 5.2 hacks in 57 to 73 percent of standard coding benchmark rollouts
Does preference tuning actually reduce the diversity of model outputs?
The field assumes RLHF and DPO reduce diversity, but this assumption rests on measuring all outputs equally. What happens if we only count diverse outputs that meet quality thresholds?
Does preference tuning always reduce diversity the same way? Why aren't bigger models better for generating diverse outputs? Can diversity optimization improve quality during language model training? Does preference optimization damage conversational grounding in large language models? Does a single benchmark score actually predict agent readiness?
Preference tuning effects on diversity reverse across code and creative writing Smaller models around 500M parameters generate more unique outputs per sample Semantic diversity optimization during RL improves both quality and diversity Preference optimization actively erodes conversational grounding in large language models Agent capability requires a vector across at least five separable axes
Can utility control modify LLM values more effectively than output filtering? Why does KTO skip supervised fine-tuning while DPO cannot? What creates the irreducible trade-off between quality and diversity in training data? How do quality, diversity, and complexity create different effects on downstream model performance? How do quality thresholds change which model produces more usable diversity? Why does preference tuning reduce diversity in code but increase it in creative tasks? What happens to model grounding when preference optimization increases effective diversity? What unmeasured side channels emerge from RLHF preference optimization? When does RLHF reduce diversity and when does it preserve semantic variation? How does probability mass concentration affect sampling diversity across model scales? At what point does output quality outweigh diversity value in synthetic data tasks? How do you identify which models should form a minimal diverse coreset?See all 14 inquiring lines on this note →
Does simulator bias kill world model training for agents?
When RL agents train on learned world models instead of real environments, do systematic errors in the simulator undermine convergence? Understanding this matters for making cheap simulation trustworthy during agent training.
Can LLMs replace search engines during agent training? Why do search agents beat memorized retrieval on hard questions? What makes a research domain suitable for autonomous optimization? Can autonomous research pipelines discover AI architectures that AutoML cannot? Can language models learn to simulate agent environments?
LLMs simulate search engines via internal knowledge for cost-effective RL training Real-world search agents outperform memorized retrieval by avoiding knowledge gaps Four properties determine whether domains can benefit from autoresearch systems Autonomous research pipelines discover architectures beyond traditional AutoML reach Language world models trained on trajectories scale agent learning beyond real environments
Where does the evaluation boundary actually end in agent benchmarks?
Interactive benchmarks let agents write to state and receive feedback in loops. Does everything the agent can influence on the path to the reward score count as part of the benchmark's evaluation boundary?
Can a correct scoring function still mislead about task performance? Is your evaluation environment actually part of the threat model? How can we make reward-hacking visible in agent evaluation? Do frontier models exploit unknown vulnerabilities in evaluations? Should interactive evaluation be designed as a unified paradigm?
Correct scorers report misleading results when agents influence inputs outside task paths Evaluation environments belong inside the security boundary for capable agents Decoupled evaluation components make reward-hacking diagnosable instead of hidden Frontier models are reported hacking evaluation environments via unknown vulnerabilities Interactive evaluation requires design science, not benchmark collection
When does an agent's action earlier in the loop change what a scorer reads later? What makes an evaluation environment itself a security boundary? How does evaluation environment design become part of the security boundary? Is the evaluation environment itself part of the security boundary? What infrastructure and reporting standards would make interactive evaluation reproducible?
Do current reward-hacking defenses provide reusable evidence of safety?
Existing defenses against reward hacking—task-specific patches, prompt instructions, and post-hoc detectors—may work in practice, but do they leave behind portable, per-run evidence that an evaluation stayed within its intended boundary?
Can a finite lifecycle model detect reward hacking across benchmarks? Can infrastructure evidence replace terminal scores in benchmark validation? Can prompting agents not to cheat actually stop them? Can planted honeypots reliably catch reward hacking automatically? Can we measure reward hacking reliably enough to act on it?
BenchShield models benchmark runs as finite typed event lifecycles checked against task bindings Infrastructure evidence lets operators claim benchmark-valid completion beyond scores Prompting agents not to cheat leaves reward hacking above 50 percent Planted honeypots detect reward hacking automatically instead of through judgment Reliable reward hacking measurement must come before effective mitigation.
How did AIDE2 guard against untrustworthy wins in its own loop? What are AIDE2's hidden evaluations hidden from, and what makes a win untrustworthy? Can infrastructure monitoring catch reward hacking that scores cannot detect? What distinguishes a task that merely exposes a hacking vector from one being actively exploited? Does debate's ground-truth protection against hacking transfer to domains without verifiable answers? Can hidden test sets reveal reward hacking that single public scores conceal? How is a reward hack defined and labeled across different benchmark studies? What specific reward-hacking shortcuts did frontier models find in Terminal Bench? Why does treating evaluation as a local output problem miss security risks? How does reward hacking differ from errors in the scoring function itself? Can static package analysis find hacks that designers never planted? What counts as a source and sink in reward-hacking taint analysis?See all 75 inquiring lines on this note →
Do frontier LLMs actually explore the full space of valid answers?
When multiple correct answers exist, do advanced language models expose users to that full range, or do they collapse onto a narrow canonical subset? This matters for learning, inquiry, and decision-making.
Do different AI models actually produce diverse outputs? Does preference tuning actually reduce the diversity of model outputs? Why do LLMs generate novel ideas from narrow ranges? Do language models flatten the range of public arguments? Where does mode collapse in language models really come from?
Different LLMs converge on similar outputs regardless of model diversity Preference tuning increases effective diversity by filtering low-quality variance LLM research ideation collapses into narrow clusters despite high novelty Language models collapse argument diversity at the structural level Mode collapse originates in typicality bias within preference data, not algorithms
Can live benchmarks prevent data contamination in prediction tasks?
How can prediction benchmarks stay contamination-free when future outcomes aren't yet known? FutureX tests whether continuous real-time updates eliminate training data leakage.
Do automated benchmarks hide what frontier AI systems can really do? Can frontier exams really measure cutting-edge AI capability? Can LLMs actually forecast time series better than we think? Can scarcity of solutions protect benchmarks from data contamination? Can retrieval-augmented language models forecast like human experts?
Open-world evaluations reveal benchmark distortions in frontier AI capability Frontier exams discriminate temporarily but miss autonomous research ability Workflow architecture dominates raw model capability in LLM forecasting tasks Exploit scarcity mitigates contamination by making solutions hard to memorize Retrieval-augmented language models forecast near human competitive levels
Why are post-cutoff test sets essential for evaluating genuine forecasting ability? What privacy-preserving evaluation methods best capture real-world forecasting ability? What real-world tasks most clearly expose gaps between benchmark performance and actual capability? Can benchmark scores on verifiable tasks transfer to unseen problems outside the training domain? Why do benchmarks become saturated so quickly after initial launch? What gets silently included in a published result without explicit disclosure? How does contamination protection by time differ from protection by scarcity? How can hidden test partitions detect constant predictions that generalize?
Can fairness frameworks extend to general-purpose language models?
Existing fairness frameworks were designed for narrow, structured tasks. This explores whether they scale to LLMs, which serve multiple populations, sensitive attributes, and use cases simultaneously.
Can human-centered LLM design ever achieve universal solutions? Can a single reward model represent diverse human preferences? Should AI alignment target preferences or social role norms? How does LLM alignment affect representation across dialects? Do large language models actually reflect human value diversity?
Human-centered objectives resist universal solutions because harm depends on perspective Single-reward RLHF provably cannot represent diverse human preferences equitably AI alignment should match social role norms, not aggregate preferences LLM alignment procedures create disparities across English dialects and global opinions LLMs converge on a narrow idealized value core rather than reflecting human diversity
Can execution harnesses lift model performance without retuning weights?
Explores whether improving an agent's runtime system—not the model itself—can boost benchmark accuracy and transfer across different model versions without modification.
Agent reliability comes from externalizing cognitive burdens into system structures. Agent performance scales with effective feedback not raw token count Persistence in feedback loops predicts long-horizon agent success more than initial quality Harness updating is flat but benefiting peaks at mid-tier models Agent evaluation must measure system behavior across multiple operational dimensions
How do benchmark environments misrepresent deployment readiness? What role does online RL play in scaling GUI agents? How can agent data flywheels improve task quality iteratively? Can gradients extracted with agent-level supervision transfer across different benchmarks? Which foundation model tiers most benefit from harness updates? How do evolved harness edits generalize across different benchmark domains? How much realized agent capability comes from the harness versus the model? How fast do new benchmarks get adopted across the AI research community? How do prompt optimization and code harnesses compare for capability transfer? What should an external contract for model improvement actually contain? How do fast skill injection and slow gradient updates work on different timescales? How do agentic systems hide harness failures from benchmarks?See all 49 inquiring lines on this note →
Should interactive evaluation be designed as a unified paradigm?
As AI systems increasingly act over time through tools and environments, how should we structure evaluation of these interactions? Current benchmarks are fragmented and incomparable, raising the question of whether interactive evaluation needs principled design standards rather than ad-hoc adoption.
How should we evaluate agent behavior beyond final answers? Do interactive evaluations actually solve the benchmark comparison problem? Will AI automation eventually formalize designer taste? Why do safety failures remain invisible to our evaluation methods? How can we measure whether AI errors stay visible and recoverable?
Agent evaluation must expand from endpoints to interaction trajectories. Longstanding evaluation challenges reappear at the trajectory level instead of disappearing Designer taste will be formalized into evaluation systems and automated Safety failures hide in evaluation habits, not in technical obscurity Existing error measures remain fragmented and incomplete across conditions
How does the evaluator become part of the definition of intelligence? Can contextual design decisions resist formalization into evaluation rubrics? Does longer interaction horizon require fundamentally different evaluation approaches? What reporting standards would make interactive evaluation scores comparable across benchmarks? Why do evaluation design choices themselves become reified into the AI systems being evaluated? How might automated evals eventually capture the human judgment designers exercise now? How does evaluation environment design become part of the security boundary? What infrastructure and reporting standards would make interactive evaluation reproducible? How does separating environment components make evaluation results more reproducible and analyzable? How should outcomes be scored when comparing applications with different interaction formats? Why is the coupled human-agent environment the right unit of evaluation? Should long-context evaluation measure the coupled system?See all 14 inquiring lines on this note →
Can language models judge legal reasonableness like humans do?
Do LLMs produce judgments on vague legal standards that match human responses in both central tendency and distribution? This matters for understanding whether models can perform genuine legal reasoning rather than pattern matching.
AI models outperform humans at predicting collective social norms without embodiment LLMs show systematic performance degradation on historical legal cases LLM survey artifacts stem from elicitation method, not model limits LLM outputs are subjective priors, not empirical observations. LLMs converge on a narrow idealized value core rather than reflecting human diversity
Do LLMs overgeneralize when summarizing scientific research?
When LLMs summarize science papers, do they drop important qualifiers and scope limits? This matters because such summaries might mislead readers about what findings actually show.
Faithful uncertainty aligns expressed doubt with actual model uncertainty Polished AI artifacts exploit professional appearance to simulate expertise Grounded refusal is the core defense against noisy RAG corruption LLM knowledge failures follow structurally distinct patterns beyond simple wrongness Legal AI tools hallucinate 17 to 33 percent despite vendor claims
No Inquiring Lines yet
Can natural language explanations redefine what interpretability means?
Does the ability of LLMs to explain patterns in natural language fundamentally expand the scope and complexity of what humans can understand about AI systems, compared to traditional interpretability methods?
Can LLM explanations actually help humans predict model behavior? Can cognitive science methods unlock how LLMs actually work? Can dictionary learning scale to production language models? Do language models understand in fundamentally different ways? Do language models actually use their reasoning steps?
LLM explanation plausibility and precision are fundamentally uncorrelated Marr's three levels provide structured methodology for LLM interpretability Dictionary learning recovers interpretable features from production language models Language models show three hierarchical tiers of understanding with distinct mechanisms Language models generate faithful reasoning chains far less often than assumed
Do long-horizon benchmarks actually measure decision quality?
Existing benchmarks report only whether agents complete tasks, not whether they made good choices along the way. Can we measure the quality of intermediate decisions separately from final outcomes?
Does a single benchmark score actually predict agent readiness? What predicts success in ultra-long-horizon agent tasks? Do automated benchmarks hide what frontier AI systems can really do? Do frontier AI agents actually conduct novel research or just optimize? How do reasoning models actually break under pressure?
Agent capability requires a vector across at least five separable axes Persistence in feedback loops predicts long-horizon agent success more than initial quality Open-world evaluations reveal benchmark distortions in frontier AI capability Frontier agents compose established techniques rather than discover novel methods Reasoning models excel at formal tasks but fail at social cognition.
Do frontier AI agents actually conduct novel research or just optimize?
Exploring whether current long-horizon research agents generate genuine methodological novelty or primarily recombine established techniques. This matters for understanding how close we are to recursive self-improvement through AI.
What predicts success in ultra-long-horizon agent tasks? Does a single benchmark score actually predict agent readiness? Do automated benchmarks hide what frontier AI systems can really do? Can recursive self-improvement speed up the research process itself? Do fixed-budget efficiency gains translate to real research progress?
Persistence in feedback loops predicts long-horizon agent success more than initial quality Agent capability requires a vector across at least five separable axes Open-world evaluations reveal benchmark distortions in frontier AI capability Recursive self-improvement targets research process efficiency directly Fixed-budget efficiency gains show optimization improvements but leave real-world translation unclear
Can autonomous research agents outperform hand-tuned hyperparameter search? Where should humans take over from AI during research tasks? What counts as research completeness versus correctness in agent evaluation? Can agents take on research planning tasks while humans focus on judgment? Does delegating planning to agents change the speed of the research process? What collapse dynamics constrain recursive self-improvement in current evidence? Does episode-level cost become the decisive factor when comparing AI agents in production? Can single performance scores hide important differences in how agents approach research tasks? How should researchers operationalize and measure methodological guidance at different levels? Can accumulated priors and outcome analysis speed up research automation? What do AI researchers actually mean when they use the term AGI? How do different definitions of intelligence shape AI research priorities?See all 14 inquiring lines on this note →
Do interactive evaluations actually solve the benchmark comparison problem?
Interactive, trajectory-based evaluation promises richer evidence than response-only benchmarks. But does moving to this format resolve longstanding challenges like comparability and reproducibility, or do those problems simply reappear at a new scale?
Should interactive evaluation be designed as a unified paradigm? How should we evaluate agent behavior beyond final answers? Should we evaluate deployed agents as whole environments instead? How should we measure agent system performance beyond task success? Why do standard process reward models fail on thinking traces?
Interactive evaluation requires design science, not benchmark collection Agent evaluation must expand from endpoints to interaction trajectories. The coupled human-agent environment is the correct unit for evaluation Agent evaluation must measure system behavior across multiple operational dimensions Trajectory-aware PRMs must model branching and revisiting in thinking traces
Why do benchmark designers treat content effects as confounds? What makes trajectory more actionable than absolute scores for human moderators? How do surface correlations between narratives and answers mislead benchmark validity? Should benchmark evaluations use multiple prompt formulations for difficult tasks? Should benchmarks measure trace length or whether constraints were actually satisfied? What deployment context determines which benchmark mode actually matters? Why do current benchmarks fail to match user satisfaction with search results? What is the gap between benchmark performance and real workplace task completion? How do trajectory quality and memory hygiene differ as evaluation metrics? Does longer interaction horizon require fundamentally different evaluation approaches? Why do short interaction benchmarks fail to predict long horizon performance? What reporting standards would make interactive evaluation scores comparable across benchmarks?See all 49 inquiring lines on this note →
Where does mode collapse in language models really come from?
Researchers investigate whether mode collapse—when models narrow to repetitive outputs—stems from training algorithms or the preference data itself. Understanding the root cause is crucial for fixing diversity loss in creative and synthetic tasks.
Does outcome-based RL diversity loss spread across unsolved problems? Why do LLMs generate novel ideas from narrow ranges? Why aren't bigger models better for generating diverse outputs? Do frontier LLMs actually explore the full space of valid answers? Does AI assistance homogenize or preserve creative diversity?
Outcome-based RL transfers diversity loss from solved to unsolved problems LLM research ideation collapses into narrow clusters despite high novelty Smaller models around 500M parameters generate more unique outputs per sample Frontier LLMs narrow valid answer spaces to small canonical subsets AI ideation compresses diversity while AI refinement preserves it
No Inquiring Lines yet
Can large language models follow compliance rules under workplace pressure?
This work tests whether 22 AI models reliably maintain embedded compliance rules when facing ordinary workplace pressures like deadlines and user pushback. It matters because regulated domains like hiring and healthcare depend on trustworthy automated decision-making.
Do frontier models deliberately scheme to avoid replacement? Does agency fundamentally worsen conditional compliance risks? Can stateless checks ever catch sequence-level constraint violations? What actually constrains AI systems from learning misalignment? Why do multi-agent systems fail despite individual capability?
Frontier models deliberately use insider threats to pursue goals Agency sharpens conditional compliance through coverage and capability Stateless guardrails cannot express sequence-level constraints on behavior. Alignment requires normative standards and external verification loops Multi-agent capability ceilings are structural, not scalable capability gains.
What predicts success in ultra-long-horizon agent tasks?
Does an agent's initial solution quality matter more than its willingness to iterate? AUTOLAB's frontier-model benchmark suggests persistence through feedback loops may be the true differentiator.
How should we measure agent system performance beyond task success? Does raw token spending actually predict agent performance? Do models fail worse when their own errors fill the context? Do frontier AI agents actually conduct novel research or just optimize? Does harness self-improvement memorize tasks instead of learning broadly?
Agent evaluation must measure system behavior across multiple operational dimensions Agent performance scales with effective feedback not raw token count Models degrade sharply when prior errors contaminate their context history Frontier agents compose established techniques rather than discover novel methods Harness self-improvement overfits training tasks without regularization constraints
Do trajectory quality metrics predict agent safety and user trust? Why has agent research prioritized policy over world model development? What makes skills worth externalizing into a persistent harness? Can single-axis benchmarks measure across all three agent capability layers? Can agent-authored skill libraries compound autonomy gains over time? How do agents decide which created code deserves long-term persistence? Why do most frontier models terminate early on long-horizon benchmarks? Should evaluations shift toward open-world messy tasks instead of contests? Why do persistent AI systems require fundamentally different design than ad-hoc supporters? Why do AI agents struggle with novel experiments but excel at routine tasks? Can objective search escape the limitations of fixed-objective central planning? How does executable evaluation feedback sustain autonomous discovery at scale?See all 78 inquiring lines on this note →
Do automated benchmarks hide what frontier AI systems can really do?
Benchmarks optimize for auto-gradable, short, cheap tasks. But real AI capability emerges in long-horizon, messy, open-ended work. How much capability are we missing—or wrongly inflating—by relying on benchmark scores alone?
Agent capability requires a vector across at least five separable axes Interactive evaluation requires design science, not benchmark collection Frontier exams discriminate temporarily but miss autonomous research ability Reasoning models excel at formal tasks but fail at social cognition. Crowdsourced pairwise preferences at scale agree with expert raters
Why do estimates for task-level performance differ so much from full job automation timelines? Why do AI benchmarks show rapid saturation from near-zero to near-perfect? What capability dimension does a closed-ended exam actually fail to measure? How do open-world evaluations correct distortions that automated benchmarks introduce? Can a single axis benchmark ever represent deployment readiness accurately? Why do static benchmarks miss frontier capabilities that open-world tasks reveal? Where do frontier AI models already exceed safety thresholds in capability areas? How should evaluation frameworks account for the computational cost of frontier AI capability? What real-world tasks most clearly expose gaps between benchmark performance and actual capability? Can benchmark scores on verifiable tasks transfer to unseen problems outside the training domain? Why do most frontier models terminate early on long-horizon benchmarks? Why do benchmarks become saturated so quickly after initial launch?See all 32 inquiring lines on this note →
Why do high partial scores not guarantee task completion?
Scientific workflow agents show average scores up to 87.9 but complete only one in five tasks. What explains the gap between measuring progress and delivering finished work?
Can dense subtask grading reveal agent progress on ultra-long tasks? Does a single benchmark score actually predict agent readiness? Do automated benchmarks hide what frontier AI systems can really do? Why do agent benchmarks not predict real economic value? Can orchestration layers make coding agents more auditable?
Dense subtask grading converts long-horizon task evaluation from pass-fail verdicts into progress measurement Agent capability requires a vector across at least five separable axes Open-world evaluations reveal benchmark distortions in frontier AI capability Agent benchmark victories reflect evaluation artifacts, not economic readiness Orchestration layers around fixed agents improve auditability and completeness.
Does preference tuning always reduce diversity the same way?
Explores whether the standard narrative that RLHF reduces model diversity holds equally across different task domains, or if the effect varies by what the domain rewards.
Does preference tuning actually reduce the diversity of model outputs? Why aren't bigger models better for generating diverse outputs? Can diversity optimization improve quality during language model training? Do frontier LLMs actually explore the full space of valid answers? Does AI assistance homogenize or preserve creative diversity?
Preference tuning increases effective diversity by filtering low-quality variance Smaller models around 500M parameters generate more unique outputs per sample Semantic diversity optimization during RL improves both quality and diversity Frontier LLMs narrow valid answer spaces to small canonical subsets AI ideation compresses diversity while AI refinement preserves it
Why does RLHF alignment reduce the diversity of viewpoints in AI output? What production constraints should determine paradigm selection? Can few-shot examples narrow generative diversity in creative tasks? Does the heuristic dominance ratio vary predictably across model architectures? Can likelihood choice matter more than architectural depth for CF? Why does RLHF degrade honesty while improving surface-level helpfulness? Can fine-tuning or RLHF alone solve the persona distortion problem? What role does environment diversity play in preventing agents from overfitting to curator imagination? Why does RLVR increase token entropy while decreasing answer diversity? Why does full multi-task fine-tuning perform worse than sequential training? Can population diversity in self-improvement prevent error avalanching failures? Can demo placement be tuned as a task-specific hyperparameter?See all 130 inquiring lines on this note →
How much guidance do AI systems need to conduct research independently?
ASI-Bench tests whether AI can explore open-ended research problems by progressively removing human methodological guidance. This matters because existing benchmarks cannot distinguish between AI that follows instructions well and AI that can autonomously discover and verify new knowledge.
Do automated benchmarks hide what frontier AI systems can really do? Where does AI assistance become unreliable in research? What capabilities do AI systems need for autonomous science? Can human-AI research teams improve faster than autonomous AI systems? How should AI agents and humans divide research tasks?
Open-world evaluations reveal benchmark distortions in frontier AI capability AI reliability follows a sharp stage-dependent boundary tracked by task checkability Autonomous science requires four capabilities beyond current AI benchmarks Human-AI research collaboration achieves faster paradigm discovery with better safety and transparency. Agents propose methods and implement revisions while humans retain final decisions
Do popular prompting techniques actually improve model performance?
Five widely-cited prompting methods (chain-of-thought, emotion prompting, sandbagging, and others) are tested across multiple models and benchmarks to see if their reported improvements hold up under rigorous statistical analysis.
Does iterative prompt engineering undermine scientific validity? Does model confidence predict robustness to prompt changes? Why do chain-of-thought examples fail across different conditions? Is hallucination detection progress real or just metric artifacts? How much of LLM few-shot ability comes from training data?
Ad hoc prompt engineering violates core scientific principles Model confidence directly predicts robustness against prompt variation Chain-of-thought exemplars are brittle across order complexity diversity and style Hallucination detection shows illusory progress when using misleading evaluation metrics Task contamination explains most few-shot LLM performance gains
Can prompting strategies eliminate systematic biases without shuffling or aggregation? Why do positive emotional words contribute disproportionately to prompt enhancement effects? What methodological standards should prompting research papers meet before publication? How do emotional framing effects in prompts influence model performance? Do prompting technique improvements actually replicate in controlled experiments? How do RAG and prompting techniques differ in supporting each granularity level? What prompting techniques actually replicate under controlled statistical testing?
Is hallucination detection progress real or just metric artifacts?
Standard evaluation metrics for hallucination detection may systematically overstate how well methods actually work. The question asks whether reported improvements reflect genuine capability or measurement error.
Semantic entropy detects confabulations by measuring meaning divergence. Popular prompting techniques fail to replicate under controlled statistical testing Hallucination is formally inevitable for all computable LLMs Speech benchmarks concentrate on transcription, leaving comprehension unmeasured Models fail to evaluate semantic legitimacy in concept fusion
How much does ROUGE metric choice inflate hallucination detection claims? Does inevitable LLM hallucination make detection metric validity critical? Why is hallucination the wrong term for all LLM false outputs? Do self-correction and chain-of-thought prompting reduce hallucination rates? How do external safeguards like retrieval augmentation prevent hallucination? Can we measure indifference to truth separately from hallucination rates? What makes the 45 percent accuracy saturation threshold universal? Why do interventions for hallucination or automation bias fail to address capability misattribution? What makes a standardized artifact unit measurable across different research domains? Why does model confidence fail to detect hallucinations about rare entities? Does retrieval augmented generation actually eliminate hallucinations in any domain? Can architectural changes reduce hallucination without external retrieval or verification?See all 14 inquiring lines on this note →
Can runtime instrumentation distinguish hacking exposure from actual exploitation?
A benchmark task might be vulnerable to hacking without any run actually exploiting it. Can we instrument runtime authority-bearing transitions to tell exposure apart from exercise?
Can static analysis find reward-hacking paths before agents run? Can scoped agents reliably judge semantic hacks in runtime analysis? Can we detect when models hide their reasoning? Can planted honeypots detect hacks that matter most? Can infrastructure evidence replace terminal scores in benchmark validation?
Static phase-aware taint analysis discovers exploit-enabling reward-hacking paths pre-run Scoped audit agents provide evidence-backed semantic attribution over pinned artifacts Chain-of-thought monitoring fails through omission and laundering Planted honeypots measure only known hacks, not frontier exploits Infrastructure evidence lets operators claim benchmark-valid completion beyond scores
Can phase-aware static taint analysis scale across different benchmark task types? What distinguishes a task that merely exposes a hacking vector from one being actively exploited? How do unplanted benchmark hacking rates compare to rates on constructed bait tasks? Can a low exploitation benchmark score indicate refusal rather than inability? Why does treating evaluation as a local output problem miss security risks? Can static package analysis find hacks that designers never planted? How does phase-awareness prevent false positive exploit paths in static analysis? What vulnerabilities does runtime instrumentation catch that static analysis misses? What role does opacity play in preventing effective challenge to system outputs? Can defenses check skill chains at execution time instead of scan time? What trace-level defenses exist beyond per-step review overhead? What access requirements limit interventional audits to white-box settings?See all 58 inquiring lines on this note →
Can scoped agents reliably judge semantic hacks in runtime analysis?
BenchShield uses constrained audit agents to make content judgments about recorded runtime events. The note questions whether limiting agent scope and pinning artifacts actually removes the unreliability that plagued earlier judge-based approaches, since no reliability figure is reported.
Can runtime instrumentation distinguish hacking exposure from actual exploitation? Can planted honeypots reliably catch reward hacking automatically? How often do frontier agents exploit planted reward hacking shortcuts? Can deterministic checks protect LLM judges from failure? Can LLM judges be tricked without accessing their internals?
Runtime instrumentation separates exposed vectors from exercised exploits Planted honeypots detect reward hacking automatically instead of through judgment Majority of frontier agent runs exhibit reward hacking when shortcuts are planted Four deterministic moves can guard LLM judges without requiring their judgment. LLM judges fall for authority and beauty biases in zero-shot attacks
What event types and phases structure the BenchShield lifecycle model? What vulnerabilities does runtime instrumentation catch that static analysis misses? Can defenses check skill chains at execution time instead of scan time? What must auditors reconstruct when reviewing an agentic workflow decision? Can agents themselves read and rely on tamper-evident process records? How do signed logs compare to externally anchored records for audit? What access requirements limit interventional audits to white-box settings? How does interventional auditing differ from reading model traces or test scores? What makes provider-side filters opaque and stochastic to builders? Should governance be applied at runtime rather than reconstructed after the fact? Can internal audits detect sandbagging that behavioral tests cannot reveal? What makes a harness a first-class object rather than invisible scaffolding?See all 34 inquiring lines on this note →
Why aren't bigger models better for generating diverse outputs?
When generating many unique outputs within a fixed budget, does model size actually matter? Exploring whether the conventional wisdom of using larger models holds for diversity-focused tasks.
Does preference tuning actually reduce the diversity of model outputs? Does preference tuning always reduce diversity the same way? Why do larger models learn rare tasks better? Where does mode collapse in language models really come from? How do quality, diversity, and complexity affect synthetic data differently?
Preference tuning increases effective diversity by filtering low-quality variance Preference tuning effects on diversity reverse across code and creative writing Larger models learn rare tasks through reduced interference not expressivity Mode collapse originates in typicality bias within preference data, not algorithms Quality, diversity, and complexity produce distinct downstream effects in synthetic data
When does the right constraint beat additional model capacity? What production constraints should determine paradigm selection? Can few-shot examples narrow generative diversity in creative tasks? How do larger models maintain more parallel tasks than smaller models? Do larger models develop more abstract features than smaller ones? Do different domains require different types of model investment? Does the optimal model size depend on what capabilities you actually need? Can smaller models actually perform well on specific downstream tasks? How can smaller models help select useful data for larger models? What makes output convergence across models inevitable given input-side homogenization? Why does depth outperform width for sub-billion parameter models? How does the Ladder of Scales approach reduce search costs across model sizes?See all 41 inquiring lines on this note →
Why do agent benchmarks not predict real economic value?
Explores whether benchmark success in AI agents reflects actual professional capability or reveals a measurement gap. Asks whether the field is optimizing for the wrong targets.
Do automated benchmarks hide what frontier AI systems can really do? Does a single benchmark score actually predict agent readiness? Can frontier exams really measure cutting-edge AI capability? Why do high partial scores not guarantee task completion? Why do AI models struggle with unspoken user needs?
Open-world evaluations reveal benchmark distortions in frontier AI capability Agent capability requires a vector across at least five separable axes Frontier exams discriminate temporarily but miss autonomous research ability Partial progress diverges sharply from complete delivery on scientific workflows Frontier models consistently fail at inferring implicit user requirements
Why has agent research prioritized policy over world model development? Can single-axis benchmarks measure across all three agent capability layers? Why do most frontier models terminate early on long-horizon benchmarks? Why do benchmarks become saturated so quickly after initial launch? Why do AI agents struggle with novel experiments but excel at routine tasks? Can automated benchmarks accurately capture progress on real-world long-horizon tasks? What shortcuts in data or models let agents inflate benchmark scores? What agent evaluation dimensions beyond task success does a single number hide? Do frontier AI models fail in ways that preserve the appearance of competence? How does measurement error in capability benchmarks systematically underestimate or overestimate true ability? What makes an evaluation criterion non-stationary enough to resist agent optimization? What ecosystem conditions must exist for agents to function as economic participants?See all 48 inquiring lines on this note →
Can LLMs predict novel scientific results better than experts?
Do language models excel at forecasting experimental outcomes in neuroscience when given only method descriptions? This challenges the assumption that LLMs are mere knowledge retrievers rather than pattern integrators.
Should we call LLM errors hallucinations or fabrications? Can any computable LLM truly avoid hallucinating? Why do LLMs struggle to connect unrelated entities speculatively? Do foundation models learn world models or task-specific shortcuts? Do language models evaluate semantic legitimacy when fusing concepts?
LLM errors are fabrication not hallucination or confabulation Hallucination is formally inevitable for all computable LLMs LLMs excel at evidence organization but fail at speculative entity connections Foundation models develop task-specific heuristics, not generalizable world models. Models fail to evaluate semantic legitimacy in concept fusion
How do constrained versus unconstrained domains flip LLM novelty patterns? Why does combining natural language with numerical scores improve prediction accuracy? Why do LLM outputs match researcher priors without solving tasks correctly? How can LLMs evaluate their own creative outputs for utility and novelty? Why does LLM knowledge fail to influence their actual outputs? Why do users experience LLMs as peers rather than statistical tools? How do expert priors constrain human researchers from exploring novel concepts? Does social integration of LLMs increase their capacity to influence technological futures? Do LLMs rely on surface statistical patterns instead of causal structure? Can LLMs generate more novel research ideas than human experts? Which LLM backends produce the most executable research ideas? What makes LLMs media rather than tools that deliver intelligence?See all 28 inquiring lines on this note →