Request for Proposals
Iliad is interested in funding new incubated research organizations. To that end, we are running a request for proposals for specific research directions that we consider to be promising.
-
We are very excited about projects that develop and experimentally test theory in ReLU networks. We see this as the sweet spot between relevance to real networks and theoretical tractability. Additionally, there are well-developed fields of mathematics that can be brought to bear on this research (algebraic geometry, differential geometry, tropical geometry, invariant theory, polyhedral geometry, combinatorics, algebraic topology). If you are a mathematician working in one of these areas please regard this as your Call to Action, and reach out to us regardless of your prior experience with AI. We want to find ways for you to be impactful and to fund your work.
ReLU Networks: Overview
One of the central challenges in AI safety is the absence of a sufficiently general mathematical account of neural network generalization. Current formal verification methods can prove limited properties, but can’t guarantee safe behavior in novel real-world settings.
We want to support research on whether ReLU networks possess an intrinsic simplicity bias analogous to Solomonoff induction. Solving this problem might provide:
A principled account of the inductive biases underlying deep neural network generalization [2]
A mechanistic understanding of how internal objectives and representations emerge during training
A connection between deep learning systems and formal frameworks from statistical learning theory and algorithmic information theory
The development of novel mathematically grounded approaches to reasoning about alignment-relevant behavior in advanced systems
Existing Research on ReLU networks
Our existing research on ReLU networks uses a toric encoding map that reveals tractable geometries, simplifying the analysis of ReLU networks. Concretely, this has been used to:
Characterize singularities
Identify two separate mechanisms: data-specific vs. architecture-specific
Relate algebraic conditions to geometric conditions: toric geometry, clean intersections
Describe the fiber (i.e. the “degeneracies”: all parameters lead to the same model behavior)
Pre-image of restricted affine sections of a toric variety
Reveals neural network symmetries, degeneracies, and redundancies
Their geometry & topology (dimension, volume, connectedness, intersections, general structure) controls the learning coefficient and training dynamics, thus implicit biases [2]
Characterize the RLCT / learning coefficient: a fundamental complexity measure stemming from singular learning theory (we are approaching completion)
Describing the loss landscape and the fiber is the natural precursor and bridge to an algorithmic information theory / algorithmic statistics description of ReLU networks and to formulating Solomonoff-like simplicity biases.
More on simplicity biases
Can the geometric degeneracies of ReLU networks induce an algorithmic-statistics-like simplicity bias, where functions with many padded representations receive larger mass? Concrete steps in this direction:
What are (approximate) padding functions in ReLU networks?
How do these relate to (near) fiber volumes? (e.g. [1] shows that minimal/irreducible networks may still be non-identifiable i.e. have non-trivial fibers, although these cases are rare)
How does this relate to singular learning theory generally, and padding to the learning coefficient specifically? [3]
Does SGD prefer functions with more padding? Does this explain grokking?
From a developmental perspective: if capabilities are acquired in discrete phases, can algorithmic statistics help formulate what these are and relate them to landscape geometry? (e.g. "subalgorithms correspond to loss landscape basins")
The loss landscape geometry problem
Beyond the simplicity bias question, the near-term goal is to produce a complete account of the loss landscape geometry — including the fiber, singularities, critical loci, and basins.
Many safety questions are ultimately questions about the geometry of training. If we have trained a model to not be deceptive, where has it moved in parameter space? Are two models that behave similarly also similar internally? Can a safety intervention remove a behavior robustly, or does it merely move the model to a nearby region where the behavior is hidden but recoverable? When models undergo sudden qualitative changes during training — grokking-like phase transitions or the emergence of new behavioral tendencies — can these be understood as movement between different branches of a solution space?
The narrow goal is to bring low-level geometry results to a self-contained place where experts in adjacent domains (e.g. AIT) can build on them directly.
There is work for:
Geometers to fully characterize the landscape
AI safety experts to translate how this can be used
e.g. AIT experts to formulate Solomonoff-like simplicity bias
[1] https://arxiv.org/abs/2605.03601
-
Projects that make AI Safety field as a whole more rigorous — which provide a strong signal to funders and new researchers about which agendas and results are trustworthy and worth pursuing. Examples would be:
Red teaming seminal results
Adversarial collaborations
Critical literature reviews
Replication efforts (not just technical)
Mechanisms for soliciting critical reviews of seminal results, via e.g. bug bounties
We are particularly interested in approaches that meet the following additional specifications:
Led by senior researchers who have experience in identifying and rectifying epistemic problems in research.
Projects which can justify their methodology based on historical results from previous efforts (which need not necessarily be scientific). For instance, “the field of psychology combated its replication crisis through ____, we will adapt these methods by ____.”
