Aryaman Reddi 🚀
Aryaman Reddi ආර්යමන් රෙඩ්ඩි

PhD Student

Technical University of Darmstadt

About Me

I am a PhD student in reinforcement learning at the LiteRL group at the Technical University of Darmstadt in partnership with the Intelligent Autonomous Systems lab and Hessian.AI, supervised by Professor Carlo D’Eramo 🎓

I am interested in developing sample-efficient techniques in deep multi-agent reinforcement learning using insights from applied mathematics and game theory 🕹️

I am a continuum hypothesis skeptic, a mereological universalist, and a Collatz conjecture supporter.

CV
Interests
  • Reinforcement Learning
  • Game Theory
  • Ethics and Philosophy
Education

  • PhD in Computer Science

    Technical University of Darmstadt, Germany

  • MEng & BA in Information and Computer Engineering

    University of Cambridge, United Kingdom

Experience

  1. Machine Learning Research Engineer

    Arm, Cambridge, United Kingdom
    • Developed an open-source tool (ML Inference Advisor) to optimise neural networks for inference on Arm GPUs using Python (PyTorch, Tensorflow, Numpy, Jupyter, Pandas), C++, Kubernetes, & Docker.
    • Achieved a 20% boost in GPU inference throughput by analyzing TensorFlow operator efficiency using deep learning clustering and pruning techniques
    • Enhanced IoT device performance by 12% by benchmarking over 30 silicon-on-chip devices using Python, Jenkins CI, SQL, & Kubernetes

Education

  1. PhD in Computer Science

    Technical University of Darmstadt, Germany

    My focus is on developing sample-efficient algorithms for exploration, coordination, and communication in multi-agent reinforcement learning using insights from applied mathematics and game theory.

    I believe bridging the gap between practical deep learning and theoretical models of stochastic optimisation is essential for scaling RL in real-world settings.

    I build algorithms which exhibit high performance in high-dimensional environments while providing mathematical insights using probability theory, linear algebra, calculus, & functional analysis.

  2. MEng & BA in Information and Computer Engineering

    University of Cambridge, United Kingdom
    • Grade: Distinction (GPA 4.0 Equivalent)
    • Received the David Thompson prize for academic achievement
    Read Thesis
Publications and Collaborations
K-Level Policy Gradients for Multi-Agent Reinforcement Learning
Multi Agent Reinforcement Learning

K-Level Policy Gradients for Multi-Agent Reinforcement Learning

Actor-critic algorithms for deep multi-agent reinforcement learning (MARL) typically employ a policy update that responds to the current strategies of other agents. While being straightforward, this approach does not account for the updates of other agents at the same update step, resulting in miscoordination. In this paper, we introduce the K-Level Policy Gradient (KPG), a method that recursively updates each agent against the updated policies of other agents, speeding up the discovery of effective coordinated policies. We theoretically prove that KPG with finite iterates achieves monotonic convergence to a local Nash equilibrium under certain conditions. We provide principled implementations of KPG by applying it to the deep MARL algorithms MAPPO, MADDPG, and FACMAC. Empirically, we demonstrate superior performance over existing deep MARL algorithms in StarCraft II and multi-agent MuJoCo.

Can “consciousness” be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis
Large Language Models

Can “consciousness” be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis

Integrated Information Theory (IIT) provides a quantitative framework for explaining consciousness phenomenon, positing that conscious systems comprise elements integrated through causal properties. We apply IIT 3.0 and 4.0 — the latest iterations of this framework — to sequences of Large Language Model (LLM) representations, analyzing data derived from existing Theory of Mind (ToM) test results. Our study systematically investigates whether the differences of ToM test performances, when presented in the LLM representations, can be revealed by IIT estimates, i.e., (IIT 3.0), (IIT 4.0), Conceptual Information (IIT 3.0), and -structure (IIT 4.0). Furthermore, we compare these metrics with the Span Representations independent of any estimate for consciousness. This additional effort aims to differentiate between potential “consciousness” phenomena and inherent separations within LLM representational space. We conduct comprehensive experiments examining variations across LLM transformer layers and linguistic spans from stimuli. Our results suggest that sequences of contemporary Transformer-based LLM representations lack statistically significant indicators of observed “consciousness” phenomena but exhibit intriguing patterns under spatio-permutational analyses.

Deep Learning Agents Trained For Avoidance Behave Like Hawks And Doves
Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning
Robot Learning

Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning

Reinforcement Learning (RL) has proven largely effective in obtaining stable locomotion gaits for legged robots. However, designing control algorithms which can robustly navigate unseen environments with obstacles remains an ongoing problem within quadruped locomotion. To tackle this, it is convenient to solve navigation tasks by means of a hierarchical approach with a low-level locomotion policy and a high-level navigation policy. Crucially, the high-level policy needs to be robust to dynamic obstacles along the path of the agent. In this work, we propose a novel way to endow navigation policies with robustness by a training process that models obstacles as adversarial agents, following the adversarial RL paradigm. Importantly, to improve the reliability of the training process, we bound the rationality of the adversarial agent resorting to quantal response equilibria, and place a curriculum over its rationality. We called this method Hierarchical policies via Quantal response Adversarial Reinforcement Learning (Hi-QARL). We demonstrate the robustness of our method by benchmarking it in unseen randomized mazes with multiple obstacles. To prove its applicability in real scenarios, our method is applied on a Unitree GO1 robot in simulation.

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
Adversarial Reinforcement Learning

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula

Robustness against adversarial attacks and distribution shifts is a long-standing goal of Reinforcement Learning (RL). To this end, Robust Adversarial Reinforcement Learning (RARL) trains a protagonist against destabilizing forces exercised by an adversary in a competitive zero-sum Markov game, whose optimal solution, i.e., rational strategy, corresponds to a Nash equilibrium. However, finding Nash equilibria requires facing complex saddle point optimization problems, which can be prohibitive to solve, especially for high-dimensional control. In this paper, we propose a novel approach for adversarial RL based on entropy regularization to ease the complexity of the saddle point optimization problem. We show that the solution of this entropy-regularized problem corresponds to a Quantal Response Equilibrium (QRE), a generalization of Nash equilibria that accounts for bounded rationality, i.e., agents sometimes play random actions instead of optimal ones. Crucially, the connection between the entropy-regularized objective and QRE enables free modulation of the rationality of the agents by simply tuning the temperature coefficient. We leverage this insight to propose our novel algorithm, Quantal Adversarial RL (QARL), which gradually increases the rationality of the adversary in a curriculum fashion until it is fully rational, easing the complexity of the optimization problem while retaining robustness. We provide extensive evidence of QARL outperforming RARL and recent baselines across several MuJoCo locomotion and navigation problems in overall performance and robustness.

Talks
Interview with Hessian.AI: How adversarial reinforcement learning trains robust AI 🟣
Blog
International Conference on Learning Representations (ICLR) 2024 📜
Symposium on Lifelong Explainable Robot Learning (SYMPLER) 2023 🦾
A Tier List of the Jurors from 12 Angry Men (1957) ⚖️
How to use Text-to-Speech in WSL to inform you when a job has finished 💻
Projects
Animating the Collatz Conjecture 💫
Machine Unlearning 🤖
More

Communities

Interesting links

Contact

✉️ aryaman{}reddi{}tu-darmstadt.de

📍 E327, S2|02 Robert-Piloty-Gebäude, Technical University of Darmstadt, Darmstadt 64289

LinkedIn