psc's website

The Dormant Neuron Phenomenon in Deep Reinforcement Learning

We identify the dormant neuron phenomenon in deep reinforcement learning, where an agent’s network suffers from an increasing number of inactive neurons, thereby affecting network expressivity.

Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro*, Utku Evci*

This blogpost is a summary of our ICML 2023 paper. The code is available here. Many more results and analyses are available in the paper, so I encouraged you to check it out if interested!

The following figure gives a nice summary of the overall findings of our work (we are reporting the Interquantile Mean (IQM) as introduced in our Statistical Precipice NeurIPS'21 paper):

June 19, 2023 Read

Albums

I’ve been in bands since I was 12. In a parallel universe I’m a full-time musician :D. This page collects the albums I’ve released so far, in reverse chronological order. Enjoy!

the 45s - #1 (2024)

With my good friend Jean-Simon Diallo we started a new project where we release 45s (i.e. singles) consisting of two songs, one written by him and one by me. He does the singing, I do the production and play the instruments. Enjoy!

March 20, 2023 Read

PongDay

I learned on the radio that last November 29th marked the 50th anniversary of the classic arcade game Pong. This game is particularly meaningful for those of us that do RL research, as it is one of the games that is part of the Arcade Learning Environment, one of the most popular benchmarks. Pong is probably the easiest game of the whole suite, so we often use it as a test to make sure our agents are learning. Learning curves below are for agents trained with the Dopamine framework.

December 5, 2022 Read

Hola, I'm psc

Pablo Samuel Castro

Señor Swesearcher at Google

Recent Posts

The Dormant Neuron Phenomenon in Deep Reinforcement Learning

Albums

the 45s - #1 (2024)

PongDay

Selected Publications

On the consistency of hyper-parameter selection in value-based deep reinforcement learning

Mixture of Experts in a Mixture of RL settings

In value-based deep reinforcement learning, a pruned network is a good network

Mixtures of experts unlock parameter scaling for deep RL

Stop Regressing - Training Value Functions via Classification for Scalable Deep RL

Small batch deep reinforcement learning

Minigrid & miniworld - Modular & customizable reinforcement learning environments for goal-oriented tasks

A Kernel Perspective on Behavioural Metrics for Markov Decision Processes

Bigger, Better, Faster, Human-level Atari with human-level efficiency

The dormant neuron phenomenon in deep reinforcement learning

Proto-Value Networks, Scaling Representation Learning with Auxiliary Tasks

Reincarnating Reinforcement Learning, Reusing Prior Computation to Accelerate Progress

The State of Sparse Training in Deep Reinforcement Learning

A general class of surrogate functions for stable and efficient reinforcement learning

MICo, Learning improved representations via sampling-based state similarity for Markov decision processes

The Difficulty of Passive Learning in Deep Reinforcement Learning

Deep Reinforcement Learning at the Edge of the Statistical Precipice

Losses, Dissonances, and Distortions

Revisiting Rainbow, Promoting more insightful and inclusive deep reinforcement learning research

Metrics and continuity in reinforcement learning

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Autonomous navigation of stratospheric balloons using reinforcement learning

Estimating Policy Functions in Payment Systems using Reinforcement Learning

Dopamine, A Research Framework for Deep Reinforcement Learning

GANterpretations

Rigging the lottery, Making all tickets winners

Scalable methods for computing state similarity in deterministic MDPs

A Geometric Perspective on Optimal Representations for Reinforcement Learning

A comparative analysis of expected and distributional reinforcement learning

ML-Jam, Performing Structured Improvisations with Pre-trained Models

Shaping the Narrative Arc, Information-Theoretic Collaborative Dialogue

An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

Distributional reinforcement learning with linear function approximation

Inverse reinforcement learning with multiple ranked experts