NIPS Demo 2008

Download our full poster!

See "attachments" at the bottom of this page or click here (warning, it's 7 MB).

Extra MultiMedia

Photos

CritterBot

CAD Drawing of the critter bot is shown below. Pictures and videos of the physical robot will be available by Sept 26, 2008.

Videos (NEW! Updated Friday, Sept 25/08)

CritterBot

Alpha Tech Demo Movie (From Sept 25/2008) in MP4 (3.4 MB)

Alpha Tech Demo Movie (same as above) in MOV (2.9 MB)

RL-Viz

Proposal

Title: RL-Glue : From Grid Worlds to Sensor Rich Robots

URL: http://glue.rl-community.org/nips-demo-2008

Co-Creators:

Adam White

Richard S. Sutton

{awhite, sutton} @ cs.ualberta.ca

Department of Computing Science

University of Alberta

Edmonton, Alberta, Canada

Phone: 780-492-4766

Fax: 780-492-1071

Creator Credits

Marc Bellemare

Michael Bowling

Andrew Butcher

Leah Hackman

Anna Koop

Mark Lee

James Neufeld

Matthew Radkie

Mike Sokolsky

Richard S. Sutton

Brian Tanner

Adam White

All from the Department of Computing Science, University of Alberta

Equipment we will bring:

2 x Laptop

2 x Projectors or Monitors

Robot!

Robot Playpen

We need:

Table X

Poster X

Power X

Special needs:

Something for the projectors to project on to if we used projectors.

Technology

We propose to demonstrate a suite of software and robotics projects for reinforcement learning (RL) that have been in development for several years in Dr. Richard Sutton’s group at the University of Alberta. Specifically, the three projects that we intend to showcase are: RL-Glue interface, the CritterBot robotic platform, and RL-Viz experimentation platform.

The demonstration will illustrate how these projects allow researchers to develop learning agents that can be evaluated in a graphical simulation (RL-Viz) and on a mobile robot (CritterBot).

RL-Glue

RL-Glue is a language- and platform-independent protocol for evaluating reinforcement learning agents with environment programs. RL-Glue separates the agent- and environment-development process so that each can be written in different languages and even executed over the Internet from different computers.

RL-Glue have had a significant influence on the way empirical comparisons are done in reinforcement learning. RL-Glue has been used to evaluate agents in four international competitions at high profile machine learning conferences. The most recent competition, held in conjunction with ICML 08, attracted over 150 teams. The final test phase of the competition included over 20 teams comprised of more than 40 participants.

RL-Glue has been used by several university instructors, in several countries, to teach reinforcement learning. Several researchers have used RL-Glue to benchmark their agents in papers published in top international conferences, including NIPS.

CritterBot

The CritterBot is an ongoing project at the University of Alberta whose goal is to add a further robotics effort to challenge, direct, and inspire the research on grounded artificial intelligence. This robot is small and mobile and outfitted with an unusually rich set of sensors, including sensors for touch, acceleration, motion, sound, vision, and several kinds of proximity.

The initial objective is for the robot to form an extended multi-level model of the relationships among its sensors and between its sensors and its actuators. We have proposed that higher-level knowledge can be grounded in raw data of sensations and actions; this robotic platform will challenge and inspire us to see if it can really be done. We also plan to use this platform as a test case for rapid learning and for the use of reinforcement learning by non-experts. We would like a person whose has no training to be able to teach the system new ways of behaving in an intuitive manner much as one might train a particularly cooperative dog.

Learning agents can interact with the CritterBot through RL-Glue just like with any other RL-Glue environment.

RL-Viz

RL-Viz provides the reinforcement learning community for the first time ever with a flexible, general, standardized, cross language and cross platform protocol/framework for managing and visualizing the interaction between agents and environments in reinforcement learning experiments. The RL-Viz project includes several state-of-the-art tasks used in learning research including, Tetris, a remote-controlled helicopter simulator provided by Andrew Ng’s team at Stanford, keep-away soccer and a real-time strategy engine. RL-Viz is a protocol and library layered on top of RL-Glue.

RL-Viz supports advanced features such as visualization of environments and agents and run-time loading of agents and environments. The software for most recent RL competition (mentioned above) was based on RL-Viz.

We will present the latest developments in the RL-Glue project and demonstrate how RL-Glue provides a novel, unified architecture for developing reinforcement learning algorithms for simulation and physical experiments. This framework makes it easier to compare the performance of agents in a variety of simulated and physical tasks.

Experience

This demonstration will give the NIPS attendees the opportunity to learn about the features of RL-Glue, use RL-Viz, and observe the CritterBot learning in real-time using modern reinforcement learning methods.

In the RL-Viz demonstration, attendees will be able to (1) make custom experiments using arbitrary combinations of agent and environment programs, (2) change various parameters at run-time and visualize the effect on the agent's behavior and value function and (2) manually advance the experiment one step at a time. The RL-Viz demonstration will allow users to directly experience how our system can be useful for developing, debugging, benchmarking, teaching, and demonstrating reinforcement-learning methods.

Participants will also be able to watch an RL-Glue agent controlling the CritterBot, observe graphic visualizations of the agent’s internal state, and send simple commands to the robot (physical and simulated) through the RL-Viz interface.

The RL-Viz and CritterBot demonstrations will allow attendees to directly experience how our tools can make it easy to interact with agents, robotic problems, and simulated environments.

We could also show some of the winning entries from the 2008 RL Competition, including a new Tetris agent that is an order of magnitude better than the previous best known learning agent.

Readiness:

RL-Glue and RL-Viz have been vetted through a year’s extensive use in the RL Competition. They are ready.

The robot’s hardware and software has been under development for over a year by a full-time electrical engineer and several computing science graduate students and is near completion. The robot recently passed it’s first set of demonstrations/tests.

The robot simulator and bridge that connects the robot to RL-Glue is well underway, and will be in testing by the end of September.

We have no doubt that all parts of our proposed demo will be ready, after considerable vetting, before NIPS.

We will be posting photos, videos, and screenshots at the demo page here:

http://glue.rl-community.org/nips-demo-2008