My research focuses on learnable perception that enables action. Below, I've clustered my publications (with code, videos, etc.) by research themes: unsupervised computer vision, algorithmic priors, state representation learning, and our robotic system for the Amazon Picking Challenge. For most recent papers, please check google scholar or follow me on twitter. If you are looking for my CV, that's at the bottom of this site. And here is my email.
Selected Publications by Topic
Unsupervised Computer Vision
Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What Matters in Unsupervised Optical Flow. ECCV, 2020. Code, PDF, BibTeX, ECCV Oral, featured in Computer Vision News' Best of ECCV,
TL;DR: A thorough study into the core components of unsupervised optical flow produces in a simple method that sets a new state of the art
Peter Karkus, Anelia Angelova, Vincent Vanhoucke, and Rico Jonschkowski. Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localization. arXiv:2005.09530, 2020. PDF, BibTeX
TL;DR: combining spatial structure and end-to-end learning for mapping and localization
TL;DR: the particle filter structure encoded in a recurrent deep network
Rico Jonschkowski and Oliver Brock. Towards Combining Robotic Algorithms and Machine Learning: End-To-End Learnable Histogram Filters. Workshop on Machine Learning Methods for High-Level Cognitive Capabilities in Robotics at IROS, 2016. PDF, BibTeX, (preliminary version, see extended version above)
TL;DR: a way to combine the algorithmic structure of Bayes filters with the end-to-end learnability of neural networks
State Representation Learning
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations. New Frontiers for Deep Learning in Robotics Workshop at RSS, 2017. PDF, BibTeX, Paper Award winner
TL;DR: unsupervised learning of where things are and how they are moving
Sebastian Höfer, Antonin Raffin, Rico Jonschkowski, Oliver Brock, and Freek Stulp. Unsupervised Learning of State Representations for Multiple Tasks. Workshop on Deep Learning for Action and Interaction at NeurIPS, 2016. PDF, BibTeX
TL;DR: a learning method for automatic detection of multiple reinforcement tasks and extraction of state representations from raw observations
Rico Jonschkowski and Oliver Brock. Learning State Representations with Robotic Priors. Autonomous Robots. Springer US 39(3):407-428, 2015. Code, Video of Robot Experiment, PDF, BibTeX, (extension of the two papers below)
Rico Jonschkowski and Oliver Brock. State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction. Proceedings of Robotics: Science and Systems, 2014. Talk, PDF, BibTeX, (preliminary version, see extended version above)
Rico Jonschkowski and Oliver Brock. Learning Task-Specific State Representations by Maximizing Slowness and Predictability. Proceedings of the 6th International Workshop on Evolutionary and Reinforcement Learning for Autonomous Robot Systems (ERLARS), 2013. PDF, BibTeX, (preliminary version, see extended version above)
TL;DR: state representations can be learned from raw sensory input by making these representations consistent with prior knowledge about interactions governed by physics = robotic priors
Amazon Picking Challenge
Rico Jonschkowski, Clemens Eppner*, Sebastian Höfer*, Roberto Martín-Martín*, and Oliver Brock. Probabilistic Multi-Class Segmentation for the Amazon Picking Challenge. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016. Code, PDF, BibTeX, IROS Best Paper Award Finalist
TL;DR: in-depth analysis of the object-segmentation method used in our winning entry to the 2015 Amazon picking challenge with lessons that should be useful towards more generic robotic perception
Clemens Eppner*, Sebastian Höfer*, Rico Jonschkowski*, Roberto Martín-Martín*, Arne Sieverling*, Vincent Wall* and Oliver Brock. Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems. Proceedings of Robotics: Science and Systems, 2016. Talk, PDF, BibTeX, RSS Best Systems Paper Award winner
TL;DR: four aspects that improve our exploration and understanding of how to build robotic systems: modularity vs. integration, computation vs. embodiment, planning vs. feedback, generality vs. assumptions
Rico Jonschkowski. Learning Robotic Perception Through Prior Knowledge. Dissertation, Technische Universität Berlin, 2018. PDF
This dissertation summarizes my pre-2018 work listed above. My committee consisting of Manfred Opper, George Konidaris, Marc Toussaint, and Oliver Brock graded it with summa cum laude and the Department for Electrical Engineering and Computer Science at TU Berlin awarded it the Best Dissertation Prize of the Dr. Wilhelmy-Stiftung.
- 05/2018 - present
Research scientist at Robotics at Google, Mountain View, California.
- 10/2012 - 05/2018
Research associate and PhD student at RBO, TU Berlin (Advisor: Oliver Brock).
Taught courses: Fundamentals of Robotics, Robotics, Advanced Robotics, Robotics Seminar, Robotics Project, Algorithms and Datastructures.
05/2018 Dr. rer. nat. (German PhD equivalent, summa cum laude)
Thesis: Learning Robotic Perception Through Prior Knowledge.
01/2017 - 04/2017 PhD Intern at DeepMind, London.
- 10/2007 - 09/2012
Study of computer science at FU Berlin.
09/2012 Master of Science (grade: 1.0, major: robotics/AI, minor: psychology).
Thesis: New Approaches to Temporal Abstraction in Hierarchical Reinforcement Learning.
01/2012 - 09/2012 Research assistant at MLR, FU Berlin (Advisor: Marc Toussaint).
2011 Study abroad at UNSW, Sydney.
10/2009 - 09/2011 Teaching assistant at FU Berlin.
Taught courses: Functional Programming, Object-Oriented Programming, Computer Science and Society, Software Engineering
01/2011 Bachelor of Science (grade: 1.4, major: computer science, minor: philosophy).
Thesis: Control of autonomous humanoid soccer robots with XABSL.
2008 - 2011 Member of RoboCup team FUmanoids (Advisor: Raul Rojas).
Awards, Prizes, Scholarships
2018 Prize from the Dr. Wilhelmy-Stiftung for the best dissertation in electrical engineering and computer science at TU Berlin.
2017 Paper Award winner at New Frontiers for Deep Learning in Robotics Workshop
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations. New Frontiers for Deep Learning in Robotics Workshop at RSS, 2017.
2016 Best Paper Award finalist at IROS
Rico Jonschkowski, Clemens Eppner, Sebastian Höfer, Roberto Martín-Martín, and Oliver Brock. Probabilistic Multi-Class Segmentation for the Amazon Picking Challenge. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016.
2016 Best Systems Paper Award winner at RSS
Clemens Eppner, Sebastian Höfer, Rico Jonschkowski, Roberto Martín-Martín, Arne Sieverling, Vincent Wall, and Oliver Brock. Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems. Robotics: Science and Systems (RSS), 2016.
2015 Winner of the Amazon Picking Challenge at ICRA15.
2015 AAAI-15 Robotics Fellowship
2011 PROMOS scholarship from FU Berlin
2011 4th place RoboCup Worldcup, 2nd place RoboCup German Open, 1st place Technical Challenge @ RoboCup German Open, 1st place RoboCup Iran Open
2010 2nd place RoboCup Worldcup, 1st place Technical Challenge @ RoboCup Worldcup, 1st place RoboCup Iran Open
2008 2nd place RoboCup German Open
2007 1st place RoboCup Junior German Open