4th Workshop on Representing and Manipulating Deformable Objects @ ICRA2024

PACIFICO Yokohama area: North Room: G304

Deformable objects (DOs) are ubiquitous in human environments. From food, clothes, cables, to body tissues, DOs are present in personal households, industrial environments, agricultural settings, and hospital rooms, to mention a few. Despite the ease humans can reliably manipulate them, they still pose a major challenge for robotics. Specifically, we identify the following open questions within the research community: i) How can we feasibly represent the state of a deformable object? ii) How do we accurately model and simulate its complex and non-linear dynamics? iii) Which hardware tools and platforms are most suitable for grasping and manipulating them? In continuation of the workshops held at ICRA in 2021, 2022, and 2023, we aim to once again gather the community in pursuit of answers to these and further questions on DOs. Our goal is to facilitate connections among scientists across diverse subfields of robotics, such as perception, simulation, control, and mechanics, spanning various stages of their careers and operating within different professional environments, including academia, industry, and research centers. Additionally, we aim to focus on the tangible advancements made in the field since the first workshop edition in 2021. We believe that this analysis will help identify promising directions and ultimately pave the way for practical, real-world solutions.

The workshop was held in hybrid mode

Links:

Content

Topics

The workshop aims to explore different aspects that will allow robots to autonomously manipulate deformable objects with greater ability and generalization. Enabling such manipulation is crucial for a variety of domains and tasks, such as domestic, industrial, and surgical contexts, which involve various forms of deformable objects. However, the complexity of representing and modeling the dynamics of these objects results in the lack of a current unified solution that can be adapted to a wide range of objects. In the past few years, there has also been an increasing interest in applying foundation models to robotic manipulation, including the use of large pre-trained vision models, language models (LLMs), and vision language models (VLMs) for more sample-efficient learning and solving language-conditioned tasks. Additionally, recent advances in imitation learning, reinforcement learning, and 3D representation models have showcased the capability of robots learning to perform more complex, dexterous, and long-horizon tasks. The release of new simulators, datasets, and low-cost robotic hardware is lowering the barrier for reproducible research, benchmarking, and reuse of data. In this workshop, we will encourage discussions on how these recent advances can improve deformable object manipulation. The workshop will focus on, but is not limited to, the following topics for deformable object manipulation:

  • Representation and state estimation
  • Simulation and modeling
  • Transfer from simulation to reality
  • Learning to manipulate using data-driven methods such as reinforcement learning and learning from demonstrations
  • Perception: state tracking, parameter identification, property detection (e.g. landmarks for garments) and classification, etc.
  • Control, visual servoing and planning
  • Use of foundation models, such as large vision and language models, and associated large datasets
  • Specialized tools, e.g. grippers, and sensors

Schedule


Time Zone: GMT +09

Time Activity
08:45 - 09:00 Workshop opening [Video]
09:00 - 09:30 David Held [Remote] - Spatially-aware Robot Learning for Deformable Object Manipulation [Video]
09:30 - 10:15 Spotlight talks #1
  • Alessio Caporali, Piotr Kicki, Kevin Galassi, Riccardo Zanella, Krzysztof Walas and Gianluca Palli - Deformable Linear Objects Manipulation with Online Model Parameters Estimation [PDF] [Video]
  • Alessio Caporali, Kevin Galassi, Matteo Pantano, Gianluca Palli - Sparse to Dense: Robotic Perception of Deformable Objects via Foundation Models [PDF] [Video]
  • Mingrui Yu, Kangchen Lv, Changhao Wang, Yongpeng Jiang, Masayoshi Tomizuka and Xiang Li - Generalizable Whole-Body Global Manipulation of Deformable Linear Objects by Dual-Arm Robot in 3-D Constrained Environments [PDF] [Video]
  • Yuhong Deng, David Hsu - Generalizable Clothes Manipulation with Large Language Model [PDF] [Video]
  • Kejia Chen∗ , Zhenshan Bing∗ , Fan Wu∗ , Yansong Wu, Liding Zhang, Sami Haddadin, Alois Knoll - Real-time Contact State Estimation in Shape Control of Deformable Linear Objects under Small Environmental Constraints [PDF] [Video]
  • Amisha Bhaskar, Rui Liu, Guangyao Shi, Pratap Tokekar - LAVA: Long-horizon Visual Action based Food Acquisition [PDF] [Video]
  • Kaifeng Zhang* , Baoyu Li* , Kris Hauser, Yunzhu Li - AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation [PDF] [Video]
  • Mingrui Yu, Boyuan Liang, Xiang Zhang, Xinghao Zhu, Xiang Li, and Masayoshi Tomizuka - In-Hand Following of Deformable Linear Objects Using Dexterous Fingers with Tactile Sensing [PDF] [Video]
  • Haoran Lu*, Yitong Li*, Ruihai Wu*, Chuanruo Ning, Yan Shen, Hao Dong - UniGarment: A Unified Simulation and Benchmark for Garment Manipulation [PDF] [Video]
  • Simeon Adebola*, Tara Sadjadpour*, Karim El-Refai*, Will Panitch, Zehan Ma, Roy Lin, Tianshuang Qiu, Shreya Ganti, Charlotte Le, Jaimyn Drake, and Ken Goldberg - Automating Deformable Gasket Assembly [PDF] [Video]
  • Alberta Longhini, Michael C. Welle, Zackory Erickson, and Danica Kragic - AdaFold: Adapting Folding Trajectories of Cloths via Feedback-loop Manipulation [PDF] [Video]
10:15 - 10:45 Coffee break + Poster presentation
10:45 - 11:15 Chelsea Finn [Remote] - Learning Long-Horizon Bi-Manual Tasks involving Deformable Object Manipulation [Video]
11:15 – 11:45 Michael Yip - Deformable Manipulation for Autonomous Surgical Robots [Video]
11:45 – 12:15 Gonzalo Lopez-Nicolas - Multi-scale analysis for shape control of texture-less objects
12:15 - 14:00 Lunch + extra Poster time
14:00 – 14:30 Jeffrey Ichnowski - Deformable Manipulator for Deformable Manipulation [Video]
14:30 – 15:00 David Hsu - Differentiable Particles for General-Purpose Deformable Object Manipulation [Video]
15:00 – 15:45 Spotlight talks 2
  • Zeqing Zhang, Guanqi Chen, Wentao Chen, Ruixing Jia, Liangjun Zhang and Jia Pan - GmClass: Granular Material Classification through Force Feedback of Robotic Manipulation [PDF] [Video]
  • Shaoxiong Yao, Yifan Zhu, and Kris Hauser - Structured Bayesian Meta-Learning for Data-Efficient Visual-Tactile Model Estimation [PDF] [Video]
  • Raquel Marcos-Saavedra, Miguel Aranda, and Gonzalo López-Nicolás - Multirobot transport of deformable objects using deformation modes [PDF] [Video]
  • Martin Filliung, Juliette Drupt, Charly Peraud, Claire Dune, Nicolas Boizot, Andrew Comport, Cedric Anthierens, Vincent Hugel - An Augmented Catenary Model for Underwater Tethered Robots [PDF] [Video]
  • Luca Beber, Edoardo Lamon, Davide Nardi, Daniele Fontanelli, Matteo Saveriano, Luigi Palopoli - A Passive Variable Impedance Control Strategy with Viscoelastic Parameters Estimation of Soft Tissues for Safe Ultrasonography [PDF] [Video]
  • Rui Liu, Amisha Bhaskar, Pratap Tokekar - Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types [PDF] [Video]
  • Jingyi Xiang, Holly Dinkel, Harry Zhao, Naixiang Gao, Brian Coltin, Trey Smith, Timothy Bretl - TrackDLO: Tracking Deformable Linear Objects Under Occlusion with Motion Coherence [PDF] [Video]
  • Chikaha Tsuji, Enrique Coronado, Pablo Osorio and Gentiane Venture - Adaptive contact-rich manipulation through few-shot imitation learning with tactile feedback and pre-trained object representations [PDF] [Video]
  • Paul Maria Scheikl, Nicolas Schreiber, Christoph Haas, Niklas Freymuth, Gerhard Neumann, Rudolf Lioutikov, and Franziska Mathis-Ullrich - Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects [PDF] [Video]
  • Ruihai Wu*, Haoran Lu*, Yiyan Wang, Yubo Wang, Hao Dong - UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence [PDF] [Video]
  • Hiroto ARASAKI, Sena TAKAHASHI, and Akio NAMIKI - Realtime Paper Shape Estimation for Origami Robot System [PDF] [Video]
15:45 – 16:15 Coffee break + Poster presentation
16:15- 16:45 Dmitry Berenson - Two routes toward long-horizon deformable object manipulation [Video]
16:45 – 17:00 Best Abstract award & Closing remarks [Video]

Call for papers

We invite participants to submit extended abstracts 3+n pages, with n pages (no page-limit) for the bibliography, in the IEEE conference style.

Submissions will be reviewed by experts of their respective field. The accepted abstracts will be made available on the workshop website but will not appear in the official IEEE conference proceedings. Participants are encouraged to submit their recent work on the topics of interest mentioned above. Contributions are encouraged, but are not required, to be original.

The review process will be single-blind, meaning the submitted paper does not need to be anonymized.

Abstracts can be submitted through Microsoft CMT: https://cmt3.research.microsoft.com/WDOICRA2024.

IEEE RAS Computer & Robot Vision workshop award

We are happy to announce the WDO Best Abstract Award sponsored by the IEEE RAS Technical Committee Computer & Robot Vision. The selected contribution will receive a prize of 400$. Any extended abstract submitted to the workshop will be automatically considered for the award.

Important dates: (dd/mm/yyyy)

  • Submission Deadline: 09/04/2024 (23:59 PST) Firm deadline 05/04/2024 26/03/2024
  • Notification date: 29/04/2024 23/04/2024
  • Final submission: 07/05/2024 (23:59 PST)
  • Workshop date: 17/05/2024

Invited Speakers (alphabetical order)

Dmitry Berenson

Dmitry Berenson


Associate Professor
University of Michigan, USA
Personal website

Talk title: Two routes toward long-horizon deformable object manipulation

Bio: Dmitry Berenson received a B.S. in Electrical and Computer Engineering from Cornell University in 2005, where he started his robotics work in Hod Lipson's lab. He went on to graduate from the Ph.D. program at the Robotics Institute at Carnegie Mellon University (CMU) in 2011, where his advisors were Siddhartha Srinivasa and James Kuffner. While at CMU, Dmitry Berenson worked in the Personal Robotics Lab and completed interships at the Digital Human Research Center in Japan, Intel Labs in Pittsburgh, and LAAS-CNRS in France. In 2012 he completed a post-doc at UC Berkeley working with Ken Goldberg and Pieter Abbeel. Dmitry Berenson was an Assistant Professor at WPI 2012-2016. He started as faculty at the University of Michigan in 2016. His current research focuses on learning and motion planning for manipulation. Dmitry Berenson has received the IEEE RAS Early Career Award and the NSF CAREER award.


Chelsea Finn

Chelsea Finn


Assistant Professor
Stanford University, USA
Personal website

Talk title: Learning Long-Horizon Bi-Manual Tasks involving Deformable Object Manipulation

Bio: Chelsea Finn is an Assistant Professor in Computer Science and Electrical Engineering at Stanford University. Her research interests lie in the capability of robots and other agents to develop broadly intelligent behavior through learning and interaction. To this end, her work has pioneered end-to-end deep learning methods for vision-based robotic manipulation, meta-learning algorithms for few-shot learning, and approaches for scaling robot learning to broad datasets. Her research has been recognized by awards such as the Sloan Fellowship, the IEEE RAS Early Academic Career Award, and the ACM doctoral dissertation award, and has been covered by various media outlets including the New York Times, Wired, and Bloomberg. Prior to Stanford, she received her Bachelor's degree in Electrical Engineering and Computer Science at MIT and her PhD in Computer Science at UC Berkeley.


David Held

David Held


Associate Professor
Carnegie Mellon University (CMU), USA
Personal website

Talk title: Spatially-aware Robot Learning for Deformable Object Manipulation

Bio: David Held is an Associate Professor at Carnegie Mellon University in the Robotics Institute and is the director of the RPAD lab: Robots Perceiving And Doing. His research focuses on perceptual robot learning, i.e. developing new methods at the intersection of robot perception and planning for robots to learn to interact with novel, perceptually challenging, and deformable objects. David has applied these ideas to robot manipulation and autonomous driving. Prior to coming to CMU, David was a post-doctoral researcher at U.C. Berkeley, and he completed his Ph.D. in Computer Science at Stanford University. David also has a B.S. and M.S. in Mechanical Engineering at MIT. David is a recipient of the Google Faculty Research Award in 2017 and the NSF CAREER Award in 2021.


David Hsu

David Hsu


Provost's Chair Professor
National University of Singapore, Singapore
Personal website

Talk title: Differentiable Particles for General-Purpose Deformable Object Manipulation

Bio: David Hsu is a professor of computer science and the Director of Smart Systems Institute at the National University of Singapore (NUS). He is an IEEE Fellow. His research lies in the intersection of robotics and AI. In recent years, he has been working on robot planning and learning under uncertainty for human-centered robots. His work won multiple international awards, including, most recently, Test of Time Award at Robotics: Science & Systems (RSS) in 2021 and IJCAI-JAIR Best Paper Prize in 2022. He has chaired or co-chaired several international robotics conferences, including WAFR 2010, RSS 2015, ICRA 2016, and CoRL 2021. He served on the editorial boards of Journal of Artificial Intelligence Research and International Journal of Robotics Research. He is currently an Editor of IEEE Transactions on Robotics.


Jeff Ichnowski

Jeff Ichnowski


Assistant Professor
Carnegie Mellon University (CMU), USA
Personal website

Talk title: Deformable Manipulator for Deformable Manipulation

Bio: Jeff Ichnowski is an assistant professor at Carnegie Mellon University's Robotics Institute. He was a postdoc at UC Berkeley's Sky Computing/RISE lab, AUTOLAB, and BAIR. Before returning to academia, he was the principal architect at SuccessFactors, Inc., one of the world's largest cloud-based software-as-a-service companies. His research explores robot algorithms and systems for high-speed motion, task, and manipulation planning, using cloud-based high-performance computing, optimization, and deep learning.


Gonzalo Lopez

Gonzalo Lopez


Professor
Universidad de Zaragoza, Spain
Personal website

Talk title: Multi-scale analysis for shape control of texture-less objects

Bio: Gonzalo Lopez-Nicolas is currently a Professor with Universidad de Zaragoza and Aragon Institute for Engineering Research (I3A). His current research interests include shape control, visual control, multi-robot systems, and the application of computer vision to robotics.


Michael Yip

Michael Yip


Associate Professor
University of California, San Diego, USA
Personal website

Talk title: Deformable Manipulation for Autonomous Surgical Robots

Bio: Michael Yip, Ph.D., is an Associate Professor at the University of California San Diego and the Director of Advanced Robotics and Controls Lab (ARClab) at UCSD. His research group works at the intersection of medical robotics, machine learning, and computer vision, with applications towards robotic surgery, physical human-robot interaction, autonomous driving, and search and rescue. His research group have won numerous awards at robotics and AI venues. Dr. Yip was previously a Research Associate with Disney Research, a Visiting Professor at Stanford University, and a Visiting Professor with Amazon Robotics.


Organizers

  • Michael C. Welle, KTH Royal Institute of Technology, Sweden
  • Martina Lippi, Roma Tre University, Italy
  • Fangyi Zhang, Queensland University of Technology (QUT), Australia
  • Lawrence Yunliang Chen, University of California, Berkeley, USA

Co-Organizers

  • Alberta Longhini, KTH Royal Institute of Technology, Sweden
  • Danica Kragic, KTH Royal Institute of Technology, Sweden
  • Daniel Seita, University of Southern California, USA
  • David Held, Carnegie Mellon University, USA
  • Peter Corke, Queensland University of Technology (QUT), Australia

Contact

If you have any questions please contact Michael Welle at the email: mwelle AT kth DOT se