Continuous Object State Recognition for Cooking Robots

Using Pre-Trained Vision-Language Models and Black-box Optimization

  • Kento Kawaharazuka
  • Naoaki Kanazawa
  • Yoshiki Obinata
  • Kei Okada
  • Masayuki Inaba
  • JSK Robotics Laboratory, The University of Tokyo, Japan

The state recognition of the environment and objects by robots is generally based on the judgement of the current state as a classification problem. On the other hand, state changes of food in cooking happen continuously and need to be captured not only at a certain time point but also continuously over time. In addition, the state changes of food are complex and cannot be easily described by manual programming. Therefore, we propose a method to recognize the continuous state changes of food for cooking robots through the spoken language using pre-trained large-scale vision-language models. By using models that can compute the similarity between images and texts continuously over time, we can capture the state changes of food while cooking. We also show that by adjusting the weighting of each text prompt based on fitting the similarity changes to a sigmoid function and then performing black-box optimization, more accurate and robust continuous state recognition can be achieved. We demonstrate the effectiveness and limitations of this method by performing the recognition of water boiling, butter melting, egg cooking, and onion stir-frying.

Robotic Continuous State Recognition Using Pre-Trained Vision-Language Models and Black-box Optimization

We propose a continuous object state recognition method for cooking robots by using pre-trained large-scale vision-language models and black-box optimization.

Overview of the Proposed Method

We obtain time series of images D, prepare a variety of text prompts, calculate the continuous similarity changes with pre-trained vision-language models and text weight, fit the similarity changes to a sigmoid function, compute the evaluation value, and iteratively optimize the text weight with black-box optimization.

Experimental Setup

The text prompts and representative images for water boiling, butter melting, egg cooking, and onion stir-frying experiments

Experiments

Bibtex

  @article{kawaharazuka2024cooking,
    author={Kento Kawaharazuka and Naoaki Kanazawa and Yoshiki Obinata and Kei Okada and Masayuki Inaba},
    title={{Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization}},
    journal={IEEE Robotics and Automation Letters (RA-L)},
    year=2024,
  }

Contact

If you have any questions, please feel free to contact Kento Kawaharazuka.