From Vision to Execution: Enabling Knowledge Representation and Reasoning in Hybrid Intelligent Robots Playing Mobile Games

Automating acts on touch surfaces opens a range of possibilities for researching and experimenting with hybrid AI approaches.

In this paper, we propose a delta robot capable of playing match-3 games and ball-sorting puzzles by acting on mobile phones. The robot recognizes objects of different colors and shapes through a vision module, is capable of making strategic decisions based on declarative models of the game's rules and of the game playing strategy, and features an effector that executes moves on physical devices.

Our solution integrates multiple AI methods, including vision processing and answer set programming. Helpful and reusable infrastructure is provided: the vision task is facilitated, while robot motion control is inherently simplified by the usage of a delta robot layout.

We illustrate the components of our robotic application and how they were integrated. Then, we briefly showcase how recognition and general knowledge can be modeled and implemented, by overviewing the implementation of representative games.

We argue that our application provides potential for KR and robotics to be combined in creative ways, and offers itself as a general controlled environment where to experiment with forms of hybrid reasoning, while relieving from implementation details.