Giving Robots Narratives to aid in their decision making

With regards to the complex task of selecting optimal behavior for an autonomous mobile agent in a foreign environment, such tasks are easily accomplished by animals. They literally have evolved through millions of years not only the necessary actuators to move about in their environment but they also evolved to have the proper sensors to perceive and map their environment in the first place for them to be able to then make the right decisions that will best suite their interest - survive and mate. When it comes to robots, such tasks are still very hard, mainly because the method we're using involve training the agent in environments that simulate or atleast is close to the environment in the real world. Reinforcement learning has proven to be very effective and the agent will probably do very well in situations during testing that is close to the ones in training. However, as we all know, the "real world" can be quite unpredictable. Many factors such as wind, heat, other animals, and various possible activities by organisms called humans in the area can alter the environment and change it from familiar to straight out alien very quickly. In line with this imagery of a foreign terrain, let us imagine the following scenario. A human randomly wakes up in an alien land, the laws of physics in this territory are different from the one he's used to but merely by observation (primarily using his eyes), he is able to come up with a connections regarding the events that unfold before him. For example, dropping a ball in this planet causes it to rise up instead of fall down. The next time, he drops a ball, he will then expect it to go up. We can break his thinking pattern into the following: 1.) See ball about to drop 2.) Understand that he is in Alien Planet X 3.) Predict it will go up

Now, since the sensors used by this human is something that can also be used and mimicked by robots, a.k.a a camera, we can try to replicate this behavior.

Firstly, an AI agent must capture the scene which it does so using a camera. Next, a computer vision algorithm detects the ball in the image and a hand. Using some language model, the information is conveyed to it that it currently located at Planet X and that the laws of gravity here are the exact opposite. Given that, it is able to come up with a text describing its prediction based on the situation similar to an output from a ChatGPT prompt. This text can then serve as a prompt for an image generation model and the product here will basically be its own simulation of its prediction of the event that will unfold (the ball dropping) in its imagination.

With these steps, a very simple human task is mimicked.