- Playing Super Mario 64 with GPT-4V
- For this reason: https://twitter.com/realGeorgeHotz/status/1758451210892329127
-
capture game output screenshot
-
Ask following questions:
- What objects are in the vicinity of Mario
- What are the hazards located relative to Mario?
-
Is there a star (objective) visible?
-
What paths or routes are available to where the star could be?
-
What is the safest route from Mario's current position to the desired location?
-
Are there any interactive elements (e.g., blocks, switches) in the scene? What do they do?
-
Based on the current game state, what would be the best next move? THink step by step and carefully answers to all questions last 10 moves
- Update to use GPT-4o-mini for lower latency and cost
- Run LLaVa and ollama locally to reduce action latency
- Create long-term plans (not just next move)
- If stuck take a 360 degree look, do a scout loop