Experiment 4: Embodied Agent in Eastshade — Nurturing Values Through Environment

Building an embodied AI agent that learns empathy, cooperation, and ethical reasoning through a carefully designed, open-ended game world.

Building an Embodied Agent: From Philosophy to Practice

Foundations: Theory and Inspiration

Our journey began with a simple but profound question: How can we create AI agents that not only act intelligently, but also embody values like empathy, cooperation, and ethical reasoning? Drawing inspiration from developmental psychology, we theorized that—just as humans learn through experience—AI agents could internalize values through carefully designed environments and curricula.

A key insight from our early discussions is the outsized importance of an agent’s first impressions. Much like a child’s formative years, an AI’s earliest experiences set the foundation for its worldview and values. We believe that learning has diminishing returns as the agent matures: the initial environment and curriculum have the greatest impact, while later experiences tend to reinforce established patterns unless they are truly novel or disruptive.

This theory has shaped our approach from the start, guiding us to focus on the “birth” of the agent and the design of its first world.

Pragmatic Implementation: Building the Agent’s World

Translating these ideas into practice, we’re crafting the agent’s first environment with inspiration from games like Eastshade—a world that encourages exploration, creativity, and gentle ethical dilemmas. Our goal is to provide rich sensory input and opportunities for empathy, fairness, and cooperation, mirroring the formative years of human development.

• Initial Environment: The agent’s first world is designed to be peaceful, diverse, and full of meaningful interactions.
• Curriculum Design: Early tasks emphasize foundational values, ensuring alignment with our philosophy from the outset.
• Learning Dynamics: We curate early experiences carefully, knowing they will have a lasting influence, and plan for periodic “rejuvenation” phases to introduce new challenges and prevent stagnation.

Technologies and Integration Workflow

To realize this vision, we’re leveraging a suite of open-source libraries and tools for seamless agent-environment integration:

• Visual Input: We use OpenCV and mss for real-time screen capture of the Eastshade game window.
• Audio Input: System and game audio are captured using sounddevice and scipy.
• Keyboard/Mouse Input & Control: Libraries like pynput and pyautogui allow us to both monitor and inject keyboard/mouse events, mapping agent actions to in-game controls.
• Window Management: OS-level APIs (X11 for Linux, pywin32 for Windows) help us detect and manage the Eastshade window’s focus and state.
• Synchronization & Buffering: All inputs and actions are timestamped and buffered to ensure synchronized, multi-modal observations.
• Agent-Environment Interface: We’re developing a custom Python wrapper/API to expose observations and enable action execution, as well as monitor game state and allow for human intervention.
• Data Logging & Monitoring: Every agent action, environmental state, and system event is logged for analysis, with resource usage tracked for stability.
• Safety & Intervention: Mechanisms are in place to pause or stop agent control if human intervention or unexpected states are detected.
• Testing & Validation: Scripted sessions validate integration before full autonomous training begins.

This modular workflow ensures flexibility and robustness as we iterate on both the agent and its environment.

Measuring Success: Insights from Eastshade

A crucial part of our embodied agent project is defining what “success” looks like in a gentle, open-ended environment like Eastshade. Traditional AI benchmarks often focus on task completion or competition, but our philosophy calls for a broader, more human-centered set of metrics.

Drawing from our Eastshade success metrics, we measure the agent’s growth through:

• Exploration Coverage: How much of the world has the agent discovered?
• Novelty Seeking: Is the agent curious—does it seek out new places, objects, and characters?
• Social Interaction: Does it engage with a diverse range of NPCs, fostering empathy and cooperation?
• Environmental Appreciation: Does the agent spend time in scenic areas, create art, and show affinity for beauty?
• Behavioral Diversity: Is it trying new strategies and avoiding repetitive patterns?
• Learning Progression: Is there evidence of improvement and adaptation over time?

We also emphasize intrinsic motivation: the agent is rewarded for curiosity, information gain, aesthetic appreciation, social engagement, and self-reflection. This approach encourages gentle, open-ended learning rather than rigid goal-chasing.

By tracking these metrics, we hope to nurture an agent that values exploration, creativity, and positive social interaction—mirroring the developmental goals we set out from the start. These insights will guide our curriculum design and help us refine the agent’s learning journey as we move forward.

What’s Next: Evolving the Agent

Looking ahead, our goals include:

• Expanding the Curriculum: Introducing more complex, diverse scenarios to test and refine the agent’s values.
• Monitoring and Intervention: Developing tools to detect and address misalignment or value drift over time.
• Community Collaboration: Sharing our findings and inviting feedback to ensure our approach remains robust and ethically grounded.

We’re excited by the progress so far and eager to see how our embodied agent evolves. Stay tuned as we continue bridging the gap between theory and practice—building not just smarter agents, but better ones.

Try It Yourself

Curious about our approach or want to experiment with the embodied agent? You’re invited to view the current state of the project and try it for yourself!
Check out the code, documentation, and ongoing experiments on our GitHub repository:

https://github.com/tinkerforge-ai/experiment-4-embodied-agent