Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] - Introduce a "Manager" agent for task evaluation and feedback #154

Open
dannleft opened this issue Nov 18, 2024 · 1 comment
Labels
core Fundamental operations and essential features of AgenticJS feature New feature or request feedback discord library Issues and enhancements directly related to the codebase of the library component of the project. multi-agent-coordination priority-focus

Comments

@dannleft
Copy link
Collaborator

Description

Is your feature request related to a problem? Please describe.
Currently, there is no centralized mechanism within KaibanJS to evaluate and provide feedback on agents' outputs or the final results of their collective tasks. This makes it challenging to ensure logical correctness (e.g., scheduling conflicts) and meet user preferences (e.g., avoiding certain conditions like heat or snow) in complex workflows.

Describe the solution you'd like
Introduce a "Manager" agent or framework that can:

  1. Evaluate: Assess both individual agents' outputs and the final results of their combined tasks based on two criteria:
    • Logical correctness: Ensure outputs meet predefined rules (e.g., no scheduling conflicts, task feasibility).
    • Preferences: Allow personalized criteria (e.g., user dislikes certain weather conditions).
  2. Feedback Mechanism: Automatically provide feedback to agents, enabling them to refine their work in subsequent iterations.
  3. Cost Optimization: Enable the use of smaller, narrow-scope models for agents, while the "Manager" leverages larger reasoning models (e.g., self-hosted or local models) for evaluation and feedback tasks.

Describe alternatives you've considered

  • Using external validation systems to evaluate agent outputs and final results, which increases the complexity of integrations.
  • Adding manual checkpoints in the workflow to review outputs, which is time-consuming and defeats the purpose of automation.

Additional context
This feature would enable:

  • Automated quality control for agent workflows, ensuring logical and preference-based correctness.
  • Advanced use cases such as trip planning, collaborative projects, and multi-agent simulations.
  • Cost optimization by balancing smaller, task-specific models with larger reasoning models for evaluation.
@dannleft dannleft added feature New feature or request feedback discord labels Nov 18, 2024
@darielnoel darielnoel added library Issues and enhancements directly related to the codebase of the library component of the project. core Fundamental operations and essential features of AgenticJS labels Dec 6, 2024
@darielnoel
Copy link
Contributor

Thank you for the feature request! This is indeed related to #164 and #153, as both address aspects of task management and agent coordination. We'll review how this "Manager" agent concept aligns with those discussions and consider it as part of the broader framework improvements. Appreciate your input!

@darielnoel darielnoel added this to the Multi-Agent Coordination 1 milestone Jan 23, 2025
@darielnoel darielnoel removed this from the Multi-Agent Coordination 1 milestone Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Fundamental operations and essential features of AgenticJS feature New feature or request feedback discord library Issues and enhancements directly related to the codebase of the library component of the project. multi-agent-coordination priority-focus
Projects
Status: No status
Development

No branches or pull requests

2 participants