Implementing AgentGroupChat with Chat-Agent, Tool-Agent, and Semantic Kernel Integration"

[nblog]

I am working in a Python environment and have a slightly more complex requirement than the examples provided in the Python sample code. The examples seem insufficient to fully cover my use case.

I want to implement an AgentGroupChat with the following structure:

AgentGroup:

• chat-agent
• tool-agent

Workflow:

1. User Input: The user provides an input.
2. Selection Strategy: A strategy determines whether to route the input to the chat-agent or the tool-agent.
3. Agent Execution: The selected agent handles the task.

---

Requirements:

tool-agent: A tool-based agent capable of invoking native functions (involving plugins).

• Problem:
  The native functions in the plugins are numerous and some have complex or non-standard rules. For example:

  • Normally, disk resizing can only expand to the right side. However, I want to allow left-side expansion under certain conditions.

• Desired Workflow:
  I want the tool-agent to:

  1. Use function selection (without arguments) to identify the most relevant function for the user input.
  2. Before invoking the function, apply a custom prompt to ensure the model understands the rules and constraints of the function.
  3. Generate reasonable arguments for the function call based on the rules and user input.

• Example Code:

# sk.py
agent_tools = ChatCompletionAgent(
    ...
    execution_settings=OpenAIChatPromptExecutionSettings(
        tool_choice="auto",
        function_choice_behavior=FunctionChoiceBehavior.Auto(),
    )
)

# tool.py
class NativePlugin:
    @kernel_function(description="Resize or move a specified partition")
    def resize_move_partition(
            self,
            disk_index: int,
            partition_index: int,
            capacity: int,
            starting_offset: int):
        pass

• Prompt Example:
  I want to provide the following prompt to guide the model when selecting and invoking the resize_move_partition function:

- Adjust partitions by expanding or shrinking using unallocated space on either the left or right side.
- Function Description:
    - Resize or move a partition.
- Function Rules:
    - Expand Partition:
        - Can only use unallocated space adjacent to the partition.
    - Shrink Partition:
        - Can shrink the partition, but it must remain larger than the used space.
- Function Hints:
    - When expanding, check for unallocated space adjacent to the partition.
    - When shrinking, ensure the resulting size is greater than the used space.
- Partition Layout:
    [ Reserved | C: | Unallocated1 | D: | E: | Unallocated2 ]
    - Correct:
        - `D:` can expand using `Unallocated1` on the left, but the starting offset must be recalculated.
    - Incorrect:
        - `D:` cannot expand using `Unallocated2` on the right because it would cross the `E:` partition.

• Challenge:
  I am unsure how to:
  1. Set a custom prompt before invoking the function.
  2. Ensure the model applies the rules in the prompt to reason about the function call and generate valid arguments.

---

Additional Notes:

• For the tool-agent, should I use a separate prompt for each function, or is there a way to dynamically generate prompts based on the function being invoked?
• For the chat-agent, are there specific libraries or frameworks that simplify RAG integration with conversational agents?

---

By providing this detailed issue, I hope to clarify the requirements and challenges I am facing. Any guidance, suggestions, or examples would be greatly appreciated.

[moonbox3]

Hi @nblog, thanks for your question. Here are my thoughts:

1. Firstly, do you really need an agent group chat to accomplish this? Why not create your chat using a chat completion AI service, and advertise the plugin(s) (tool(s)) to the model. Then based on the context, it can call the tool(s) or not? This would allow for additional user input if the model requires it. I'd say if this particular method doesn't get you your desired results, then you can start to think about how agents might help.
2. If you do decide to move to the agent framework, there are many things to thing about:
   • The way we've explored gathering user input as part of an agent chat can be seen in the following code example on our learn site: https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/examples/example-agent-collaboration?pivots=programming-language-python#the-chat-loop. You'll want to experiment with this.
   • In the Python Agent framework, it is currently missing the ability to specify a prompt template config. That's going to be handled in Python Agents - Support PromptTemplateConfig by any agent #8699 as well as allowing for kernel argument override as part of Python Agents: Allow Kernel and KernelArguments override when invoking ChatCompletionAgent and OpenAIAssistantAgent` #8022. I plan to pick these up next week.
   • How do you plan to terminate the group chat? Do you let it run for a default number of iterations? Do you exit on user input? On agent input? You'll want to explore our termination strategies or bring your own, as you override the TerminationStrategy base class.

To make sure I do understand your use-case well enough to help, I do have some clarifying questions for you:

Set a custom prompt before invoking the function.

Can you help me understand what setting a custom prompt before invoking the function means? A normal chat flow with a model is:

• The user queries the model with an initial question or ask.
• If using auto function calling, the model will have access to tools that it can use to help generate the answer. If you configure FunctionChoiceBehavior.Auto() you're allowing the model to make a decision on its own to use a tool or not to generate an answer.
• If the model wants to invoke a function it returns a tool call, which is handled in SK as FunctionCallContent.
• SK then invokes the local native function, and generates FunctionResultContent.
• The model receives the FunctionResultContent and generates a final answer using the includes content in the chat history
• TextContent is then returned to the user hopefully satisfying the original query.

This same flow happens in both a chat completion scenario and an agent-based scenario. The only thing that differs is the context that it is run -- whether as a chat bot type set up, a particular workflow kicking it off, or as part of an agent group chat.

Is your "custom prompt" part of the original ask to the model, or does it get used at some other point in the process?

For the tool-agent, should I use a separate prompt for each function, or is there a way to dynamically generate prompts based on the function being invoked?

Similar to my question above, I'm not sure I understand "should I use a separate prompt for each function, or is there a way to dynamically generate prompts based on the function being invoked?"

The tool is advertised to the model based on its JSON Schema. You can visit the following blog post to see how the native function gets turned into this JSON schema. If you want to provide more context to the model you can make sure you provide semantically meaningful and descriptive function names, descriptions, and arguments as they are all sent to the model. This would help with your comment about The native functions in the plugins are numerous and some have complex or non-standard rules.

Use function selection (without arguments) to identify the most relevant function for the user input.

What do you mean you want function selection (without arguments) to identify the most relevant function for user input? Based on the sample function you provided above (resize_move_partition), there are four required arguments. If the model thinks, based on the current conversation context, that it should call that function, and it doesn't have those four required parameters it will ask the user for that information before it makes the function call. "I'd love to help you resize your partition. Can you provide me the capacity?" (Perhaps in the conversation context, the other three function arguments were already provided).

Before invoking the function, apply a custom prompt to ensure the model understands the rules and constraints of the function.

I haven't specifically tried this, but I suppose you could experiment with a filter to help with this? This part of the flow still isn't clear for me, so if you can elaborate more, it would help.

[nblog]

Thank you for your suggestion! After careful consideration, I decided to use an agent group approach. This allows me to debug each agent individually and then integrate them into a group, which aligns better with my needs—especially as I anticipate having multiple agents in the future.

Currently, I am facing challenges with the tool-agent. My plugin contains over 100 functions, many of which have complex rules. Initially, I tried embedding these rules directly into the function descriptions, like this:

@kernel_function(description=\
"""
- Adjust partitions by expanding or shrinking using unallocated space on either the left or right side.
- Function Description:
    - Resize or move a partition.
- Function Rules:
    - Expand Partition:
        - Can only use unallocated space adjacent to the partition.
    - Shrink Partition:
        - Can shrink the partition, but it must remain larger than the used space.
- Function Hints:
    - When expanding, check for unallocated space adjacent to the partition.
    - When shrinking, ensure the resulting size is greater than the used space.
- Partition Layout:
    [ Reserved | C: | Unallocated1 | D: | E: | Unallocated2 ]
    - Correct:
        - `D:` can expand using `Unallocated1` on the left, but the starting offset must be recalculated.
    - Incorrect:
        - `D:` cannot expand using `Unallocated2` on the right because it would cross the `E:` partition.
""")
def resize_move_partition():
    ...

However, this approach has two major issues:

1. It consumes a significant amount of tokens, which can be inefficient.
2. The large number of functions and detailed descriptions often lead to confusion or incorrect responses.

To address this, I switched to a function selection approach. Instead of including detailed rules in the function descriptions, I now provide only the function names and concise descriptions. The goal is to identify the most relevant function based on the user’s question. For example:

{{chat_history}}

<functions>
<func>resize_move_partition</func><doc>Resize or move a specified partition</doc>
...
</functions>

User Question: {{question}}

Please identify the most relevant function from the list above and return its name only.

Once the function is identified, I plan to use its detailed rules and logic to guide the next steps. For example:

{{chat_history}}

- Adjust partitions by expanding or shrinking using unallocated space on either the left or right side.
- Function Description:
    - Resize or move a partition.
- Function Rules:
    - Expand Partition:
        - Can only use unallocated space adjacent to the partition.
    - Shrink Partition:
        - Can shrink the partition, but it must remain larger than the used space.
- Function Hints:
    - When expanding, check for unallocated space adjacent to the partition.
    - When shrinking, ensure the resulting size is greater than the used space.
- Partition Layout:
    [ Reserved | C: | Unallocated1 | D: | E: | Unallocated2 ]
    - Correct:
        - `D:` can expand using `Unallocated1` on the left, but the starting offset must be recalculated.
    - Incorrect:
        - `D:` cannot expand using `Unallocated2` on the right because it would cross the `E:` partition.

Additionally, I want to leverage prompt templates to dynamically select the appropriate model based on the complexity of the function. For example:

• For simpler functions, I might use a lightweight model like gpt-4o-mini.
• For functions with complex rules, I would prefer a more powerful model like gpt-4o.

Given the above, I would greatly appreciate any guidance, suggestions, or examples on:

1. How to better structure and optimize the tool-agent for handling a large number of functions with complex rules.
2. Best practices for combining function selection and detailed rule-based reasoning in Semantic Kernel.
3. How to implement dynamic prompt templates for model selection based on function complexity.

Thank you in advance for your help!