Add SambaNova models to Autogen

[snova-rodrigom]

What feature would you like to be added?

SambaNova Cloud LLM models

Why is this needed?

Integration implementation of SambaNova models to Autogen

[ekzhu]

You are welcome to contribute and create community packages. For non-openai, server-side LLM support, we want to take a very careful approach not to overextend and over stretch our ability to maintain them.

[rysweet]

@snova-rodrigom you can have a look at the existing extensions to get an idea of how to do this.

[jackgerrits]

Looks like SambaNova provides an OpenAI compatible api is that right?

[snova-rodrigom]

hello guys, thanks for your quick responses. @rysweet could you please reference to me to any documentation about existing documentations? and yes @jackgerrits SambaNova is OpenAI compatible :)

[victordibia]

Hi @snova-rodrigom ,

Documentation on model clients is here https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/framework/model-clients.html

You would typically need to set the model base_url, model and api-key following what you get from your provider - Samba Nova

from autogen_core.models import UserMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient

# Create an OpenAI model client.
model_client = OpenAIChatCompletionClient(
    model="Meta-Llama-3.1-8B-Instruct",
    # api_key="YOUR SAMBANOVA API KEY",  
    base_url="https://api.sambanova.ai/v1" ,  
)

[rodrigo-92]

Thank you @victordibia, I tried what you suggested, however I'm getting the following error ValueError: model_info is required when model name is not a valid OpenAI model . Trace stack bellow:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:358, in BaseOpenAIChatCompletionClient.__init__(self, client, create_args, model_capabilities, model_info)
    [357](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:357) try:
--> [358](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:358)     self._model_info = _model_info.get_info(create_args["model"])
    [359](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:359) except KeyError as err:

File ~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_model_info.py:155, in get_info(model)
    [154](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_model_info.py:154) resolved_model = resolve_model(model)
--> [155](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_model_info.py:155) return _MODEL_INFO[resolved_model]

KeyError: 'Meta-Llama-3.1-70B-Instruct'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In[6], [line 5](vscode-notebook-cell:?execution_count=6&line=5)
      [2](vscode-notebook-cell:?execution_count=6&line=2) from autogen_ext.models.openai import OpenAIChatCompletionClient
      [4](vscode-notebook-cell:?execution_count=6&line=4) # Create an OpenAI model client.
----> [5](vscode-notebook-cell:?execution_count=6&line=5) model_client = OpenAIChatCompletionClient(
      [6](vscode-notebook-cell:?execution_count=6&line=6)     model="Meta-Llama-3.1-70B-Instruct",
      [7](vscode-notebook-cell:?execution_count=6&line=7)     api_key="66b92009-0107-46c3-8bcf-e41e4e07c37[8](vscode-notebook-cell:?execution_count=6&line=8)",  
      8     base_url="https://api.sambanova.ai/v1" ,  
      [9](vscode-notebook-cell:?execution_count=6&line=9) )

File ~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1020, in OpenAIChatCompletionClient.__init__(self, **kwargs)
   [1018](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1018) create_args = _create_args_from_config(copied_args)
   [1019](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1019) self._raw_config: Dict[str, Any] = copied_args
-> [1020](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1020) super().__init__(
   [1021](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1021)     client=client, create_args=create_args, model_capabilities=model_capabilities, model_info=model_info
   [1022](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:1022) )

File ~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:360, in BaseOpenAIChatCompletionClient.__init__(self, client, create_args, model_capabilities, model_info)
    [358](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:358)         self._model_info = _model_info.get_info(create_args["model"])
    [359](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:359)     except KeyError as err:
--> [360](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:360)         raise ValueError("model_info is required when model name is not a valid OpenAI model") from err
    [361](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:361) elif model_capabilities is not None and model_info is not None:
    [362](https://file+.vscode-resource.vscode-cdn.net/Users/rodrigom/github_repos/autogen/test/~/github_repos/autogen/.venv/lib/python3.11/site-packages/autogen_ext/models/openai/_openai_client.py:362)     raise ValueError("model_capabilities and model_info are mutually exclusive")

ValueError: model_info is required when model name is not a valid OpenAI model

[ekzhu]

@rodrigo-92 as the error message indicates, please include model_info. See example:

from autogen_core.models import UserMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient

# Create an OpenAI model client.
model_client = OpenAIChatCompletionClient(
    model="Meta-Llama-3.1-8B-Instruct",
    # api_key="YOUR SAMBANOVA API KEY",  
    base_url="https://api.sambanova.ai/v1" ,  
    model_info={"json_output": False, ...},
)

https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.OpenAIChatCompletionClient

[rodrigo-92]

thanks @ekzhu, it's working! Regarding documentation about existing extensions, could you point me to them? Is is possible to add SambaNova as an additional extension? I'm willing to cooperate with a PR since I work for them hehe.

[ekzhu]

A community extension is the way to go here. Who other than yourself is the best person to maintain this 😄

You can look at: https://microsoft.github.io/autogen/stable/user-guide/extensions-user-guide/discover.html for existing extensions we have indexed, and https://microsoft.github.io/autogen/stable/user-guide/extensions-user-guide/create-your-own.html for creating your own extensions.

[rodrigo-92]

awesome, thanks for the references. I'll check with team what would be the best approach.

[ekzhu]

Using this issue to track the integration.