AI Proxy plugin support for OpenAI ChatCompletion base64 image inputs

[derekelewis]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong 3.8.0

Current Behavior

Calls made when passing a URL to image URL appear to work fine; however, when passing an image directly as base64 encoded (per https://platform.openai.com/docs/guides/vision), the following error is returned to the client. The call works as expected when using the OpenAI API directly.

python model_vision_test.py
Traceback (most recent call last):
  File "/Users/dlewis/model_vision_test.py", line 15, in <module>
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 829, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 1278, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 955, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 1059, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'content-type header does not match request body, or bad JSON formatting'}}
(mlenv) ➜  ~ python model_vision_test.py
Traceback (most recent call last):
  File "/Users/dlewis/model_vision_test.py", line 15, in <module>
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 829, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 1278, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 955, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/Users/dlewis/mlenv/lib/python3.11/site-packages/openai/_base_client.py", line 1059, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'content-type header does not match request body, or bad JSON formatting'}}

Expected Behavior

The API call should return the same when passing a base64 encoded image directly vs the image URL.

Steps To Reproduce

Example code to reproduce:

import base64

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/openai")

with open(
    "000000039769.jpg",
    "rb",
) as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in the image?",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
                },
            ],
        }
    ],
)

print(response)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in the image?",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image}"}
                },
            ],
        }
    ],
)

print(response)

Anything else?

No response