ImageContent has no option (uri, mimeType) ?

[apipino]

• I'm using Google connector 'AddGoogleAIGeminiChatCompletion' (in vb.net)
• I'm uploading 'd:\image.jpg' with api 'Google AI Media-Upload'. ref.: "https://ai.google.dev/api/files"
• for example I am getting 'uri' = "https://generativelanguage.googleapis.com/v1beta/files/qsv8wazc13qj" (uri file without extension)
• VBCode: chatHistory.AddUserMessage(New ChatMessageContentItemCollection From {New TextContent('ask'), New ImageContent(New Uri('imgUri'))}) : response = chat.GetChatMessageContentAsync(chatHistory, settings, kernel)
• Error: Exception ex.Message = "Image content 'MimeType' is empty."

has no option ImageContent(uri, mimeType) (?)

[moonbox3]

@TaoChenOSU could you have a look, please?

[apipino]

@Krzysztof318

[moonbox3]

Ah, sorry, I overlooked the in vb.net comment. Disregard, @TaoChenOSU.

[evchaki]

@RogerBarreto can you take a look at this?

[Krzysztof318]

Regarding to gemini docs LINK even using uploaded image via Uri gemini api requires providing mimeType with vision request. Create new ImageContent with Uri constructor and then specify the mimetype via this property

semantic-kernel/dotnet/src/SemanticKernel.Abstractions/Contents/KernelContent.cs

Line 30 in 113cea0

public string? MimeType { get; set; }

@apipino Let me know does it work, maybe gemini connector needs some changes for uri image processing.

[apipino]

@Krzysztof318 (Note: I have difficulty in English, I'm weak in programming, I don't know how to use GitHub :P)
Dim imgUri As String = Await GoogleUploadMediaFile(imgPath)
Dim imageContent As New ImageContent(imgUri) With {.MimeType = "image/jpeg"} [X] <<==
==>> [X] System.UriFormatException: 'Invalid data uri. Scheme should start with "data:"'
chatHistory.AddUserMessage(New ChatMessageContentItemCollection From {New TextContent(questão), imageContent})

in https://learn.microsoft.com/en-us/dotnet/api/microsoft.semantickernel.imagecontent?view=semantic-kernel-dotnet
have:
ImageContent(ReadOnlyMemory, String mimeType)
ImageContent(String)
ImageContent(Uri)

it does not have: ImageContent(Uri, String mimeType)

[Krzysztof318]

Yeah, it doesn't. That's why I said use property MimeType of this type. If it still won't work please provide full stack trace and code.

[apipino]

Krzysztof318 , Yes, now IT WORKED. ... although yesterday I had done the same test and it had given an error.

      Dim imgUri As Uri = New Uri(Await GoogleUploadMediaFile(imgPath))
      Dim imageContent As New ImageContent(imgUri) With {.MimeType = "image/jpeg"}
      chatHistory.AddUserMessage(New ChatMessageContentItemCollection From {New TextContent(questão), imageContent})
      response = chat.GetChatMessageContentAsync(chatHistory, settings, kernel) ' bot assistant message
      response.Wait() ' Wait for task to complete

Thank you very much!

[apipino]

but I'm waiting for the implementation in the mold "ImageContent(Uri, String mimeType)"

[apipino]

@Krzysztof318 - I don't know if this can help. the code below in VB.net works using "Imports System.Net.Http":
Dim jsonData As String = "{'contents': [{'parts':[{'text': 'Can you tell me about this image?'},{'file_data': {'mime_type': 'image/jpeg', 'file_uri': '" & fileUri & "'}}]}]}"

  Private Async Function GoogleAIUploadMediaFile(filePath As String) As Task(Of String)
    Dim GEMINI_API_KEY As String = Environment.GetEnvironmentVariable("GEMINI_API_KEY")
    If String.IsNullOrEmpty(GEMINI_API_KEY) Then Return Nothing ' API key is missing
    Dim httpGoogleUpload = $"https://generativelanguage.googleapis.com/upload/v1beta/files?key={GEMINI_API_KEY}"
    Dim ext As String = Path.GetExtension(filePath)
    Dim mimeType As String = Interaction.Switch(ext = ".bmp", "image/bmp", ext = ".png", "image/png", ext = ".gif", "image/gif", ext = ".jpg" OrElse ext = ".jpeg", "image/jpeg", True, Nothing)
    If mimeType Is Nothing Then Return Nothing ' Unsupported MIME type
    Dim fileSize As Integer = (New FileInfo(filePath)).Length

    Using client As New HttpClient()
      Dim uploadUrl As IEnumerable(Of String) = Nothing

      Try
        ' Initial request to define metadata
        Using request = New HttpRequestMessage(HttpMethod.Post, httpGoogleUpload)
          request.Headers.Add("X-Goog-Upload-Protocol", "resumable")
          request.Headers.Add("X-Goog-Upload-Command", "start")
          request.Headers.Add("X-Goog-Upload-Header-Content-Length", fileSize.ToString())
          request.Headers.Add("X-Goog-Upload-Header-Content-Type", mimeType)
          request.Content = New StringContent($"{{""file"": {{""display_name"": ""{Path.GetFileName(filePath)}""}}}}", Encoding.UTF8, "application/json")

          Using response As HttpResponseMessage = Await client.SendAsync(request)
            response.EnsureSuccessStatusCode()
            ' Extracts the upload URL from the response headers
            If Not response.Headers.TryGetValues("X-Goog-Upload-URL", uploadUrl) Then Return Nothing ' Error: Upload URL not found
          End Using
        End Using

        ' Upload image
        Using request = New HttpRequestMessage(HttpMethod.Put, uploadUrl(0).ToString)
          request.Headers.Add("X-Goog-Upload-Header-Content-Length", fileSize.ToString())
          request.Headers.Add("X-Goog-Upload-Offset", "0")
          request.Headers.Add("X-Goog-Upload-Command", "upload, finalize")
          request.Content = New StreamContent(File.OpenRead(filePath))

          Using response As HttpResponseMessage = Await client.SendAsync(request)
            response.EnsureSuccessStatusCode()
            ' Extract the URI from the final response file
            Dim responseBody As String = Await response.Content.ReadAsStringAsync()
            Dim jsonDoc = JsonDocument.Parse(responseBody)
            Dim fileUri As String = jsonDoc.RootElement.GetProperty("file").GetProperty("uri").GetString()
            Return fileUri
          End Using
        End Using
      Catch ex As Exception
      End Try
    End Using
    Return Nothing
  End Function

  Sub GoogleAIGenerateContent(fileUri As String)
    Dim request As HttpWebRequest = CType(WebRequest.Create("https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=" & Environment.GetEnvironmentVariable("GEMINI_API_KEY")), HttpWebRequest)
    request.Method = "POST"
    request.ContentType = "application/json"
    Dim jsonData As String = "{'contents': [{'parts':[{'text': 'Can you tell me about this image?'},{'file_data': {'mime_type': 'image/jpeg', 'file_uri': '" & fileUri & "'}}]}]}"
    Using requestStream As Stream = request.GetRequestStream()
      Dim data As Byte() = Encoding.UTF8.GetBytes(jsonData)
      requestStream.Write(data, 0, data.Length)
    End Using
    Dim response As HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
    Using reader As New StreamReader(response.GetResponseStream())
      File.WriteAllText("response.json", reader.ReadToEnd())
    End Using
    Dim content As String = File.ReadAllText("response.json")
    Dim jsonResponse As JsonNode = JsonNode.Parse(content)
    Dim parts As JsonArray = jsonResponse("candidates")?.AsArray()(0)?("content")?("parts")?.AsArray()
    If parts IsNot Nothing Then
      For Each part As JsonNode In parts
        Dim textNode = part?("text")
        If textNode IsNot Nothing Then Resposta.AppendText(textNode.ToString() & Environment.NewLine)
      Next
    End If
  End Sub

[RogerBarreto]

@apipino There are 3 constructors as you can see in the thread.

• ImageContent(ReadOnlyMemory, String mimeType)
  This should be used when you have the Binary information of your image with the mimeType provided

• ImageContent(String dataUri)
  This should be used when you have a base64 binary representation of your image (data-uri)

• ImageContent(Uri)
  This should be used when your image is external, provided by a valid public internet address

That said, analyzing your problem, you should be using the Uri constructor and providing the mimeType as a property.

i.e:

'This should not be a string, use the Uri type constructor (3 option above)
Dim imgUri As Uri = New Uri(Await GoogleUploadMediaFile(imgPath)) 
Dim imageContent As New ImageContent(imgUri) With {.MimeType = "image/jpeg"} 

[apipino]

@RogerBarreto, Yes, now IT WORKED. ... although yesterday I had done the same test and it had given an error.

      Dim imgUri As Uri = New Uri(Await GoogleUploadMediaFile(imgPath))
      Dim imageContent As New ImageContent(imgUri) With {.MimeType = "image/jpeg"}
      chatHistory.AddUserMessage(New ChatMessageContentItemCollection From {New TextContent(questão), imageContent})
      response = chat.GetChatMessageContentAsync(chatHistory, settings, kernel) ' bot assistant message
      response.Wait() ' Aguardar a conclusão da tarefa

Thank you very much!