Improved stability of litellm models for reasoning models.

[JoelNiklaus]

No description provided.

[satpalsr]

Hey @JoelNiklaus, since you have made most commits around LiteLLM.
I have few questions/requests

1. Why not expose entire LiteLLM config as parameter or something?
2. I'm actually looking to run open-r1 evals using for instance models deployed at deepinfra with temp=0. Is it possible right now?
   

[JoelNiklaus]

Hi @satpalsr,

1. When I integrated litellm, I made it very similar to the existing openai_model. But I agree with you that we may want to configure more. I am sure the maintainers are open if you want to open a PR for that :)
2. I am not familiar with deepinfra, but since it seems supported by litellm it should work out of the box.

[satpalsr]

@JoelNiklaus Thanks. I'll just drop it as separate issue for anyone to pick.
For time being, I just modified the OpenAIClient code & got my evals done.

[NathanHB]

Why are we removing think tags from the answer here ? I think it should be done in the metric function no ?

[JoelNiklaus]

If we are evaluating a reasoning model the grader will look at the thinking tokens unless we remove them. We would need to remove them in every metric function otherwise.