Mlx generate text, by default halusonates more

[kishoretvk]

Issue Description
Problem: When using the mix generate text command with verbose set to false, and the following parameters:

Temperature: 0.1 or 0
Top p: 1
The LLM models seem to hallucinate more frequently.

Models Used:

Mistral 7B v3
IBM Granite
Mistral 7B v2
Use Case: Text-to-SQL

Method Used: mlx.generate_text

Steps to Reproduce
Set up the environment with the specified models.
Run the mix generate text command with verbose set to false.
Use a temperature of 0.1 or 0 and top p of 1.
Observe the output for hallucinations.
Expected Behavior
The models should generate accurate and coherent SQL queries without hallucinations.

Actual Behavior
The models produce outputs that are factually incorrect or nonsensical, indicating hallucinations.

Additional Information
Dataset: 20,000 examples split for training and validation.
Hardware: 128GB M3 Max

[awni]

Could you share the commands you used and the precise model paths?

Hardware: 40 GB A100 GPU.

Also MLX is meant for Apple silicon.. is that a typo or were you running it on a linux machine?

[kishoretvk]

Sorry my bad 128 gb ram
M3 max

[awni]

How about the commands / model paths?

Also when you say "by default halusonates more" what are you comparing against?