Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_custom_hf_dataset not handling the text_feature argument properly #1087

Open
chimezie opened this issue Nov 3, 2024 · 0 comments
Open

Comments

@chimezie
Copy link
Contributor

chimezie commented Nov 3, 2024

If you use a hf_dataset configuration such as (for e.x.)

hf_dataset:
  name: "Open-Orca/OpenOrca"
  train_split: "train[:90%]"
  valid_split: "train[-10%:]"
  text_feature: "response"

It is supposed to work the same as the (local) text data format, but it comes up against:

        if prompt_feature and completion_feature:
            return CompletionsDataset(ds, tokenizer, prompt_feature, completion_feature)
        elif text_feature:
            return Dataset(train_ds, text_key=text_feature)
        else:
            raise ValueError(
                "Specify either a prompt and completion feature or a text "
                "feature for the Hugging Face dataset."
            )

which errors out because train_ds is not defined

chimezie added a commit to chimezie/mlx-examples that referenced this issue Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant