Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cargo.lock #14483

Merged
merged 27 commits into from
Feb 8, 2025
Merged

Add Cargo.lock #14483

merged 27 commits into from
Feb 8, 2025

Conversation

mbrobbel
Copy link
Contributor

@mbrobbel mbrobbel commented Feb 4, 2025

Which issue does this PR close?

Rationale for this change

See linked issue.

What changes are included in this PR?

  • Remove Cargo.lock from .gitignore
  • Add --locked to cargo in CI
  • Move datafusion-cli back to workspace
  • Remove cli lock file check in CI
  • Add Cargo.lock file for dev/depcheck
  • Remove cargo update from release instructions
  • Update lock files with cargo update
  • Update dependabot config
  • Update cli Dockerfile

Are these changes tested?

CI.

Are there any user-facing changes?

Added Cargo.lock file.

@github-actions github-actions bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Feb 4, 2025
@mbrobbel mbrobbel marked this pull request as ready for review February 4, 2025 15:24
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @mbrobbel -- I think this is quite nice and a significant improvement over the current status. It is also quite nice to be able to do cargo run -p datafusion-cli

I think we can avoid checking in dev/depcheck/Cargo.lock as depcheck is a test / CI tool (and we didn't check in Cargo.lock for proto/gen for exampele:

As this has potential to impact a bunch of workflows I think we should leave this open for a few days to let people have a chance to respond, but all in all I like it a lot ❤️

README.md Show resolved Hide resolved
datafusion-cli/Cargo.toml Show resolved Hide resolved
@xudong963 xudong963 self-requested a review February 5, 2025 07:36
@@ -87,7 +87,7 @@ jobs:
- name: Run tests
run: |
cd datafusion
cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib --tests --features=force_hash_collisions,avro,extended_tests
cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib --tests --features=force_hash_collisions,avro,extended_tests --locked
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo test shouldn't need the --locked flag (here & other places)

it would suffice to verify cargo lock updatodateness once in single job

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I added it everywhere to let all these jobs fail early when the lock file needs to be updated (to save CI resources), because in that case all those jobs are going to have to run again when the lock file is updated. But I'm also fine with only keeping it for one check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fail-fast is not unreasonable, i didn't think about this.

otoh one could say same thing about formatting issues. there is definitely tradeoff between conserving CI resources and maxing our single PR feedback signal.
let's put cargo.lock consistency in the "formatting bucket", ie reuse previous design decision for now. i am open to revisiting it.

Copy link
Contributor Author

@mbrobbel mbrobbel Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll just keep it for

which is required to pass before a bunch of additional jobs are started.

Edit: I meant to link:

linux-build-lib:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!
can you please add a code comment there why --locked is useful?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md Outdated Show resolved Hide resolved
benchmarks/bench.sh Outdated Show resolved Hide resolved
ci/scripts/rust_clippy.sh Show resolved Hide resolved
@findepi
Copy link
Member

findepi commented Feb 5, 2025

@mbrobbel many changes in this PR are about adding --locked to various cargo invocations.
I think it should be possible to verify dependencies in one place, and then keep eg running cargo test without --locked.
The benefit would be not only smaller PR diff, but also easier to comprehend cargo incantation on CI (when a build fails, the command line matters).

If, however, it's necessary to have --locked in cargo test, then i would recommend adding a code comment explaining why it's so. It's not obvious, at least to me.

@alamb
Copy link
Contributor

alamb commented Feb 6, 2025

I will leave this open for another day or so for additional comments and then plan to merge

.github/workflows/rust.yml Outdated Show resolved Hide resolved
@@ -60,7 +60,10 @@ jobs:
with:
rust-version: stable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a nice idea to only use --locked in one test 👍

@alamb
Copy link
Contributor

alamb commented Feb 8, 2025

I merged up from main to resolve a conflict and applied the suggestion. I plan to merge this once we get a clean CI run

@alamb
Copy link
Contributor

alamb commented Feb 8, 2025

🚀 let's give it a try

@alamb alamb merged commit 94d2baf into apache:main Feb 8, 2025
26 checks passed
@alamb
Copy link
Contributor

alamb commented Feb 8, 2025

Thanks @mbrobbel and @findepi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development-process Related to development process of DataFusion documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Discuss: Check in Cargo.lock file?
3 participants