refactor: move DataSource to datafusion-datasource

[logan-keede]

Which issue does this PR close?

• Part of [Epic] Split datasources out from datafusion crate (datafusion/core) #14444.

Rationale for this change

What changes are included in this PR?

refactor of DataSource

Are these changes tested?

not completely, only cargo check work for now

Are there any user-facing changes?

yes, but only if they are building from source.

[alamb]

So exciting...

[logan-keede]

@alamb any Idea on what to do with unit tests in physical-plan? I cant think of anything except pulling functions that depend on the component that I moved, to core_integration tests or maybe to datasource crate. is there a better method?

[alamb]

@alamb any Idea on what to do with unit tests in physical-plan? I cant think of anything except pulling functions that depend on the component that I moved, to core_integration tests or maybe to datasource crate. is there a better method?

I looked at the errors, and most of them appear to be related to the tests in physical_plan using MemoryExec (well now DataSource with a MemoryExec).

Some other ideas:

1. Add a dev-dependency between datafusion-physical-plananddatafusion-datasource`
2. (better) Add a mock ExecutionPlan only for testing in physical_plan -- this would end up with some amount of duplicate code with DataSourceExec but I think that would be ok for testing.
3. Move the tests, as you say, into tests in core_integration or something

[alamb]

BTW thank you for working on this -- I suggest we try the "make a mock ExecutionPlan" approach -- that would be the cleanest I think and would be a good way for you to learn more about how ExecutionPlans worked

I can try and find time to sketch it out if needed

[logan-keede]

BTW thank you for working on this -- I suggest we try the "make a mock ExecutionPlan" approach -- that would be the cleanest I think and would be a good way for you to learn more about how ExecutionPlans worked

Thanks for the suggestions. I will try to make MockExecutionPlan.

[logan-keede]

@alamb, I have implemented a MockMemorySourceConfig and MockDataSource which is essentially same as what was previously in physical_plan but now for tests only. You can find the implementation here. The tests are passing on my PC, although there are some warnings, so cleaning up that part is still pending.

I think this is different from what you were suggesting. It would be pretty helpful if you can point me in the right direction.

Thanks,
Logan

[alamb]

@alamb, I have implemented a MockMemorySourceConfig and MockDataSource which is essentially same as what was previously in physical_plan but now for tests only. You can find the implementation here. The tests are passing on my PC, although there are some warnings, so cleaning up that part is still pending.

I think this is different from what you were suggesting. It would be pretty helpful if you can point me in the right direction.

Thanks, Logan

I think this sounds reasonable -- I was just saying that the split between MockMemorySourceConfig and MockDataSource is likely not necessary. There is a split in the current DataSourceConfig and DataSourceExec to permit sharing between difefrent data sources

[logan-keede]

I think this sounds reasonable -- I was just saying that the split between MockMemorySourceConfig and MockDataSource is likely not necessary.

It does look that way, I will try to make it one Mock struct.

[logan-keede]

@alamb What do we want to do with benches? (they are causing circular dependency)
except that I think this is ready for review.