Key Value store with large data and local file system adopter taking forever to load

[dhrumil4u360]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/core

Issue description

1 - Create new project with local file system as storage driver
2 - Take Key Value store which has 150K+ keys and size around 15GB+
3 - Try to open that key value store. e.g. Actor.openKeyValueStore('name')
4 - You will see code will take forever on that line.

This works on small KVs. Ideally while opening KV, it should not try to load everything!

Code sample

Above steps to reproduce is enough. When it comes to code, just take large KV in local file system and run one line : Actor.openKeyValueStore('name')

Package version

3.11.5

Node.js version

18

Operating system

linux

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

I am running this on server machine with 4GB memory and 200 GB disk

[B4nan]

So you want to load 15gb of data to memory, while you have 4gb of RAM. Sounds like that is your issue, not much we can do about that.

[dhrumil4u360]

I understand your point. I mentioned that intentionally to make a point. My point is why this line

Actor.openKeyValueStore('name')

is attempting to load everything in memory. It should never attempt to load everything in memory. Whole point of KV is that it reads only given key and retrieve it's value. That means if someone is running 1 GB instance on Apify platform, but trying to connect to KV which is of 15 GB, it will not work. But it's working fine there. Because cloud storage adopter never try to load everything into memory from data storage. It only loads value against given key.

But why Local storage is not behaving like that and attempting to load everything into memory!

[B4nan]

Actor.openKeyValueStore does not load all the data to memory, I can imagine its more about the number of items than the total size. Could be about the purging logic of default storages, or something like that. I can also imagine this is about storing 150k+ files in the same directory, that is always a highway to hell. If it's about that, we could deal with this internally, but such changes could be seen as breaking.

Hard to tell what's happening without seeing a complete reproductions. Saying Above steps to reproduce is enough won't help much, we need to see the code, as well as the data, since this is all about the data most likely. Performance issues require complete reproductions.

[dhrumil4u360]

I am all here to assist with whatever it takes. I can provide sample code along with data. The concern is how do I handover data which is like 15GB+ ? Let me know the process for it and I can handover code file which is nothing but one simple function with couple of lines and data folder which is KV storage folder.

[B4nan]

We don't want your data, instead, prepare a script that will generate something in the same scale as you have. Or even better, in a smaller scale that is enough to reproduce the problem. I can imagine this really won't be about the data size, if the open call is the bottleneck, since that really does not read the data. So first just try to generate empty files, sounds like a simple for loop to me.

cc @vladfrangu since he built the memory storage.

[B4nan]

Before we find a solution to this, I wanted to let you know there is an alternative storage client you can use locally, backed by sqlite, which shouldn't suffer from this problem.

https://npmjs.com/package/@apify/storage-local
https://github.com/apify/apify-storage-local-js

You can pass that in the Actor.init() call:

import { ApifyStorageLocal } from '@apify/storage-local';

await Actor.init({
  storage: new ApifyStorageLocal(),
});

[dhrumil4u360]

@B4nan Thank you so much for work around. I tested it and it worked. I also created POC which can reproduce this behavior.

1 - Extract the attached file and do npm install
2 - Run it once : npm start
3 - It will generate KV with 100k keys. If you notice, KV will initialize in no time in first run
4 - Once complete, run it again with npm start
5 - Notice that it will freeze here. Ideally, that line should not do anything with size of data.

Thanks,
Dhrumil
bottleneck-poc.zip

[B4nan]

Thanks, we'll take a look at some point, but I believe the problem is what I said above, too many items in a single folder, we'll probably fix that in next major version, so it will take some time probably. Opened #2723 so we don't forget about it.

[dhrumil4u360]

@B4nan Thank you.