How to optimize server ?

[ducvu91]

I have 2 point relate to server.

1. task run very slow, especially Find neighbors of items. my server have 8cpu 4 core, 16g RAM.
2. how to optimize cache of redis ? i just run testing with about 50k items, but 16G ram redis it not enought. while in reality i have over 5 milion items. and over 1000 category.
3. Simply calculate, 50k items => 16G RAM, 500k item => 160G RAM, 5000k => 1600G RAM ? OMG, terrible. i was change database type for cache, but it very very slow.

Thanks.
Would be very grateful if there is any solution to help me improve

[unknownYuan]

ram的使用不会线性增长

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

RAM usage does not increase linearly

[zhenghaoz]

Tell me the version name please. 0.4 or 0.5

[ducvu91]

i think version is 0.5. i just pull latest again.
Now i reduce data categories each item but the task Find neighbors of items still slow. it run over 30 minute but just not yet 50%.
My data 42k item 0 user, 0 feedback, per item have multi label and multi categories. label i think 10 from 20 label per item, categories is from 2 to 5 per item

[ducvu91]

hello,
any solution ?
Now my data only have 14k item, only 2 categories. it used 7.5G Ram. idk i wrong config or something wrong else but this used too much resource ?
My production data : over 5 milion item, each item have 5-15 categories (i used categories for multi purpose, not only category), over 30-50 label. over 1 milion users, 24k active users per day, so I think the feedback will be huge.
@zhenghaoz should we have an optimize for it or new solution for it. If it still same, it can't be run on production, for research and educational purposes only.
One more: task run very slow. how to improvement it ?
I look forward to your solution and reply.
Thanks again.

[kienvtqhi]

hello, any solution ? Now my data only have 14k item, only 2 categories. it used 7.5G Ram. idk i wrong config or something wrong else but this used too much resource ? My production data : over 5 milion item, each item have 5-15 categories (i used categories for multi purpose, not only category), over 30-50 label. over 1 milion users, 24k active users per day, so I think the feedback will be huge. @zhenghaoz should we have an optimize for it or new solution for it. If it still same, it can't be run on production, for research and educational purposes only. One more: task run very slow. how to improvement it ? I look forward to your solution and reply. Thanks again.

Hello, Did you find any solution to optimize the redis cache size and make the worker run more fast? Our [dev] version took 2-3 minutes to create ranking for each users. 50k users - 50k items - 3 categories - 200k feedbacks.