Can VectorDBBench be configured to set the number of concurrent operations for inserting data into Elasticsearch?

[midonKing]

When we were conducting the regular VectorDBBench test to evaluate the performance of Elasticsearch, we found that the insert efficiency of Elasticsearch was extremely slow. We speculate that this is because VectorDBBench did not perform concurrent insertions when importing data. In reality, Elasticsearch in production applications supports multiple concurrent insertions. The insert performance test results here do not match the expectations. Can VectorDBBench set the number of concurrent insertions for Elasticsearch?

[XuanYang-cn]

@midonKing You could try streaming cases, we do concurrent insert and search there. The insert concurrency is more flexible, you can change it from 100 rows/s to 4000 rows/s (around 20MB/s), and higher if you like.

[midonKing]

Thank you. When using the Customize Streaming Test, there is this prompt when entering the data. I set the concurrency to 32. Does this mean that the data is written concurrently from 32 channels? Is the efficiency of 3s too low?

[midonKing]

After numerous test attempts, I found that neither increasing the insert_rate, increasing the concurrencies, nor increasing the number_of_shards could improve the insert efficiency. Under the test specification of 1 million data points, the insert time was around 800 seconds. Is this the limit? Are there any other methods to improve the efficiency?

[alwayslove2013]

@midonKing The limit of insert_rate is mainly restricted by the Milvus instance, and vdbbench can only increase the frequency of insertion requests by the client.

If you need a higher insertion rate, importing data in the form of files is likely the best option.
You can refer to the detailed guide here: https://milvus.io/docs/import-data.md