What happens if upsert/write operations happen too frequently?

Support
1

I am building an application where users post their contents and search contents by others. As such, the upsert/write needs to happen by one vector at a time (i.e., when a user makes a new post). As the user base grows, this will result in highly frequent upsert/write requests where, again, each request contains a single vector.

  1. How does this affect, if at all, the upsert/write performance? Will one request block another? Should I make all requests asynchronous as per this suggestion? If so, should I manually set the value for the pool_threads parameter? What is the max value I can use?

  2. Will this frequent upsert/write of single vectors overburden the DB (because it needs to be updated or “re-processed” too frequently)?

  3. Will this frequent upsert/write of single vectors negatively affect query performance?

2

Hi @yspark, sorry for the late reply.

  1. Requests to an index has an upper limit which depends on the pod type and size. Different index configurations may have different performance characteristics but you can dig deeper about QPS and index sizes.

  2. If you overload the DB with upserts/writes, that may result in 504 errors.

  3. Yes, this may cause read and write issues. The remedy is to slow down the batching of upserts/writes. We are actively working improving this.

1 Like