Hi there,
I’m trying to figure out why single and concurrent queries are taking a long time. My setup is a serverless one, hosted in the US. Our web data provider is hosted in the EU. So there might be some RTT delays degrading the query performance, but I’m not sure if that’s really causing the issue.
I’m running similar indices with virtually the same amount of vectors + metadata on Ireland and East-Virginia and I’m getting 800ms - 1s of additional response times.
Before I make the call to move everything over from the Virginia based one to Ireland, I want to make sure that this is indeed somewhat expected, because the index has about 4 million vectors and moving those brings along a cost with it.
I can imagine including/excluding the meta-data also adds up to the response time as those need to be transferred across as well.
Anyone confirming this would be a great help to make the call to move from US to EU, which is much close to where our app provider is located. I just expected more of a 100+ ms delay, not a near second one. My best guess is that those RTT are not predictable and influenced by a variety of factors.
Hi @jan.vansteenlandt, and thanks for your question!
I wonder if there are ways you could gather additional diagnostics before deciding on a move:
- Are you able to see similar RTT with traceroute or a similar network diagnostic tool going from your US location to Ireland? In other words, I’m attempting to eliminate variables to confirm if the slowness is coming from your index operations or not
- Can you instrument your services with tracing or an observability tool that could provide additional sanity checks?
- Can you re-try your same tests while omitting metadata and omitting values (vectors?) Do you get similar results?
Hope this helps!
Best,
Zack
Hi Zack,
I’ve gathered these requests with their runtime + the amount of parallel requests that were done. We make 20 concurrent requests per batch of requests…it seems the response times are not really related to the amount of queries that are being done. What’s your take? I’ve pasted them on the bottom of the response. Since last week I’ve omitted both values and metadata. As you can see from the last log lines, that doesn’t really impact things as much it seems.
What I may be observing though Zack is that after an initial query is done, subsequent (other) queries are returning much faster. Is there some kind of “spin up” time on those serverless indexes?
The traceroute, I’ve only gotten the traceroute from my local machine which is in the same country as our provider (Belgium):
8 * prs-bb1-link.ip.twelve99.net (62.115.116.238) 18.841 ms *
9 ash-bb2-link.ip.twelve99.net (62.115.112.242) 106.495 ms 104.496 ms 106.722 ms
10 lax-b22-link.ip.twelve99.net (62.115.121.220) 157.966 ms 160.231 ms 158.951 ms
11 amazon-ic-325184.ip.twelve99-cust.net (62.115.155.113) 158.971 ms 160.607 ms 162.145 ms
Response logs
[2024-05-20 13:11:48]Pinecone query took 0.79687404632568 seconds with a total requests of: 2
[2024-05-20 13:12:58]Pinecone query took 1.9508829116821 seconds with a total requests of: 12
[2024-05-20 13:12:58]Pinecone query took 2.4459478855133 seconds with a total requests of: 41
[2024-05-20 13:13:18]Pinecone query took 1.1790900230408 seconds with a total requests of: 12
[2024-05-20 13:13:23]Pinecone query took 0.78165698051453 seconds with a total requests of: 2
[2024-05-20 13:15:23]Pinecone query took 0.68325996398926 seconds with a total requests of: 12
[2024-05-20 13:15:26]Pinecone query took 0.8060998916626 seconds with a total requests of: 2
[2024-05-20 13:15:32]Pinecone query took 1.0838761329651 seconds with a total requests of: 12
[2024-05-20 13:15:36]Pinecone query took 0.7878110408783 seconds with a total requests of: 2
[2024-05-20 13:16:09]Pinecone query took 0.62884187698364 seconds with a total requests of: 12
[2024-05-20 13:16:13]Pinecone query took 1.3124890327454 seconds with a total requests of: 2
[2024-05-20 13:16:17]Pinecone query took 2.1897661685944 seconds with a total requests of: 41
[2024-05-20 13:16:18]Pinecone query took 0.60330986976624 seconds with a total requests of: 12
[2024-05-20 13:16:21]Pinecone query took 0.69950199127197 seconds with a total requests of: 2
[2024-05-20 13:16:26]Pinecone query took 0.63297390937805 seconds with a total requests of: 12
[2024-05-20 13:16:29]Pinecone query took 0.70969891548157 seconds with a total requests of: 2
[2024-05-20 13:16:50]Pinecone query took 1.0683038234711 seconds with a total requests of: 41
[2024-05-20 13:17:03]Pinecone query took 0.63805103302002 seconds with a total requests of: 12
[2024-05-20 13:17:05]Pinecone query took 0.70410799980164 seconds with a total requests of: 2
[2024-05-20 13:17:23]Pinecone query took 0.78600406646729 seconds with a total requests of: 41
[2024-05-20 13:17:35]Pinecone query took 1.2981760501862 seconds with a total requests of: 12
[2024-05-20 13:28:47]Pinecone query took 2.5000829696655 seconds with a total requests of: 12
[2024-05-20 13:28:48]Pinecone query took 0.68466711044312 seconds with a total requests of: 12
[2024-05-20 13:29:06]Pinecone query took 2.0580418109894 seconds with a total requests of: 12
[2024-05-20 13:29:19]Pinecone query took 1.3027520179749 seconds with a total requests of: 12
[2024-05-20 13:40:40]Pinecone query took 3.1273419857025 seconds with a total requests of: 12
[2024-05-20 13:41:05]Pinecone query took 1.0375440120697 seconds with a total requests of: 12
[2024-05-20 13:43:09]Pinecone query took 1.6899719238281 seconds with a total requests of: 12
[2024-05-20 13:43:13]Pinecone query took 1.7942810058594 seconds with a total requests of: 2
[2024-05-20 13:44:41]Pinecone query took 1.1279890537262 seconds with a total requests of: 12
[2024-05-20 13:44:44]Pinecone query took 0.69678092002869 seconds with a total requests of: 2
[2024-05-20 13:45:32]Pinecone query took 0.70369410514832 seconds with a total requests of: 12
[2024-05-20 13:45:36]Pinecone query took 0.88787317276001 seconds with a total requests of: 2
[2024-05-20 13:45:48]Pinecone query took 1.7192399501801 seconds with a total requests of: 12
[2024-05-20 13:45:48]Pinecone query took 0.7590970993042 seconds with a total requests of: 12
[2024-05-20 13:45:50]Pinecone query took 0.69885802268982 seconds with a total requests of: 2
[2024-05-20 13:45:52]Pinecone query took 0.69782590866089 seconds with a total requests of: 2
[2024-05-20 13:48:34]Pinecone query took 2.1268858909607 seconds with a total requests of: 12
[2024-05-20 13:49:29]Pinecone query took 1.1262099742889 seconds with a total requests of: 12
[2024-05-20 13:51:31]Pinecone query took 0.91015791893005 seconds with a total requests of: 13
[2024-05-20 13:51:50]Pinecone query took 1.0035011768341 seconds with a total requests of: 13
[2024-05-20 13:53:30]Pinecone query took 1.0082778930664 seconds with a total requests of: 13
[2024-05-20 13:57:44]Pinecone query took 1.9017050266266 seconds with a total requests of: 14
[2024-05-20 13:57:49]Pinecone query took 0.78634810447693 seconds with a total requests of: 14
[2024-05-20 13:59:03]Pinecone query took 2.0340130329132 seconds with a total requests of: 41
[2024-05-21 08:29:07]Pinecone query took 4.0044040679932 seconds with a total requests of: 26
[2024-05-21 08:29:08]Pinecone query took 5.2336230278015 seconds with a total requests of: 8
[2024-05-21 08:32:24]Pinecone query took 2.8436148166656 seconds with a total requests of: 26
[2024-05-21 08:32:27]Pinecone query took 1.522068977356 seconds with a total requests of: 26
[2024-05-21 08:35:15]Pinecone query took 3.3819329738617 seconds with a total requests of: 8
[2024-05-21 08:37:08]Pinecone query took 1.1995079517365 seconds with a total requests of: 8
[2024-05-21 08:53:06]Pinecone query took 4.8764698505402 seconds with a total requests of: 8
[2024-05-21 08:53:06]Pinecone query took 5.6187660694122 seconds with a total requests of: 8
[2024-05-21 08:53:06]Pinecone query took 4.8202819824219 seconds with a total requests of: 8