Interesting. Filtering against a list of vector IDs isn’t supported, and as you rightly pointed out, having high cardinality in metadata can impact overall index performance. But there might be an option using sparse-dense vectors, instead. You could encode the vector IDs into the sparse vectors and use that to filter against in the query.
This notebook gives an example of doing something similar, but in e-commerce search when filtering on different types of products. But the principle is basically the same.