Inconsistent metadata filtering with pod index

Support
, , ,
1

We currently have a pod-based index with the following index stats:

{'dimension': 1536,
 'index_fullness': 0.0,
 'namespaces': {'': {'vector_count': 2109}},
 'total_vector_count': 2109}

When running this query:

results = index.query(
    vector=embedding,
    filter={
        "partners": "28",
    },
    top_k=5,
    include_metadata=True,
)

Results ignore the filter by partner:

{'matches': [{'id': 'https://portales.inacap.cl/carreras/Area-Agroindustria-y-Medioambiente/Ingenieria-Agricola/index',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': [],
                           'provider': 'INACAP',
                           'uuid': '9aa08af4-a329-4c5f-a08c-dff4fcf2c005'},
              'score': 0.881919146,
              'values': []},
             {'id': 'https://portal.inacap.cl/carreras/area-agroindustria-y-medioambiente/ingenieria-agricola',
              'metadata': {'is_available_to_independent_learners': False,
                           'partners': ['28'],
                           'provider': 'INACAP',
                           'uuid': 'c07fef11-0383-4137-8adf-b2e9b83b8803'},
              'score': 0.87269491,
              'values': []},
             {'id': 'https://portales.inacap.cl/carreras/Area-Agroindustria-y-Medioambiente/Tecnico-Agricola/index',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': [],
                           'provider': 'INACAP',
                           'uuid': 'c323f0a2-98f9-457b-9b10-e00435d74dc7'},
              'score': 0.860435963,
              'values': []},
             {'id': 'https://portal.inacap.cl/carreras/area-agroindustria-y-medioambiente/tecnico-agricola',
              'metadata': {'is_available_to_independent_learners': False,
                           'partners': ['28'],
                           'provider': 'INACAP'},
              'score': 0.852069318,
              'values': []},
             {'id': 'https://portales.inacap.cl/carreras/Area-Mecanica/Ingenieria-en-Mantenimiento-Industrial/index',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': [],
                           'provider': 'INACAP',
                           'uuid': 'c77a00bd-bb84-40ae-b167-55753f90b4f9'},
              'score': 0.825636387,
              'values': []}],
 'namespace': ''}

But when running the same query with a different partner value:

results = index.query(
    vector=embedding,
    filter={
        "partners": "3",
    },
    top_k=5,
    include_metadata=True,
)

The filter by partner works as expected:

{'matches': [{'id': '346320',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': ['3'],
                           'provider': 'careervillage',
                           'provider_id': '346320',
                           'text': '346320',
                           'uuid': 'e507f4a1-a940-473c-833c-e366865f6f2b'},
              'score': 0.794816077,
              'values': []},
             {'id': '82829',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': ['3'],
                           'provider': 'careervillage',
                           'provider_id': '82829',
                           'text': '82829',
                           'uuid': 'b8f8c8a6-66b6-4fff-b4a7-cf38b8d9724b'},
              'score': 0.742262185,
              'values': []},
             {'id': '49148',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': ['3'],
                           'provider': 'careervillage',
                           'provider_id': '49148',
                           'text': '49148',
                           'uuid': '10cc1236-4bc9-47dd-bf3f-fee8f2b672d5'},
              'score': 0.733973742,
              'values': []},
             {'id': '447766',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': ['3'],
                           'provider': 'careervillage',
                           'provider_id': '447766',
                           'text': '447766',
                           'uuid': '9e55b361-8910-40c7-b81a-f0145506b927'},
              'score': 0.729193449,
              'values': []},
             {'id': '19897',
              'metadata': {'is_available_to_independent_learners': True,
                           'partners': ['3'],
                           'provider': 'careervillage',
                           'provider_id': '19897',
                           'text': '19897',
                           'uuid': '61633863-5838-405e-b20b-b535dfbbe08b'},
              'score': 0.723347783,
              'values': []}],
 'namespace': ''}

Is this a Pinecone bug? a problem with the index configuration?

Thanks in advance!

2

Hello @rvivas,

The filter code looks incorrect, the filter operation has not been passed in. Can you try this this filter below and see if that resolves the issue:

    filter={
        "partners": {"$eq": "28"}
    },

I will look into this further as an error should be thrown if the filter is incorrect.