RuntimeError: Already borrowed

Let’s keep it open until we have a nicer fix

I found a really ugly solutionhack: instanciate multiple pipelines, and ensure they are not used concurrently via mutex.

in rest_api/controller/search.py I replaced the PIPELINE and /query code by:

from random import randrange
from threading import Lock

NUMBER_OF_PIPELINES = 6
PIPELINES = []
MUTEXES = []
for i in range(NUMBER_OF_PIPELINES):
    logger.info(f"Loading pipeline: {i}")
    PIPELINES.append(Pipeline.load_from_yaml(Path(PIPELINE_YAML_PATH), pipeline_name=QUERY_PIPELINE_NAME))
    MUTEXES.append(Lock())
    logger.info(f"Loaded pipeline nodes: {PIPELINES[i].graph.nodes.keys()}")

PIPELINE = PIPELINES[0]
concurrency_limiter = RequestLimiter(CONCURRENT_REQUEST_PER_WORKER)


@router.post("/query", response_model=Response)
def query(request: Request):
    logger.error("QUERY got request")
    with concurrency_limiter.run():
        pipeline_to_use = randrange(NUMBER_OF_PIPELINES)
        logger.error(f"QUERY Running on pipeline {pipeline_to_use}")
        MUTEXES[pipeline_to_use].acquire()
        try:
            logger.error(f'QUERY concurrency limit not reached, using pipeline: {pipeline_to_use}')
            result = _process_request(PIPELINES[pipeline_to_use], request)
            logger.error("QUERY request processed")
            return result
        finally:
            MUTEXES[pipeline_to_use].release()

I needed a quick fix because my client has some tests coming soon, but that’s really not a valid workaround.