Gradually increasing CPU load on using sentence embeddings model with kmeans

Question

I am having a ML based production application, using flask, deployed on GCP server using gunicorn workers. In each incoming request, a text sentence is received.

It is using sentence transformers (All-MiniLM-L6-v2 model), which is loaded globally one time, to create embeddings of the incoming text and then use pre trained kmeans (also loaded globally) to predict/map it to a intent cluster. Basically, goal is to find intent of the sentence.

I have ample resources and the requests are also constant in number and texts are also similar, but still each day the CPU load is gradually increasing. Avg response time on 1st day was around 200 ms average, after 10 days now it is 400 ms.

I have tried deleting the embedding variable using 'del' command in the code itself, also forcing python garbage collector using 'gc.collect()' in a thread which executes after the main process execution is completed, but still the issue is coming.

One thing I have noticed is that if I dont use del and gc.collect(), the RAM starts to go down gradually. With both these, RAM is constant but now CPU usage is gradually going up day by day, hence the load and response time.

I have spent weeks on this issue trying to debug it but have got no solution, any help would be appreciated.

Answer 1

It sounds like your intent prediction function has an execution time that depends on the number of previous sentence classification requests.

You can debug the time spent by a single function with a profiler, for example the integrated Python profilers.

If it is not intended to have an increasingly slower function and you are actually just using pre-trained models and the classification of a single sentence should not affect the classification of future sentences, you could use serverless functions, since you don't need to keep any state between different request.