The conversion from row to sqlalchemy model is what is consuming the memory. After profiling (would need to confirm), a single token seems to occupy about ~3.0 KB of memory.

A batch of 10k tokens would use about ~30-40MiB, which seems like a fair threshold amount

Investigation: We need to find the threshold

I feel like celery would be a good candidate for this kind of stuff. That said I don’t think running celery as a subprocess of wazo-auth is a good idea. Having celeryd running on its own make it easier to monitor and scale I guess.

We have this architecture issue in a lot of services. IMHO we need to have different processes: 1 to serve HTTP server and a new one for each external task

Launching everything in the service, remove the concept of microservice and doesn’t scale well

For the current issue, as said by about keeping all object in memory, we could scope deletion by batch instead of a query without limit. Because delete need to retrieve object to send event

My feeling is that the ORM is consuming all the memory creating objects to delete them.