wazo-confd: out of memory and crash when listing call permissions on a huge stack

Description

When calling API to list call permissions on a stack with many (> 1000) users and groups

then wazo-confd runs out of memory and gets killed by system with OOM error.

 

Technical explanation:

The joins in the call permission listing query produce a cartesian product between users, groups and extensions. When a stack with thousands of users, groups and/or extensions are in a call permission, millions of rows are returned, crashing the service.

The fix:

A modification to the relationship loading strategy prevents creating this cartesian product and prevents

It is fixed by changing from joinedload to selectinload

While this fixes the issue with the cartesian product, the only side effect of this change is a higher number of queries sent to postgres for the call permission. This increase of queries should have no noticeable effect of postgres (cpu+ram)

Zendesk Ticket IDs

None

Activity

Show:

Sébastien Duthil January 10, 2024 at 2:04 PM
Edited

Regression introduced by this change in 23.10:

Done

Details

Priority

Assignee

Reporter

Approvers

Pascal Crausaz

Fix versions

Sprint

Story Points

Zendesk Support

Created January 5, 2024 at 4:41 PM
Updated January 10, 2024 at 7:17 PM
Resolved January 5, 2024 at 5:13 PM