Slow GET /users

Description

Listing users is very slow when many users are configured.

With 10k users doing a GET on /users with a pagination size of 25 takes takes 48s on my test VM

Zendesk Ticket IDs

None

Attachments

1
  • 30 Oct 2023, 07:08 PM

Activity

Show:

Pascal Cadotte October 30, 2023 at 7:22 PM

Places that use `GET /users` without pagination.

  • wazo-chatd

  • wazo-calld

  • wazo-upgrade upgrade scripts

Pascal Cadotte October 30, 2023 at 7:08 PM
Edited

Context:
In the past wazo-confd has been optimized to be fast to return many results when querying users. These changes made it possible to GET /users with over 5000 users with relationships to many resources. Using customized loading strategies resulted in lower memory usage and faster execution with large result set.

On the other hand, these changes also added a significant cost to queries with small result set made against a large dataset. For example a `GET /users?limit=1` is almost as long as an unpaginated query when using pagination.

For small result sets (< ~100 to 250 users depending on the spec of the server) doing a lazyload is significantly faster.

 

On this graphic where I did not add a scale because 2 servers will yield different numbers. We want to take advantage of lazy loading (blue line) when the result set will be small. Then we want to use the query that has been optimized for large dataset.

The blue dashed line is the threshold where we want to start using the optimized query. The line is thin on this graphic but in real life this line is not at the same position. For this task I used the value 101. This is probably lower than what it could be on large scale installation but then most use cases with pagination will use a number lower than 101 anyway.

In an ideal world pagination would be mandatory and the `GET /users` resource would not return as much stuff as it does making optimization attempts less of a compromise.

Done

Details

Priority

Assignee

Reporter

Approvers

Julien Alie

Pair

Julien Alie

Fix versions

Sprint

Zendesk Support

Created October 10, 2023 at 2:59 PM
Updated November 1, 2023 at 7:31 PM
Resolved October 31, 2023 at 6:46 PM