The google directory sync crashes after an hour

[thekoushik]

We tried running the google directory sync via jackson npm lib(1.18.7) with node18 and 1gb ram and after an hour it crashed with heap out of memory error FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory. The only log it showed is Starting the sync process . Can you suggest how much memory required for syncing 5k users?

[deepakprabhakara]

Thanks for reporting this, we'll test this and get back to you.

[thekoushik]

Already tried the latest version. It only starts by logging Starting the sync process for 2 directories and after an hour, it exits without logging the sync complete log.
I've checked from the admin portal, only first 50 users can be visible(because the pagination always returns the first page) and no groups are synced.

[niwsa]

@thekoushik We have identified an issue while fetching the users from Google provider. Working on a fix, will update you soon.

[niwsa]

@thekoushik We have made a fix for the crash in release v1.23.0. The pagination view fixes (only the first fifty visible) will be released shortly once #2572 is merged.

[niwsa]

@thekoushik https://github.com/boxyhq/jackson/releases/tag/v1.23.1 is out. Let us know once you are able to test it.

[thekoushik]

@deepakprabhakara and @niwsa, thank you so much for releasing the fix. Now google directory sync finishes successfully and admin portal paginations are also working.

[niwsa]

Just a quick note on benchmarking. In our tests using the production build of next.js, the heap memory hit a max of ~100MB and the whole sync operation for 5k users took around 76 seconds.

[thekoushik]

@niwsa Thanks for the heads up. We are using 4gb because of the crash. We will try reducing it and test optimum memory limit.