A 504 error usually means that a proxy server between you and the OJS server timed out waiting for a response from OJS.
With nginx, it is probably not a separate proxy server, but nginx itself. Per that link, try increasing the timeouts to allow OJS more time for the upgrade.