So, after extensive debugging last night @rvdb and I think we were finally able to solve the issue. Hurray 
First, I need to confess that we indeed were running a modified version were I had applied the ‘patch’ from OJS 3 behind reverse proxy - how to achieve? - #9 by hermann to set $allowProtocolRelative = true. Without it, OJS would create http URLs for Javascript and CSS files, hence the client received an ugly and not-working page due to mixed-content issues. That was the original situation when we posted that issue here.
Testing with the ‘official’ OJS Docker images showed the same result, i.e. http URLs for Javascript and CSS files behind our Nginx proxy.
@asmecher’s hint about the $_SERVER variable and a related typo3 issue (Bug #29693: Respect HTTP_X_FORWARDED_PROTO in SSL check - TYPO3 Core - TYPO3 Forge – seems they have been there as well) brought us on the track that the HTTPS key was probably missing from the $_SERVER variable. [Looking for it now I think the relevant part might be pkp-lib/PKPRequest.inc.php at c5f40d7d390d0aedb9a4f9063914652aa6e3dc6c · pkp/pkp-lib · GitHub]
Long story short: the remedy for us was to add SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on to the apache config which will properly inject that HTTPS key to the $_SERVER variable IFF the proxy sends the X-Forwarded-Proto header.