After searching and searching we found that this trick solved our problem (but wasn’t needed in ojs 2.4.x)
By adding the following line: RewriteCond %{REQUEST_URI} !api/v1/ in the .htaccess file:
RewriteCond %{SERVER_NAME} ^.custom.domain.com
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !api/v1/
RewriteRule ^(.)$ index.php/custom/$1 [L]
Now it seems to work. Thanks for your help!