I had a similar need. If you look at the OJS publishing workflow, you can see that the workflow can be thought of as having several phases (submission, reviewing, copy editing, production, and publishing). There is a bit of documentation about how to use OJS for only part of the workflow, but it’s pretty limited.
We (iacr.org) use a separate submission and group review process for two of our journals based on a modification of the HotCRP open source system. HotCRP is also used by ACM, Usenix, and IEEE. It is primarily intended for conferences, but can be used for a journal review system in which papers are submitted periodically by a deadline (e.g., quarterly) and all papers for that deadline are reviewed by a group of reviewers. As a rough measure of the complexity, HotCRP is about 160K lines of PHP, whereas OJS is 1.4M lines of PHP. In reality, HotCRP is quite complicated because it is written in a very terse style of code. We don’t take submissions through OJS - we take them through HotCRP. The output from HotCRP can be thought of as a set of authorizations for authors to upload their post-review versions of accepted papers.
The review process of HotCRP sounds quite a bit different from what you want, but this illustrates the value of making the steps of a publishing workflow into pluggable components. There is quite a bit of experimentation going on in the peer review part of publishing, so it would be nice to be able to extend OJS in this way. There are several other systems that implement a peer review system (e.g., easychair, CMT, openreview.net, etc). openreview.net is used for the NeurIPS conference, which received 12,343 submissions in 2023. As far as I know, only HotCRP is open source but you still might be able to use another review system as a black box.
The documentation suggests using the “quick submit” plugin to enter papers into OJS if you are only using it for publishing. We find that approach to be time-consuming since it involves a lot of data entry (it would be completely infeasible for NeurIPS). It seems theoretically possible that you could use the REST API to bypass the review process and create submissions based on the output of HotCRP. I have not used that API and it looks like it would be quite a bit of work. Apparently author agreement to copyright assignment is part of the submission process, but HotCRP does not support that and you would have to build something for that (we built it into our extended version of HotCRP).
I suspect that turning OJS into a more modular system where you can replace the reviewing or copy editing modules would take quite a bit of engineering effort. For our newest journal, we ended up building completely different alternative to OJS. This was motivated in part by our dependence on LaTeX for the copy editing and production, the difficulty of using OJS for only part of the publishing process, and our desire to support richer metadata about publications (e.g, multiple affiliations, more detail than just ROR for affiliations, funding support, unauthenticated ORCID, better support for mathematics in titles and abstracts, import of citations via BibTeX, etc). The publishing practices in computer science are quite a bit different from other disciplines, so I’m not sure how much demand there is for the features we wanted. OJS hits a sweet spot for a lot of academic publishing, but it wasn’t flexible enough for us and didn’t seem to be evolving fast enough. In fairness, they can only address so much with the engineering resources they have and they need to prioritize based on the demand.