Decoupling the application for cloud-native development and deployment scenarios according to the Twelve Factors (12factor.net/)

Feature Requests
1

Describe the problem you would like to solve
The OJS application comes in many parts and each installation is an assemblage of many custom components. Its inherent design burdens application and system maintainers with having to deal with a high amount of mutable state and many, often synchronous, service integrations.

First published in 2011 ans subsequently iterated upon, the patterns published as The Twelve-Factor App have significantly improved software development practices in distributed systems, together with widespread availability of cgroups containers ultimately leading to the emergence of what would later be dubbed OCI Containers (Open Container Initiative) in 2013 and Kubernetes in 2014.

Next to using a single source of truth in version control for multiple deployments as its first principle, further dimensions of work for a distributed systems engineer are called out by the 12 factors, emphasis mine.

  1. Explicitly declare and isolate dependencies (II. Dependencies)
  2. Store config in the environment (III. Config)
  3. Treat backing services as attached resources (IV. Backing services)
  4. Strictly separate build and run stages (V. Build, release, run)
  5. Execute the app as one or more stateless processes (VI. Processes)
  6. Export services via port binding (VII. Port binding)
  7. Scale out via the process model (VIII. Concurrency)
  8. Maximize robustness with fast startup and graceful shutdown (IX. Disposability)
  9. Keep development, staging, and production as similar as possible (X. Dev/prod parity)
  10. Treat logs as event streams (XI. Logs)
  11. Run admin/management tasks as one-off processes (XII. Admin processes)

These parentalpractical advises, taken normatively, help to isolate side-effects in your application and allow to deeply integrate it into its execution environment, whether it be (a) a local shell, (b) a local container, (c) a remote deploy to an operating system or (d) a remote deploy into a container. They were originally formulated for PaaS environments by Heroku and have since been applied to and extended to containers.

Describe the solution you’d like
I would like to see an official stance of PKP according to the dimensions outlined by the 12 factors, ideally in some kind of roadmap that allows tracking the implementation status of each of the vectors. While some will probably already be accounted for, others will need deep reconsideration/refactoring of the systems to increase their robustness.

This activity spans the whole Software development lifecycle.

Who is asking for this feature?
This feature is needed by technical DevOps people engaged in developing and running PKP apps following SRE (Site Reliability Engineering) principles.

Additional information

  1. Dependency management with Composer through composer.json instead of git submodules had been discussed and validated in Managing PKP depencies with Composer - #8 by asmecher first.
  2. Storing the configuration in the environment is currently of concern to the development of the containers, but in the long term will require upstream support in pkp-lib, e.g. by using stock Laravel Configuration. Migrations will have to be considered here as well.
    1. https://github.com/pkp/containers/issues/2
      1. Please also find ideas about an early proof of concept in Consider implementing the phpdotenv usage to handle configuration · pkp/pkp-lib#8015 from 2021-05-19, which seems depreciated by the native Laravel support.
    2. Add PHP error page and application env config value · Issue #10027 · pkp/pkp-lib · GitHub
    3. Add a config option to set the database engine to InnoDB · Issue #7311 · pkp/pkp-lib · GitHub
    4. Wrong or missing config.inc.php variable breaks the upgrade · Issue #9781 · pkp/pkp-lib · GitHub
  3. When a DevOps engineer is concerned with handling mutable state of their applications, they prefer decoupling those into network services, such as databases for structured data and object storage for BLOBs, only falling back to file systems on block volumes in absence of the prior. This state is it then which is held by the backing services that offer interaction by their protocols. Sending emails would be another special case of this.
    Currently the application(s) store all state in the local file system and cannot decouple their sometimes large document repositories from the actual deployment of the application. If the document store would live in a file system abstraction, such as S3, the application process could be scheduled across separate compute instances. (Also see 6. Processes and 8. Concurrency) Some works allowing for such mode of operation have already begun, others are already considered.
    1. Add support for other file storage drivers (Flysystem) · Issue #11120 · pkp/pkp-lib · GitHub
      1. pkp-lib#9009 also suggests to consider file system migrations.
    2. As with the prior kind of mutable state, database migrations migrate the evolution in the schema of the application domain and also need to be considered. They are in reported progress of being migrated to Laravel’s Eloquent integration.
      1. Eloquent has a single and very specific open issue at Port Genre and GenreDAO to Use Eloquent Model · Issue #10133 · pkp/pkp-lib · GitHub . Would there be another one for tracking the overall migration from the current implementation over to it?
      2. Database: Seeding - Laravel 12.x - The PHP Framework For Web Artisans
      3. Database: Migrations - Laravel 12.x - The PHP Framework For Web Artisans
      4. Ideally, together with (3.), we can also provide the conventional pattern to supply access parameters for the database with a URL supplied in a DATABASE_URL-like variable.
  4. Dealing with build and run stages of the application as separate processes working from different source artifacts, here source code, there e.g. immutable containers, means to automate build and deploy more in CI to benefit from declarative configuration, isolation of side-effects. Standardising development and deployment patterns around OCI containers is one way to achieve this. Getting rid of git submodules through (2.) will help to alleviate this. Offering immutable and reusable build artifacts with declarative dependency management also improves security by providing observability into structured data of the dependency tree in aggregated composer.json files, eventually being used to construct a BOM. Signing git commits and (container) builds will further help to increase confidence in the robustness of the deployment artifacts.
  5. Processes are stateless in so far, as they don’t require a local file system and externalise all side effects (application state, configuration state) into (4.) backing services. This requires reading the configuration configuration without involving stateless artifacts, such as configuration files, directly from the environment (3.)
  6. This is made easier by declarative configuration (3.).
  7. Separate instances of the application can be run behind a load balancer in the (private) cloud, when at least (3.), (4.) and (6.) are met.
  8. This basically means that PID 1 in your container knows how to react to SIGHUP to avoid receiving a SIGKILL after 10 seconds.
  9. Declarative configuration of immutable and stateless deployment environments helps reproducibility and standardisation of their components. Strict adherence to principle (1.) is a sufficient condition for this.
  10. The log streams can then be fed into external log aggregation, correlation and analytics environments.
  11. This is often made easier with providing a CLI for the local server process and/or for the API. Task execution runners can temporarily help to standardise common actions in absence of the former (see just). For Laravel this is called the Artisan Console.
    1. General improvements to the CLI · pkp/pkp-lib · Discussion #7980
    2. Plugin compatibility with queue jobs on CLI mode · pkp/pkp-lib · Discussion #9556

When these things, and more, are into place, we can easily separate the side-effects of a distributed application and scale them independently. Decoupling application from state in backing services further allows to run stateless replicas that allow for high-availability and load balancing through horizontal scaling. Following conventional patterns seen elsewhere will also improve the Developer Experience (DevEx) and recognisability of components in the OJS ecosystem. Open Systems Design emphasises loose-coupling of components, to provide for resilience and easier manageable, locally bound state.

Other people have suggested to extend the list of namely 12 patterns to more dimensions, which could nowadays maybe read something like:

  1. Secrets management in distributed environments
  2. Unit testing of commits and end-to-end testing of deployments. Some might argue this to be part of (1.) already.
  3. Visibility and Observability through monitoring, logs and distributed tracing (OpenTelemetry), eventually as an extension to (11.).

next to probably other related SRE practices and Platform capabilities.

What comes to your mind when you think about making the PKP application platform more robust and resilient for the next decades to come?

—

PS: Links to open and ongoing issues related to the subjects above can be found in the Addendum: Identifying the 12 factors in the PKP application platform library and (OJS) container manifests - HedgeDoc

2

As a note: During research for prior art in this problem domain, there have also been signals about increasing stability of the application itself by introducing PHP type hints, which are now possible.

This is specific to how the ORM integrates the application model with the database and how language servers are able to assess structural errors in the AST ahead of runtime.

Moving towards more functional programming idioms then allows to apply programming patterns uncovered in other functional languages, such as Parse, don’t validate.

1 Like
3

Hi @jonr!

I agree in principle that these seem like good ideas, but that and a dollar won’t quite buy you a cup of coffee :slight_smile:

I can’t give you a timeline to our adoption of these, as our development is generally motivated by functional needs. We need to fit progressive renovation of the application in opportunistically, and at any given moment there will be several partial renovations going on at once. Some are easier and some are harder.

For example, right now we are partway through moving our old storage models over to Eloquent; adopting Laravel as a general toolset; adding type hinting; and much more. Each of these interrelates and has risks and rewards depending on when and where we disturb the codebase.

Therefore I’d suggest highlighting areas of specific difficulty in configuring OJS for containerization and working from there rather than starting from the high level principles and working down. It’s more likely we’ll be able to triage and incorporate changes that way.

(I’m also interested in the specific adoption of these ideas within the PHP and Laravel ecosystem. For example, PHP isn’t the best functional programming platform out there, so the principles will sometimes bump into the platform – but Laravel will likely have eased the way towards good containerization practices in a PHP- and community-friendly way; with that in mind, we can identify ways to kill two birds with one stone by moving closer to Laravel standards and tools, which is a long term goal on its own.)

Thanks,
Alec Smecher
Public Knowledge Project Team

4

Good afternoon Alec.

That’s very kind of you to propose, thank you. Yet this does not need to be a contradiction in itself, I suppose. The rationale of my narrative is guided by the active interaction with the parts of the system. What I’m trying to convey is that these “high-level principles”, as you call then, can be (and are) used (by others since more than a decade) as practical and empirical guidelines to reason about and to inform decisions on next steps and priorities for Software developments, leading to more resilient systems and accessible code bases. We share the same computational constraints of von-Neumann architecture and emergent behaviours in distributed systems when building for the ecosystems of the PKP application platform library and the applications OJS, OPS and OMP.

Once implemented, these changes don’t cost you a penny, so to say, but by avoiding the Not-Invented-Here syndrome rather have your platform benefit from stable and steady upstream developments, while needing less cognitive overhead to maintain.

Taken from the list above, current “areas of specific difficulty in configuring OJS for containerization” are:

  • Dependencies (II.), handling has security and reproducibility (X.) implications
  • Config (III.), applying it from an ephemeral outside that can always change and is interpreted at run-time, not as persistent state in a file on a file system on the disk
  • Backing services (IV.), treating state as something you want to lose, not something you want to keep
  • Build (V.), release, run, separation of build-time and run-time, esp. with regards to plugins
  • Processes (VI.), statelessness of; e.g. by preferring to run in PHP-FPM and treating OJS as a an ephemerally configurable application web server and not as an Apache2 instance with service-specific configuration.
  • Concurrency (VIII.), high-availability
  • Dev/prod parity (X.), maybe better reframed as “Reproducibility” nowadays
  • Logs (XI.), maybe better reframed as “Observability” nowadays, involving OpenTelemetry for logs, metrics and traces
  • Admin processes (XII.), leaning towards completeness and matching models in the web application UI, the REST API and the CLI

I don’t know anything worth mentioning about the domains VII or IX.

Areas where OJS/pkp-lib as upstream projects can help out with, are, in this priority:

  1. Discarding git submodules for “package management” and opting for declarative configuration with a chosen package manager; likely Composer (II.), actually the best next step, because already tested
  2. Fast-tracking with priority some of the fundamental Laravel adoptions
    • Configuration (III.)
    • Flysystem S3 support (IV.), actually the highest priority to tackle unbound growth of state in large instances
  3. Primarily treating plugins as immutable release artifacts during build time of immutable release artifacts of the applications
    • not as mutable state during run time
    • offering that as a last resort, but not recommended, conventional path of development

These are the most pressing issues and most likely “breaking changes” ahead of us, any time you like.

Moving on with support for:

  1. Model and database migrations with Laravel and Eloquent (IV.)
  2. Adopting a Containerfile and a compose.yml in the upstream repositories for a local development environment and inviting for cloud-native development models (declarative configuration, immutable artifacts, reproducibility; X.)
  3. Improving the common tasks in the pkp-lib CLI and the Laravel Artisan Console. (XII.)

will allow to have everything feel wholesome and accessible in the same time.

As a developer of a networked web application, I need to couple the components in my distributed system loosely, in order to manage side-effects cleanly (configuration state, file system state, database state, key-value store state, …), while accounting for eventual consistency within running networked applications and the consequences of the CAP theorem.

Thanks again for your suggestion to prioritise this together.

Mirroring the stance in a poignant way, does this mean for me as a systems and application developer that the functional needs of developing, building, installing, running, maintaining and debugging your systems cannot be recognised as legitimately motivated functional needs from a very existing stakeholder in your FLOSS community?

The idea with DevOps was to have “both sides” recognise each other (=in both directions) more intrinsically. Is that something that sounds useful, given the prior art of the last decade? Or more provocatively: “If not, how can PKP respond to the high maintenance cost and effort that people have in using its software due to postponing basic maintenance work on its codebases and carrying along a lot of technical debt for extended periods of time?”, said in a provocative way. I know about the realities of establishing a livelihood, but containers are not new, and these patterns are not new, neither. Or more humoristically: How come you managed to successfully navigate around those in the last years? Wouldn’t they have come up every now and then as maintenance items on their own?

I understand that you are trying to avoid risks and new cost centers. Please allow me to incur further and try to gain your trust in this stance, since I already have your ear.

Taking these steps will make it easier to run PKP Software for everyone, not just container users. Caring about the interaction surface of developers with the application means to recognise the levels of abstraction involved and dealing with each concept in its most efficient manner.

Esp. in resource constrained environments, with a limited throughput of changes per time, such as ours, it has been shown that explicit (=declarative) handling of non-changing (=immutable) Software pieces greatly decreases deployment times, and also brings support for reproducible development and ephemeral environments, e.g. locally and for review applications in CI. How come you don’t need this, given all the mutable state coupled so closely to your application?

While Ansible has often been compared with Docker, it is not “the same sauce”, as it still involves dealing with a lot of mutable state in the development and deployment of a role, while that is left to stateless application containers during run time, which are happy to externalise side-effects into separate systems, dealing with their eventually generic part of the application, building on known protocols (SQL, SMTP, S3, …). These questions don’t go away if we don’t look at them.

Yes, I am asking for changing the architecture of parts of the system.

No, I am not asking for becoming a purely functional implementation.

Instead, I’m kindly asking you to first recognise the dimensions outlined by the 12 factors as important maintenance priorities, because they are inherent conditions of the logic that is at work behind all software development and deployment. Then we can jump on each one step by step, one after another. Given the ordering of the outline above, it seems to me we would also already have a roadmap at hands :slight_smile:

Is there a public roadmap on these slow-paced, high-level epics? Any of the open GitHub projects at Projects · pkp · GitHub ?

I’m happy to write and file the tracking issues, in case they were missing, as long as you guide me where to put them.

Also this has me think of refactorability of the codebase. When you can move things around with confidence and good isolation of components, it means your separation of concerns is clean and abstractions don’t leak too much and a large part of the API surface is well covered, also by end-to-end integration tests. Maybe, eventually, introducing the specific interfaces of the Laravel Config, Eloquent, Flysystem, Logs and CLI adapters will ease you from a maintenance burden you never knew was holding you back, while robustly taking kare of parts of your application in a maintainable way.

Very much like the external dependencies (Operating system/Container processes + file systems, Database, Reverse Proxy/Load Balancer, DNS, Transactional email service, git Forge + CI, Container registry, Package registry, Queue) take kare of other parts. That’s the exciting promise of FLOSS, to have everyone take away maintenance burden from each other, by taking kare of defined and loosely-coupled components of the overall systems, sharing the outcomes and therefore reducing the cognitive load imposed on each.

Please let me restate: I’m not trying to turn the pkp-lib application platform or the OJS/OMP/OPS derivatives into purely-functional implementations. Those aspects were only mentioned (as a joke) aside, while you are completely right to pursue type hints, LSP support and more of those idioms.

If you so will, you can also read these principles as the digest of decades of experience of thousands of people, pressed into openly available Standards for everyone to learn from and to adapt to their needs. Pure FLOSS and community-oriented spirit lived like in not many other communities.

While the twelve factors may appear like a convoluted list of opinionated suggestions, in fact by their nature as design patterns used as guidelines in Software development, they rather become constraints and sufficient conditions to be working under the conditions of the CAP theorem and eventual consistency. From this perspective, these factors are not suggestions, but turn into requirements to meet when narrowing down the runtime conditions of our applications, again primarily with regards to the von-Neumann side effects Compute, Memory, Networking and Storage. This is what I meant with stating it doesn’t go away, because this demand is current, actual and pressing.

That said, they are made to challenge your conceptions and to add other perspectives to your repertoire. We’ll work towards the outlined course of interventions with workarounds from within pkp/containers to highlight certain concepts with small proofs of concepts. But some of these enquiries will require upstream adoption and from you to radically change some of your most accustomed to habits and conventions. I’m inviting you to take the chance and follow into this not-so-unknown territory.

Let me know when you feel ready, if all of this is strictly up to you alone. I’ll wait.

Else feel free to let me know who else is involved in these basic architectural decisions or implementations. I’m happy to make the case again in front of the closer or wider community as needed. Anyone else passing by this conversation is also invited to contribute experiences from their everyday. How do you like to run and maintain your Software?

Many thanks for all your attention until here and best of luck juggling all the different requirements from the many parties involved in this endeavour!

Hoping to have contributed a substantial and constructive critique and looking forward to diving further into the fields of software delivery, continuous integration and continuous deployment.

Kindest, Jon

1 Like