Multi-namespace migrations with doctrine/migrations 3.0

Last week I've announced the first alpha release for doctrine/migrations 3.0. This new major release is the result of almost 6 months of work. Brings a completely refactored/rewritten internal structure and some interesting new features.

This post will focus on the ability to collect migrations from multiple folders/namespaces and to specify dependencies between them (this feature has been introduced with the 3.0 major release and was not available before).

Multi-namespace migrations

Let's suppose we have a multi-installation application(such as example Wordpress, Magento, Drupal or similar). All this applications have some kind of optional Modules/Plugins/Extensions.

Modules/Plugins/Extensions by definition are optional, and if a module defines migrations, then even the database structure will be different from installation to installation. When modules have dependencies to other modules, then the execution order depends on those dependencies and not only on the chronological order.

Consider the following application diagram:

            +--------+
            |  Core  |
            +---+----+
                |
         +------+------+
         |             |
         v             v
    +--------+    +--------+
    |Module A|    |Module B|
    +--------+    +---+----+
                      |
                      v
                  +--------+
                  |Module C|
                  +--------+

We can see that there is a Core application, then the modules Module A, Module B, and Module C. We can see also that Module C has a dependency on Module B.

If the Module C defines some migrations, they most likely will depend on structures defined in the Module B, thus they should be executed after all the migrations in ModuleB have been executed. On the other hand, since Module A does not depend on Module B, their execution order relative to each other is not important.

In previous versions of doctrine/migrations this case was not handled, let's see how this use case can be solved by the upcoming 3.0 release.

Use case

Here we are going to see an example of how multi-namespace migrations can be implemented using doctrine/migrations 3.0.

Directory layout

Our example application has the following directory layout:

├── config/
|   └── cli-config.php
├── src/
|   ├── Core/
|   |  └── Migrations/
|   |     └── Version182289181.php
|   ├── ModuleA/
|   |  └── Migrations/
|   |     └── Version182289181.php
|   ├── ModuleB/    
|   |  └── Migrations/
|   |      └── Version098766776.php
|   └── ModuleC/
|      └── Migrations/
|         └── Version987689789.php
├── vendor/
└── composer.json

• composer.json and vendor are the standard composer file and vendor directory, containing the list of packages we depend on and the source code. By default executable files are available in the vendor/bin directory, and doctrine/migrations offers the vendor/bin/doctrine-migrations executable command to manage database migrations.
• src contains our source code, in this case, the application is divided into 4 parts, the Core and 3 modules. (This is just an example directory layout, in your application you can use any other layout you prefer.)
• config/cli-config.php is loaded by doctrine/migrations to configure itself. For other ways to integrate doctrine/migrations, please take a look at the official documentation.

Note that this is just an example, your application can have a different layout that might depend on how modules are loaded, framework in use and many other factors. The important thing is that the configurations defined in the config/cli-config.php file are able to discover the migration classes.

Configurations

This is the content of our cli-config.php:

use Doctrine\DBAL\DriverManager;
use Doctrine\Migrations\Configuration\Connection\ExistingConnection;
use Doctrine\Migrations\Configuration\Migration\ConfigurationArray;
use Doctrine\Migrations\DependencyFactory;
use Doctrine\Migrations\Version\Comparator;
use App\Core\ProjectVersionComparator;

// setup database connection
$conn = DriverManager::getConnection(['driver' => 'pdo_sqlite', 'memory' => true]);

// define configurations
$config = new ConfigurationArray([
    'migrations_paths'      => [
        'App\Core\Migrations' => 'src/Core/Migrations',
        'App\ModuleA\Migrations' => 'src/ModuleA/Migrations',
        'App\ModuleB\Migrations' => 'src/ModuleB/Migrations',
        'App\ModuleC\Migrations' => 'src/ModuleC/Migrations',
     ],
]);

$di = DependencyFactory::fromConnection($config, new ExistingConnection($conn));

// define custom migration sorting
$di->setService(Comparator::class, new ProjectVersionComparator());

return $di;

Analyzing line-by-line this file, we can see that:

• $conn is the connection to your database using the DBAL library.
• $config defines the doctrine migration configurations. The only mandatory parameter is migrations_paths that is a key/value array that tells to doctrine/migrations where to find the migration files for each module (keys are namespace prefixes and values are directory locations. Many other configuration options are explained in the official documentation.
• $di is the dependency factory that will be used by doctrine to retrieve connection, configurations and other dependencies.
• as last thing, calling $di->setService(Comparator::class, ...) we define a custom version comparator (App\Core\ProjectVersionComparator) that will have the responsibility of deciding the execution order for our migrations.

Version comparator

The version comparator decides the execution order and must implement the Doctrine\Migrations\Version\Comparator interface. The class implementing the interface has the responsibility to figure out the dependencies between migrations and sort the migrations accordingly.

Let's take a look at the custom version comparator:

namespace App\Core;

use Doctrine\Migrations\Version\Comparator;
use Doctrine\Migrations\Version\AlphabeticalComparator;
use MJS\TopSort\Implementations\ArraySort;

class ProjectVersionComparator implements Comparator
{
    private $dependencies;
    private $defaultSorter;

    public function __construct()
    {
        $this->defaultSorter = new AlphabeticalComparator();
        $this->dependencies = $this->buildDependencies();
    }

    private function buildDependencies(): array
    {        
        $sorter = new ArraySort();

        $sorter->add('App\Core');
        $sorter->add('App\ModuleA', ['App\Core']);
        $sorter->add('App\ModuleB', ['App\Core']);
        $sorter->add('App\ModuleC', ['App\ModuleB']);

        return array_flip($sorter->sort());
    }

    private function getNamespacePrefix(Version $version): string
    {
        if (preg_match('~^App\[^\]+~', (string)$version, $mch)) {
            return $mch[0];
        }       

        throw new Exception('Can not find the namespace prefix for the provide migration version.');
    }   

    public function compare(Version $a, Version $b): int
    {
        $prefixA = $this->getNamespacePrefix($a);
        $prefixB = $this->getNamespacePrefix($b);

        return $this->dependencies[$prefixA] <=> $this->dependencies[$prefixB] 
            ?: $this->defaultSorter->compare($a, $b);
    }
}

Analyzing method-by-method this class, we can see that:

__construct()

The constructor initializes two variables that will be used by the main compare() method.

• $this->defaultSorter: is an instance of new AlphabeticalComparator and initializes the default alphabetical doctrine migration sorter (will be used later as fallback sorting).
• $this->dependencies: is initialized by $this->buildDependencies() and is an array containing the sorted dependencies between modules

buildDependencies()

This method is the core of our dependency resolution strategy. Uses the marcj/topsort.php library to perform topological sorting and provide a sorted array of our dependency graph. Later that array will be used to sort migrations.

For each namespace, we define its dependent namespace. In our case we have:

• App\Core has no dependencies
• App\ModuleA depends on App\Core
• App\ModuleB depends on App\Core
• App\ModuleC depends on App\ModuleB (the dependency on App\Core is optional here because App\ModuleB depends already on it)

In the end, $this->dependencies will be an array having as keys the application namespaces and as value the execution order. To be more precise, will look like this:

[
 'App\Core' => 0,
 'App\ModuleA' => 1, 
 'App\ModuleB' => 2,
 'App\ModuleC' => 3,
]

Note that the hard-coded dependencies in the function buildDependencies() is just an example on how to detect relations between packages. In an ideal situation, modules could be independent composer packages, then a better way to resolve migration dependencies would be using the package relations defined in the composer.json file.

getNamespacePrefix()

This is just a utility method that is able to extract the namespace prefix from a doctrine/migrations version name. As an example, if the migration version is App\ModuleA\Migrations\Version6569787886, it will return App\ModuleA. It is used to have an easy lookup in the $this->dependencies array and locate the execution order of the migration based on the package to which it belongs.

compare()

This method does the real job of deciding which migration comes first and which comes later.

Let's see line-by-line what is happening:

• using getNamespacePrefix() we get the namespace prefixes for the two migration versions we are comparing

• using the space ship operator (<=>) we check the order of execution of the two namespace prefixes against the $this->dependencies array (initialized in the constructor using the buildDependencies() method)

  • if the two migrations belong to two packages that depend on each other, the <=> will return 1 or -1 depending on which comes first.
  • if the two migrations belong to the same package, the spaceship operator will return 0, so we fallback to the default doctrine alphabetical sorting.

Running migrations and other commands

Most of the other commands have the same output and input argument. A complete list of available commands can be found on the official documentation.

Empty migrations

bin/doctrine-migrations generate --namespace 'App\ModuleA\Migrations'

Will generate an empty migration in the src/Core/Migrations directory and having as namespace App\Core\Migrations. By omitting the --namespace argument, migrations will be generated in the first defined migrations_paths element.

An optional parameter --filter-expression will allow you to include in the diff only changes for a particular subset of your schema.

Diff migrations

bin/doctrine-migrations diff --namespace 'App\ModuleA\Migrations'

Will generate a migration in the src/Core/Migrations directory and having as namespace App\Core\Migrations. By omitting the --namespace argument, migrations will be generated in the first defined migrations_paths element and all the entities will be considered. (This command is available only if you are using doctine/orm)

Symfony integration

If you are using Symfony, doctrine/migrations 3.0 comes with doctrine/doctrine-migrations-bundle 3.0.

Because of how the integration is done we do not need the config/cli-config.php file to define connections, dependency factories or configurations. We also do not need the doctrine migrations executable file anymore since Symfony has already its own fully integrated bin/console command.

Symfony's bundle system allows us also to organize our core application and modules in bundles.

Most of the configurations are defined inside doctrine_migrations.yaml.

doctrine_migrations:
    migrations_paths:
        'App\Core\Migrations': '%kernel.project_dir%/Migrations'
        'MyCompany\ModuleA\Migrations': '%kernel.root_dir%/vendor/my-company/mod-a/src/Migrations'
        'MyCompany\ModuleB\Migrations': '%kernel.root_dir%/vendor/my-company/mod-b/src/Migrations'
        'MyCompany\ModuleC\Migrations': '%kernel.root_dir%/vendor/my-company/mod-c/src/Migrations'
    services: 
        'Doctrine\Migrations\Version\Comparator': 'App\Core\ProjectVersionComparator'

This configuration implies that:

• your Core module is in the src directory and the other modules are in the vendor directory installed as composer packages
• App\Core\ProjectVersionComparator is a service based on the App\Core\ProjectVersionComparator class we saw before (here more info on how to define Symfony services)

The doctrine/doctrine-migrations-bundle will auto-configure itself to use the default DBAL connection or the default ORM entity manager if available. There are many other configuration parameters explained in the official documentation.

Depending on which Symfony version you are using, this file could be located inside config/packages or its content has to be placed inside config/config.yml. If you are using Symfony Flex, a skeleton of this file will be auto-generated.

Extra: Custom migration factories (aka Decorating services)

Sometimes migrations need external services to get some data before running migrations. Use cases are fetching default data and inserting them into newly created tables, resolving IP addresses to countries if you are adding the "country" column to an existing user table, and many other situations.

This can be done by defining a custom factory on the doctrine_migrations.yaml file:

doctrine_migrations:
    # ...
    services: 
        'Doctrine\Migrations\Version\MigrationFactory': 'App\Core\ProjectVersionFactory'

services:
    App\Core\ProjectVersionFactory.inner:
      class: Doctrine\Migrations\Version\DbalMigrationFactory
      factory: ['@doctrine.migrations.dependency_factory', 'getMigrationFactory']

    App\Core\ProjectVersionFactory:
      arguments: ['@App\Core\ProjectVersionFactory.inner', '@service_container']

The App\Core\ProjectVersionFactory can look like this:

namespace App\Core;

use Doctrine\Migrations\AbstractMigration;
use Doctrine\Migrations\Version\MigrationFactory;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

class ProjectVersionFactory implements MigrationFactory
{
    private $container;
    private $migrationFactory;

    public function __construct(MigrationFactory $migrationFactory, ContainerInterface $container)
    {
        $this->container = $container;
        $this->migrationFactory = $migrationFactory;
    }

    public function createVersion(string $migrationClassName): AbstractMigration
    {        
        $instance = $this->migrationFactory->createVersion($migrationClassName);

        if ($instance instanceof ContainerAwareInterface) {
            $instance->setContainer($this->container);
        } 

        return $instance;
    }
}

Custom migration factories can be defined also in the project without Symfony by declaring them in the cli-config.php file.

// ...

$oldMigrationFactory = $di->getMigrationFactory();
$di->setService(MigrationFactory::class, new ProjectVersionFactory($oldMigrationFactory, $container));

return $di;

Extra: Auto registered migrations

This is a feature useful for bundle authors/maintainers and allows a bundle to "append" some migrations to the list of migrations to be executed by the core application. It is done using the Symfony's prepend extension interface.

namespace MyCompany\ModuleA\DependencyInjection;

use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Extension\PrependExtensionInterface;
use Symfony\Component\HttpKernel\DependencyInjection\Extension;

class MyCompanyModuleAExtension extends Extension implements PrependExtensionInterface
{
    // ...
    public function prepend(ContainerBuilder $container)
    {
        $container->prependExtensionConfig('doctrine_migrations', [
            'migrations_paths' => [
                'MyCompany\ModuleA\Migrations' => __DIR__. '/../Migrations',
            ]
        ]);
    }
}

Using the Symfony dependency injection and the prepend extension feature, ModuleA is able to auto-register its migrations into the Core application.

Of course, the sorting algorithm provided by the ProjectVersionComparator class must be able to sort accordingly the migrations provided by ModuleA.

Conclusion

That's it. With these configurations, doctrine/migrations is able to run migrations from multiple namespaces and the execution order will depend on the dependencies between the different modules.

What is next

doctrine/migrations 3.0 is still in alpha, to be able to deliver a good stable release it is important that you test the pre-release and share your feedback!

To try the alpha version, you can run:

composer require 'doctrine/migrations:^3.0@alpha'

You can have a look also to the upgrading document.

If you are using Symfony:

composer require 'doctrine/doctrine-migrations-bundle:^3.0@alpha' 'doctrine/migrations:^3.0@alpha'

You can have a look also to the upgrading document for the symfony bundle.