Migrate is one of the most established modules in the Drupal ecosystem. So much so that with Drupal 8, a decision has been made to get some of its functionality ported and added to Drupal core. An important reason was that the traditional upgrade between major releases was replaced with a migration of Drupal 6 or 7 content and configuration to Drupal 8.
Not all the functionality of the Migrate module has been moved into core though. Rather, we have the basic framework within the core migrate module and the functionality serving the upgrade path from Drupal 6 and 7 within the core migrate_drupal module. What’s left can be found in a host of contrib modules. The most important is Migrate Tools which contains, among other things, drush commands and the UI for running migrations. Additionally, the Migrate Source CSV, Migrate Source XML and Migrate Source JSON modules provide plugins for the most used types of migration sources.
In this article we are going to look at how migration works in Drupal 8 by migrating some content into node entities. For simplicity, the data we play with resides in tables in the same database as our Drupal installation. If you want to see the final code, you can check it out in this repository.
Drupal 8 Migration Theory
The structure of a migration in Drupal 8 is made up of three main parts: the source, the process and the destination, all three being built using the new Drupal 8 plugin system. These three parts form a pipeline. The source plugin represents the raw data we are migrating and is responsible for delivering individual rows of it. This data is passed to one or more process plugins that perform any needed manipulation on each row field. Finally, once the process plugins finish preparing the data, the destination plugins save it into Drupal entities (either content or configuration).
Creating a migration involves using such plugins (by either extending or directly using core classes). All of these are then brought together into a special migration configuration entity. This is shipped as module config that gets imported when the module is first enabled or can be constructed using a template (a case we won’t be exploring today). The migration configuration entity provides references to the main plugins used + additional metadata about the migration.
Movie Migration
Let us now get down to it and write a couple of migrations. Our data model is the following: we have two tables in our database: movies and movies_genres. The former has an ID, name and description while the latter has an ID, name and movie ID that maps to a movie from the first table. For the sake of simplicity, this mapping is one on one to avoid the complication of a third table. Here is the MySQL script that you can use to create these tables and populate with a few test records (if you want to follow along):
CREATE TABLE `movies` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`description` text,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;
CREATE TABLE `movies_genres` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`movie_id` int(11) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;
INSERT INTO `movies` (`id`, `name`, `description`)
VALUES
(1, ‘Big Lebowsky’, ‘My favorite movie, hands down.’),
(2, ‘Pulp fiction’, ‘Or this is my favorite movie?’);
INSERT INTO `movies_genres` (`id`, `movie_id`, `name`)
VALUES
(1, 1, ‘Comedy’),
(2, 1, ‘Noir’),
(3, 2, ‘Crime’);
What we want to achieve is migrate the movies into basic Drupal Article nodes and the genres into taxonomy terms of the Tags vocabulary (which the Article nodes reference via a field). Naturally, we also want the migration to mirror the association between the movies and the genres.
Genres
Let us first take care of the genre migration because in Drupal, the movies will depend on them (they will reference them).
Inside the config/install folder of our module, we need the following configuration entity file called migrate.migration.genres.yml:
id: genres
label: Genres
migration_group: demo
source:
plugin: genres
key: default
destination:
plugin: entity:taxonomy_term
process:
vid:
plugin: default_value
default_value: tags
name: name
The first three elements of this configuration are the migration ID, label and group it belongs to. The following three keys are responsible for the three main parts of the migration pipeline mentioned earlier. Let’s talk about the latter three separately.
Source
Under the source key we specify which plugin the migration should use to represent and deliver the source data (in our case the plugin with the ID of genres). The key key is used to specify which database should our source query use (that is where our data is).
So in the Plugin/migrate/source folder of our module, let’s create our SQL based source plugin for our genres:
Genres.php
namespace DrupaldemoPluginmigratesource;
use DrupalmigratePluginmigratesourceSqlBase;
class Genres extends SqlBase {
public function query() {
$query = $this->select(‘movies_genres’, ‘g’)
->fields(‘g’, [‘id’, ‘movie_id’, ‘name’]);
return $query;
}
public function fields() {
$fields = [
‘id’ => $this->t(‘Genre ID’),
‘movie_id’ => $this->t(‘Movie ID’),
‘name’ => $this->t(‘Genre name’),
];
return $fields;
}
public function getIds() {
return [
‘id’ => [
‘type’ => ‘integer’,
‘alias’ => ‘g’,
],
];
}
}
Since we are using a SQL source, we need to have our own source plugin class to provide some information as to how the data needs to be read. It’s not enough to use an existing one from core. The query() method creates the query for the genre data, the fields() method defines each individual row field and the getIds() method specifies the source row field that acts as the unique ID. Nothing complicated happening here.
We are extending from SqlBase which, among other things, looks for the plugin configuration element named key to learn which database it should run the query on. In our case, the default one, as we detailed in the migration configuration entity.
And this is all we need for our simple source plugin.
Destination
For the migration destination, we use the default core taxonomy term destination plugin which handles everything for us. No need to specify anything more.
Process
Under the process key of the migration, we list each destination field we want populated and one or more process plugins that transform the source row fields into data for the destination fields. Since we want the genres to be all terms of the Tags vocabulary, for the vid field we use the default_value plugin which accepts a default_value key that indicates the value each record will have. Since all will be in the same vocabulary, this works well for us.
Lastly, for the term name field we can simply specify the source row field name without an explicit plugin name. This will, under the hood, use the get plugin that simply takes the data from the source and copies it over unaltered to the destination.
For more information on how you can chain multiple process plugins in the pipeline or what other such plugins you have available from core, I recommend you check out the documentation.
Movies
Now that our genres are importable, let’s take a look at the movies migration configuration that resides in the same folder as the previous (config/install):
migrate.migration.movies.yml
id: movies
label: Movies
migration_group: demo
source:
plugin: movies
key: default
destination:
plugin: entity:node
process:
type:
plugin: default_value
default_value: article
title: name
body: description
field_tags:
plugin: migration
migration: genres
source: genres
migration_dependencies:
required:
– genres
We notice the same metadata as before, the three main parts of the migration (source, process and destination) but also the explicit dependency which needs to be met before this migration can be successfully run.
Source
Like before, let’s take a look at the movies source plugin, located in the same place as the genres source plugin (Plugin/migrate/source ):
Movies.php:
namespace DrupaldemoPluginmigratesource;
use DrupalmigratePluginmigratesourceSqlBase;
use DrupalmigrateRow;
class Movies extends SqlBase {
public function query() {
$query = $this->select(‘movies’, ‘d’)
->fields(‘d’, [‘id’, ‘name’, ‘description’]);
return $query;
}
public function fields() {
$fields = [
‘id’ => $this->t(‘Movie ID’),
‘name’ => $this->t(‘Movie Name’),
‘description’ => $this->t(‘Movie Description’),
‘genres’ => $this->t(‘Movie Genres’),
];
return $fields;
}
public function getIds() {
return [
‘id’ => [
‘type’ => ‘integer’,
‘alias’ => ‘d’,
],
];
}
public function prepareRow(Row $row) {
$genres = $this->select(‘movies_genres’, ‘g’)
->fields(‘g’, [‘id’])
->condition(‘movie_id’, $row->getSourceProperty(‘id’))
->execute()
->fetchCol();
$row->setSourceProperty(‘genres’, $genres);
return parent::prepareRow($row);
}
}
We have the same three required methods as before, that do the same thing: query for and define the data. However, here we also use the prepareRow() method in order to alter the row data and available fields. The purpose is to select the ID of the movie genre that matches the current row (movie). That value is populated into a new source field called genres, which we will see in a minute how it’s used to save the Tags taxonomy term reference.
Destination
In this case, we use the node entity destination plugin and we need nothing more.
Process
There are four fields on the Article node we want populated with movie data. First, for the node type we use the same technique as before for the taxonomy vocabulary and set article to be the default value. Second and third, for the title and body fields we map the movie name and description source fields unaltered.
Lastly, for the tags field we use the migration process plugin that allows us to translate the ID of the genre (that we added earlier to the genres source row field) into the ID of its corresponding taxonomy term. This plugin does this for us by checking the migration mapping of the genres and reading these IDs. And this is why the genres migration is also marked as a dependency for the movies import.
Activating and Running the Migration
Now that we have our two migration configuration entities and all the relevant plugins, it’s time to enable our module for the first time and have the configuration imported by Drupal. If your module was already enabled, uninstall it and then enable it again. This will make the config import happen.
Additionally, in order to run the migrations via Drush (which is the recommended way of doing it), install the Migrate Tools module. Then all that’s left to do is to use the commands to migrate or rollback the movies and genres.
To see the available migrations and their status:
drush migrate-status
To import all migrations:
drush migrate-import –all
To roll all migrations back:
drush migrate-rollback –all
Conclusion
And there we have it – a simple migration to illustrate how we can now import, track and roll back migrations in Drupal 8. We’ve seen how the plugin system is used to represent all these different components of functionality, and how the migration definition has been turned into configuration that brings these elements together.
There is much more to learn, though. You can use different source plugins, such as for data in CSV or JSON, complex process plugins (or write your own), or even custom destination plugins for whatever data structure you may have. Good luck!
