After much head scratching, our DevOps guru Mike came up with a strategy which would allow us to perform a swing migration - leaving the live environment untouched but allowing us to suck it down into a test environment with the same PHP version on the new server, and another environment with the new version of PHP, pointing at the same data.
The way we achieved this was through containerisation - in our case we use Docker containers.
Most of the resources we found online about using Mautic were related to spinning up a brand new Mautic instance, but in our case we had an active instance which had been in use for nearly two years, so this wasn’t really an option for us.
The scripts we created allow us to do the following:
- Set up the environments and ssh connections
- Spin up Docker containers for PHP, MySQL, Cron
- Suck down the current instance - files and database
- Dump the live database into a script
- Create a .env file as a ‘template’ for Docker
- Map the local filesystem into the Docker containers
- Map the scripts directory into the MySQL container
- Fire up the MySQL database container which imports the database scripts
- Run script to build Mautic, import database and configure settings
It’s probably helpful to start with an overview of the server architecture we are working with.
Our architecture is configured to leverage containerisation, using the Docker API extensively to build containers, update configuration files and keep our proxy (Nginx) up to date with what’s going on in our containers.
Our current setup is to have a container for each application - so in the case of Mautic we have the following containers:
- PHP 7.x
- Cut-down minimal Linux container (for running Cron tasks)
- Mailhog (mail server for internal testing)
We have a naming convention such that containers are given the name of the client, followed by the application, and then the script name.
This means that we can easily identify the purpose of each container and reduces the risk of modifying the wrong containers.
We host our file structure locally on the server hardware for live instances rather than containerising the file systems. We do things slightly differently for our demo instances which we will explore in a future blog post.
The first task is to create a directory for the application - in this case Mautic - within which we create three sub-folders: -
As a brief explanation of the structure:
This is the naming convention we use in the directory structure - the name of the client, followed by the application.
This will be where the filesystem for the application will reside
This is where the database filesystem for the application will reside
This is where we locate any scripts that are required.
Setting up the environments
We use virtualhosts to set up each domain, so these need to be set up within your instance as appropriate.
Downloading the current instance
Once the directories are created, we import the current instance into the folder web_data.
As an example, we would pull from: /var/www/html/mautic into: /viryagroup-mautic/web_data
Download a database dump
Once the files are imported, the next task is to import the database. We make a database dump into /viryagroup-mautic/web_data which will be used to resurrect the data once the containers are built.
Create .env file
Next, we create the .env file based on the configuration from the files downloaded from the live instance. The .env file includes all the configuration settings required to reinstate the database.
Map file system
Once the .env file is created, we map the filesystem directory for the virtual host, so that it points at the appropriate location. In this case, we would map: /var/www/html into: /viryagroup-mautic/web_data
Build required containers
At this point we need to build two Docker containers - as we need to spin up our Mautic instance in the current PHP version, run the update, and then check that it works OK with the latest PHP version.
In our case, we have the following containers:
At this point we were not doing any major version changes with MySQL, so we only need to build one MySQL container: viryagroup-mautic-MySQL
Mailhog is a great tool for testing, especially if you want to ensure no mail goes out from development areas or until you are ready. This is a simple one really - spin up the container and configure: viryagroup-mautic-mailhog
As Mautic relies heavily on cron jobs, after researching the different ways to manage this in a containerised environment we decided to build a simple, cut-back Linux container to manage cron jobs on a per-account basis. We build it thus: viryagroup-mautic-cron
Once all the containers are set up and running, we can kick off with preparing the migration.
What we are wanting to achieve is:
- Database containing current live database PHP 5.3.x container pointed at viryagroup-mautic-web_data
- PHP 7.x container pointed at viryagroup-mautic-web_data
To kick things off, we map the MySQL container to viryagroup-mautic-db_data and kick-start the import by loading up the database dump within viryagroup-mautic-db_scripts.
This builds the database from the live dataset, which will be connected to by the PHP containers.
Our buildMautic.sh script then sets about building Mautic and running the various tasks that are needed.
We now have the ability to fire up Mautic using PHP 5.3, and PHP 7.x - using the same fileset and database.
What we need to do is:
- Check Mautic in the PHP 5.3 environment, ensure everything is as expected
- Run Mautic update to latest version at command line (as we were working with a large data set, so command line update is the recommended route)
- Check Mautic in the PHP 7.x environment, ensure everything is as expected
One of the huge benefits we found through using containerisation and the scripts we wrote was that when inevitably we hit some problems, within a few clicks we could nuke the instance, pull it down again, rebuild, and try again. All of this could be done without touching the live dataset.
Bumps we hit along the way
We encountered multiple issues with regards to zero-dated date/time fields which needed fixing before we could complete the migration - these were flagged up in logs when we tried to update. We also hit a few snags with foreign characters, fields with invalid data (a historical data import had thrown every job one person had held through their entire life as a pipe-separated list in the ‘role’ field! Oops!).
Once we had achieved a working installation on the latest version of PHP and Mautic, we were able to apply and test other improvements such as implementing SSL, before we pushed the Mautic instance live. This environment then became ‘live’ and we were able to use the same processes to clone down and keep in sync a development and staging area.
We hope this has been helpful for people who might need to be considering similar swing-migrations - if you need some help with the containerisation and scripts, drop us a line. Comments or questions? Leave a message below!