The Taming of the Project

If you could simplify one typical task or a part of the programming process, what would it be? My answer will always be the same: setting up the project. The start of every new project has the same boring steps. You prepare your docker file(s), run the containers, log in to it to create a database and run any necessary migrations and installation commands depending on your project. In time you may prepare bash scripts to automate part of the tasks or take up a tool like Ansible to help you manage the local infrastructure. No matter how hard you try, however, it always feels like it takes too long. The reasons for this may vary. Perhaps the process fails for some reason and you need to debug it. It may be that the project has a particular use case that your playbooks and scripts aren’t prepared for. The database could also be quite large and import just takes forever to complete.

Now imagine you work for not that big development house that has a dozen projects or so. A new person joins the project as their expertise is necessary to develop a new feature or to fix an elusive bug. The budget is tight, though. You don’t want to bill the customer for the time this person will need to set up the project locally. So what’s the next step?

Hatimeria Automation Toolkit, or HAT, is a small tool that we’ve been developing for some time in order to help us automate part of the typical setup and deployment processes. HAT is a Node-based application built on top of Shipit which itself is a port of Capistrano to Javascript. It can deploy projects to remote servers. It can create a local instance in a Docker container. It can log you to the system, whether it’s remote or local. It provides a way to adjust the deployment process to specifics of the project. It can execute typical tasks with a few keystrokes. What it cannot do itself is convincing people to give it a chance.

Complication, where problems unfold

But let’s back up a little bit. Why do we need a tool for those tasks in the first place? As it was mentioned before, setting up a project can be a tedious process. You need to check which version of the programming language to use, create a new database, import it and execute any number of installation scripts provided by the project… and this part only handles the local installation. Deploying applications to the remote server is usually more complex. You need to take into an account not only sending updated files to the server but also several additional tasks. Those tasks include: putting the server into the maintenance mode, making sure any background tasks won’t break during file update, running migrations, or generating assets (i.e. running grunt or gulp).

At Hatimeria, we work with ecommerce projects and a large chunk of them are using Magento 2. Every time you clone a project you need to install packages with the composer. The problem that may occur is that the client may have some extensions bought from Magento Marketplace. These are available only for the account that purchased it so unless they are free of charge you need a way to share access keys to the Magento composer repository.

Once it’s sorted out, another common problem you can face is that Magento databases can be quite big. The bigger they are the larger the catalogue of products is. As a result, it takes a considerable amount of time to import it. Additionally, the database usually contains several configuration entries that need to be changed to match your local environment. Those entries are fronted URL paths, admin account, payment gateway keys, etc. Therefore, to avoid importing the database every time, you can create a separate container for it.

*The final major issue that you probably will encounter is the version of the language your project is using. Back in the days, it was much easier to do the local development on your host machine. Installing any AMP stack gave you all the elements you needed. Today, however, things are a bit more complicated due to the platform differences which your dependencies require. In our case, Magento 2 can have quite a range of requirements when it comes to PHP itself. Magento 2.1.18, which was released in June 2019 and in 2020 can still be found in some projects, requires PHP no later than 7.1.x. The newest version 2.4.0 won’t run on PHP lower than 7.3.0. Relying on a single PHP version on your host system is no longer viable. You need a way to run projects on different versions without too much hassle of switching between them.

Climax that reveals the hero

How is HAT responding to those needs? It encompasses a wide range of cases that can streamline typical setup and deployment tasks. It provides a way to separate projects and allows them to run on different versions of dependencies.

Using Docker Compose resolves the issue of using different versions. You can have one project using the newest version and a legacy one that still runs on PHP 7.1.3. By separating services to containers you can easily avoid the need to import the database each time you want to upgrade the container image. It also allows you to add additional services (i.e. RabbitMQ, Elasticsearch) to the project with ease.

It can run additional tasks after the database import that will update configuration values and reset the admin password. It also automatically fires all the necessary commands to migrate the database, generate DI files or static assets. Although on your localhost not all of those tasks are necessary, they are also run during the deployment process.

Aside from the local installation, we use it mainly to deploy our projects. We devised a simple YAML files structure which describes the target environment and enables to deploy Node.js, Magento, WordPress, and Symfony applications. HAT executes not only standard Shipit tasks such as cloning the repository and sending the files to the server. It can also run several additional tasks. As an example, for Node projects it can run build scripts, for Magento, it runs dependency injection compilation and executes migration when necessary. Furthermore, we can also run Gulp or Grunt for WordPress and Magento 2 themes. Additionally, if the project has some very specific requirements, there is a way to inject custom logic into the process.

To maintain a high-security level, it’s worth using solutions such as HashiCorp Vault service to keep all the sensitive data. You can store composer authentication data in the Vault so HAT can use them to install composer packages. Thanks to such an approach, it is really easy for the new person to join the project and have it running on their machine within less than 30 minutes, depending on the database size.

For macOS systems, where docker is known to cause some performance issues, HAT can only share the part of the project that you are actively developing, leaving all other dependencies (such as Magento or Symfony core packages) not synchronised. In case you need to copy them, there are commands that quickly will copy your host file state to the container.

Conclusion, for better or worse

All in all, HAT is a nifty little tool that can be of much help. It is not ideal, it is not polished but it does the job. So if your heart is pure and good, it will allow you to speed up the menial tasks. But if you come at it as an entitled little dev who snarks when something doesn’t go smoothly then… or you know, I will let you see it for yourself.

We believe that HAT is mature enough to be released to the public. The aim is mostly to allow our partners to use the same deployment process as we do here at Hatimeria. We hope it can provide some support to other developers in their never-ending quest of creating and deploying projects. You are most welcome to install it using the npm repository Package code also contains a more detailed technical description of the tool and the suggested way to work with it. Feel free to let us know if you come across any problems.