Türchen 19: Kickstart your Magento Dev System with Vagrant

Vagrant is a tool that allows portable virtual development environments, using virtual machines. If multiple team members work on different devices on a project, a unified environment can be ensured. It only has to be set up once and can be reproduced as often as needed.

But it is also interesting to single developers who work on multiple projects. Often the different production systems have different PHP versions installed or they need specific extensions and system configurations. Especially with PHP where much behaviour depends on global configuration, you often get errors due to different systems.

With Vagrant you can define the whole environment for each project and also put this definition under version control. Ideally this resembles the production system as close as possible. Also, your own system is less cluttered up with software that you needed for some past project.

Although Vagrant manages virtual machines, it is not a VM provider itself, it works with different providers, such as VirtualBox, VMware and even AWS.

Vagrant also works with many different provisioning tools, i.e. tools for automated system setup, like Puppet, but also comes with very simple built-in provisioners to copy files and run shell scripts. You can combine all of those.

To summarize the advantages of Vagrant:

  • Operating system and global configuration as close as possible to the production server
  • Reproducible environment
  • Versioned system configuration
  • Automated setting up of new environments

But isn't this, like, super-complicated?

You might be reluctant to learning new technologies (Vagrant, Puppet) and fear the amount of initial work to set everything up. Then I have good news: I already did most of it for you. In this article I introduce the /> integer_net Magento Vagrantbox that should get you started in no time, for new projects as well as for existing projects.

Some technical details beforehand

Terminology:

Box, Vagrantbox
The virtual machine, managed by Vagrant
Host
The computer that runs Vagrant
Provisioning
The automatic setup that gets executed on the first start and when requested explicitly, after configuration changes.

File synchronization

Somehow the project files from the host need to be available on the box. Vagrant offers different synchronization strategies, currently NFS, Rsync, SMB and VirtualBox Shared Folders. All of them have their pros and cons but because file system performance is essential for Magento development, the best choice is Rsync: files from the host are synchronized to the box in real time by a file system watcher process. This way you work on the native file system of the host but the web server uses the native file system of the VM.

Puppet and PuPHPet

For provisioning we use a combination of Puppet and shell scripts, built with the GUI configurator PuPHPet. This makes it easy for you to change server configuration without knowing the Puppet language, because it is all in one human readable YAML file
puphpet/config.yaml
which you can either edit manually or drag and drop into the configurator at http://puphpet.com to edit it. You will get a big zip file to download but all you need from it is the config.yaml to replace your original version. The PuPHPet configurator is pretty self-explanatory. If you want to know more, it has a good "About everything" page with some background, also about Vagrant.

VM Performance

Pay attention to the memory (RAM) you assign to your boxes, especially when you start multiple boxes at the same time, because it is subtracted from the host. I'd recommend to start with 512M for Magento and increase only when necessary. We use VirtualBox basically because it's the standard provider and it's entirely free, but honestly it's also the least performant one. Consider VMware if you need a performance boost. With PuPHPet you can change the provider with a single click.

Magento setup

Magento will be set up with shell scripts, that can be found in
puphpet/files/exec-once
and will run in alphabetical order on the first provisioning. When you add new scripts to the directory or change existing scripts, those will run on the next provisioning.

Let's set it up!

1. Install required Software

2. Download our project boilerplate

Download and extract https://github.com/schmengler/magento-boilerplate/archive/master.zip – this will be your project directory. Navigate to the directory and run
./up –
You will be asked for a domain and IP address. Try „webguys2014.local“ and 192.168.56.111. Add this combination to your systems
/etc/hosts
file. You can also choose between different project structure templates. Currently these are:
  • Modman+Composer: choose this if you manage your source with modman to separate it from core code. Using Composer for external dependencies is also supported but optional.
  • All-in-one: choose this if all your source code lives under the Magento web root. You will be asked for a directory for your sources, which must be a subdirectory of the project directory. Choose „src“ if you are not sure.
Now you have the option to continue immediately, this will set up a new Magento installation with some default settings. But let's wait with that for a minute to see what needs to be adjusted most of the time.

EITHER 3a. Choose parameters for new installation

If you used the Composer+Modman template, Magento gets installed with Composer by default. You can change the version in composer.json. You can also add or remove extensions there to your liking. If you want to install the sample data as well, look into
puphpet/files/exec-once/90-magento.sh
and adjust the n98-magerun install command (change
--installSampleData=no
to
--installSampleData=yes
). If you used the All-in-one template, you can change the Magento version and choose to install the sample data in
<span style="font-weight: normal">puphpet/files/exec-once/70-install-magento.sh</span>
.

OR 3b. Add existing source code

If you used the Composer+Modman template, first replace the composer.json file with your own or remove it if you don't use Composer. If you use Composer, make sure, you require „aoepeople/composer-installers“ - the box is configured to work with „Fabrizio's Lightweight Approach“ (see http://fbrnc.net/blog/2014/11/how-to-install-a-magento-module). The following composer configuration is required for the composer installers to work:
"extra":{<br>
       "installer-paths": {<br>            "www/"
: [ "type:magento-source" ],<br>           
".modman/{$name}/" : [ "type:magento-module" ]<br>
       }<br>    },<br>
And this one is recommended to automatically trigger modman deployment:
"scripts":{<br>
       "post-install-cmd": "cd /home/vagrant &&
modman deploy-all --force",<br>        "post-update-cmd":
"cd /home/vagrant && modman deploy-all --force"<br>
   }
If you reference private repositories in composer.json you may copy your private SSH key to puphpet/files/dot/.ssh – the file name must end in _id_rsa, then it will be automatically registered with the SSH agent on the box. Then add your own modules under
src
and import them in
src/modman
like this:
@import MyModule
Note that absolutely no custom code will be in the www directory, only the untouched Magento core (the directory is not synchronized to the box, just sits there for browsing and debugging). If parts of your code are not deployed with modman, you should still make it a module under
src
, for example as
src/Base
with a modman file like this:
@shell rsync -avz --update --omit-dir-times
--no-o --no-p --no-g --exclude="modman" $MODULE/. $PROJECT
If you used the All-in-one template, add the code to the directory you specified before. If you want or have to use a separate repository, you can do this with a Git submodule as well. To import an existing database, place a compressed dump (
*.sql.gz
) into
systemstorage/live/database/
- specific Magento settings for the development system may be configured in
puphpet/files/exec-once/91-magento-config.sh
– the default base URL and a few other default settings are already there, extend it as needed.

4. Provisioning...

Now that everything is prepared, you can run the „up“ script again (or press „y“ if it's still waiting for your input) … and wait … The first start is going to take quite a while, so go grab a coffee or read the rest of the article while waiting. Don't worry about single lines of red messages scrolling by, they are usually no problem. Only if you see whole pages of red, a closer look might be required. When everything was successful, you will be greeted with this menu: You will see the rsync watcher working and have some shortcuts available:
  • L: Login to the box with SSH (same as „
    vagrant
        ssh
    “)
  • M: Login to the box with a direct call of „
    n98-magerun
        shell
    “. Useful if you only need magerun commands (like „cache:flush“). Exit the console with Ctrl-D.
  • H: Shut down the box (same as „
    vagrant
        halt
    “, but also kills the rsync watcher first)
  • S: Suspend the box (same as „
    vagrant
        suspend
    “, again with killing the rsync watcher first)
  • R: Restart the rsync watcher process (it sometimes crashes, for example after the host has been in power-saving mode)
  • P: Provision the box while it runs, for example after a change in
    config.yaml
    or if there are new/changed files in
    puphpet/files/exec-once
    .
But nothing of should be necessary right now, just leave the window open and...

5. Fire up your browser

Browse to http://webguys2014.local/. If you are greeted with the Magento installation wizard, Magentos
app/etc/local.xml
file is incomplete. Now you have two options:
  1. Follow the installation wizard (DB access is root/root)
  2. Adjust
    etc/local.xml
    – this file is linked directly into the Magento installation, so that you can put it into version control. But the existing file is incomplete, you need to add install date and encryption key. It makes sense to copy these from an existing system. Then you need to flush the Magento cache (
    n98-magerun cache:flush
    in the box) and open the store again.
Voilà: Access the backend at http://webguys2014.local/admin with username "admin" and password "test123" At http://webguys2014.local:1080/ you will find the Mailcatcher web interface. All outgoing mails are captured there (instead of really sending them)

Configuration / usage

The „up“ script

After the initial configuration, you can still use "up" as a convenience script to start the box and access some common commands with a single keystroke from the startup menu. But if you prefer, you can use the Vagrant commands directly as well:
  • vagrant up
    to start a box
  • vagrant rsync-auto
    to start the rsync watcher
  • vagrant ssh
    to log in via SSH
  • vagrant provision
    to provision a running box
  • vagrant halt
    to shutdown a running box

SSH connection from other tools (IDE, MySQL client, ...)

You can run
vagrant ssh-config
to see get information on how to access the box. It will show you something like:
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile C:/Users/fs/Projekte/intern/webguys2014/puphpet/files/dot/ssh/id_rsa
  IdentityFile C:/Users/fs/Projekte/intern/webguys2014/puphpet/files/dot/ssh/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL
  ForwardAgent yes
Vagrant uses port forwarding from
127.0.0.1:2222
to port
22
of the box. The thing is, this port will change, if it is already in use, like from another box. So I prefer using the IP of the box (e.g.
192.168.56.111
) with port
22
The username is always "vagrant" and the private key
puphpet/files/dot/ssh/insecure_private_key
.

Crontab

The file
etc/crontab
is installed as crontab on provisioning. This way the cron configuration can be versioned as well. The placeholder
{DOMAIN}
will be replaced with the content of
etc/domain
which was generated on the first setup we did before.

Magento configuration

Configuration of Magento that is specific to the development system should be done in
puphpet/files/exec-once/91-magento-config.sh
instead of manually in the backend so that it will be done automatically the next time somebody installs the box. Just add the necessary n98-magerun commands, for example to add a different base URL for a second store view:
n98-magerun config:set --scope="stores" --scope-id=2 web/unsecure/base_url http://en.webguys2014.local/

Usage of the systemstorage directory

With
/vagrant/update_systemstorage.sh
(executed in the box) a snapshot of database and media directory gets saved in
systemstorage/devbox
. This way you also can put it into version control or share it with other developers. However, due to the big amount of binary data I would not recommend to use the project Git repository. Thanks to Fabrizio Branca for the original version in https://github.com/AOEpeople/Magento_SampleProject!

Dotfiles

User specific Unix configuration files, the so called dotfiles, are copied on provisioning from
puphpet/files/dot
to the home directory in the box. You can replace the existing default files with your own. Many developers maintain their personal dotfiles, so it might be sensible to exclude the directory from version control (i.e. add it to .gitignore).

Xdebug

Xdebug is configured with the default IDE key "xdebug". You can change this together with other Xdebug configuration in
puphpet/config.yaml
. To debug CLI scripts, you can use
xdebug
as an alias for
php
. In this case, the IDE key "xdebug" is hard coded. Examples:
  • xdebug -d memory_limit=1024M www/shell/indexer.php
  • xdebug vendor/phpunit/phpunit/composer/bin/phpunit --filter IntegerNet_Solr
Don't forget to setup path mapping in your IDE to use the debugger.

Further reading, contribution

Check out the Vagrant documentation to learn more about the possibilities of Vagrant. For the German speakers amongst you, check out my Vagrant blog series on integer-net.de that covers the topic in a bit more detail. And once again the magento-boilerplate Github repo and the PuPHPet configurator. Both are under active development and constantly improved. PuPHPet is on GitHub as well. Any questions, remarks, critic or discussion on the magento-boilerplate Magento Vagrantbox is very welcome. The box has been used as foundation for a number of projects by now and optimized every time. But there is still a lot of potential.


Ein Beitrag von Fabian Schmengler
Fabian's avatar

Fabian Schmengler is Magento developer at integer_net, based in Aachen, Germany. His focus lies in backend development, conceptual design and test automation. Since 2011 Fabian is a Magento Certified Developer and since 2014 a Magento Certified Solution Specialist. He also blogs at schmengler-se.de/blog (English and German), is active on StackExchange, Twitter and Github.

Alle Beiträge von Fabian

Kommentare
Austin Peterson am

Hi,

I am trying to use puphpet with an already existing magento site.

On first load magento didn't seem to load right, I was seeing the base theme kind of all messed up insetad of my theme. So I decided to add some apache modules in my yaml file. well these aren't installing when I provision and prevents apache from starting:

Starting httpd: httpd: Syntax error on line 36 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.d/curl.load: Cannot load modules/mod_curl.so into server: /etc/httpd/modules/mod_curl.so: cannot open shared object file: No such file or directory

do you have a configuration for an already existing magento site (mine is in a git repo. also the root of my site is not the index, root of my site is actually on folder up from site because we store large files outside of repo and alias them such as var, media images folder).

Claudio am

I have this problem on Windows 8.1 64 bit:

==> default: Info: Class[Apache::Service]: Scheduling refresh of Service[httpd] ==> default: Notice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 81 events ==> default: Info: Creating state file /var/lib/puppet/state/state.yaml ==> default: Notice: Finished catalog run in 559.06 seconds The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong.

How can I solve it?

Fabian Schmengler am

Hi Justin,

right, the file is generated after running "up" the first time, but you'll get the chance to edit it before the VM is actually built the first time. If you want to see the templates where it gets generated from, look in puphpet/templates/allinone and puphpet/templates/modman

Changes to php.ini configuration can be easily done in the config.yaml file.

In the modman/composer setup, you'll specify the Magento version in composer.json. The important bit there is the download link to a tar.gz archive, usually from magentocommerce.com

In the allinone setup, look into puphpet/files/exec-once/70-install-magento.sh and change this line: https://github.com/schmengler/magento-boilerplate/blob/master/puphpet/templates/allinone/to-copy/puphpet/files/exec-once/70-install-magento.sh#L6

Justin am

Thanks for this tutorial. I'm not seeing a YAML file in puphpet/config.yaml - Does it get created after running the up command?

Could you provide me with some details on how I can specify the Magento version to install? Also, is it easy to extend this box to add some edits to the php.ini file on install?

Hiren Modi am

very nice post with detailed information.. looking forward to read the next post on magento.

Retrospektive: Das Blog 2014 - schmengler-se.de am

[…] auf integer-net.de. Eine englische Zusammenfassung gab es im Webguys Adventskalender: Türchen 19: Kickstart your Magento Dev System with Vagrant. Zwei der vier Teaser in diesem Blog haben es in die Top Ten 2014 geschafft (siehe […]

Dein Kommentar