From fc6819ce00e974f9b8d316278ad1e5b09da65855 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 02:04:41 -0200 Subject: [PATCH 01/18] Import viewdocs-yeti layout --- docs/template.html | 130 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/template.html diff --git a/docs/template.html b/docs/template.html new file mode 100644 index 0000000..5cc5b97 --- /dev/null +++ b/docs/template.html @@ -0,0 +1,130 @@ + + + + {{NAME}} :: viewdocs.io + + + + + + + + + + + + +
+ + +
+ {{CONTENT}} +
+
+ + + + + + + + From eb544ae7730392bb57fdaa03c7d1de88d7779ac8 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 02:10:11 -0200 Subject: [PATCH 02/18] docs index --- docs/index.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..ddedf92 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,49 @@ +# vagrant-cachier + +A [Vagrant](http://www.vagrantup.com/) plugin that helps you reduce the amount of +coffee you drink while waiting for boxes to be provisioned by sharing a common +package cache among similiar VM instances. Kinda like [vagrant-apt_cache](https://github.com/avit/vagrant-apt_cache) +or [this magical snippet](http://gist.github.com/juanje/3797297) but targetting +multiple package managers and Linux distros. + + +## Installation + +Make sure you have Vagrant 1.2+ and run: + +``` +vagrant plugin install vagrant-cachier +``` + +## Quick start + +The easiest way to set things up is just to enable [cache buckets auto detection](config/auto-detection) +from within your `Vagrantfile`: + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'your-box' + config.cache.auto_detect = true + # If you are using VirtualBox, you might want to enable NFS for shared folders + # config.cache.enable_nfs = true +end +``` + +For more information please check out the links on the menu above. + + +## Compatible providers + +* Vagrant's built in VirtualBox provider +* [vagrant-lxc](https://github.com/fgrehm/vagrant-lxc) +* [VMware providers](http://www.vagrantup.com/vmware) with NFS enabled (See + [GH-24](https://github.com/fgrehm/vagrant-cachier/issues/24) for more info) + + +## Contributing + +1. Fork it +2. Create your feature branch (`git checkout -b my-new-feature`) +3. Commit your changes (`git commit -am 'Add some feature'`) +4. Push to the branch (`git push origin my-new-feature`) +5. Create new Pull Request From 94de55a2413eb24d51491f94bd8942f5e6ea5b9b Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 02:42:49 -0200 Subject: [PATCH 03/18] docs: How does it work? --- docs/how-does-it-work.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 docs/how-does-it-work.md diff --git a/docs/how-does-it-work.md b/docs/how-does-it-work.md new file mode 100644 index 0000000..80a8f5d --- /dev/null +++ b/docs/how-does-it-work.md @@ -0,0 +1,32 @@ +## How does it work? + +On vagrant-cachier's _"jargon"_, cached packages are kept in _cache buckets_. +Those _buckets_ are basically directories that are shared between the host machine +and VMs that are kept around between `vagrant destroy`s. Each _bucket_ (or synced +folder if you prefer) is meant to cache specific types of packages, like [apt](buckets/apt)'s +`.deb`s or [RubyGems](buckets/rubygems) `.gem`s. Please have a look at the +"Available Buckets" menu above for more information on each bucket. + +Regarding configurations, right now the plugin does not make any assumptions for +you and you have to configure things properly from your `Vagrantfile`. In other +words, _the plugin is disabled by default_. + +Cache buckets will always be available from `/tmp/vagrant-cachier` on your guest and +the appropriate folders will get symlinked to the right path _after_ the machine is +up but _right before_ it gets provisioned. We _could_ potentially do it on one go +and share bucket's folders directly to the right path if we were only using VirtualBox +because it shares folders _after_ booting the machine but the LXC provider does that +_as part of_ the boot process (synced folders are actually `lxc-start` parameters) +and as of now we are not able to get some information that this plugin requires +about the guest machine / container before it is actually up and running. + +Under the hood, the plugin will monkey patch `Vagrant::Builtin::Provision` and +will set things up for each configured cache bucket _before and after_ running each +defined provisioner. Before halting the machine, it will also revert the changes +required to set things up by hooking into calls to `Vagrant::Builtin::GracefulHalt` +so that you can repackage the machine for others to use without requiring users to +install the plugin as well. + +Please keep in mind that this plugin won't do magic, if you are compiling things +during provisioning or manually downloading packages outside of a bucket you +won't see that much of improvement. From 2734186347d54f37484f1b4c0b3f3675cdd779f4 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:09:28 -0200 Subject: [PATCH 04/18] Benchmarks --- docs/benchmarks.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/benchmarks.md diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..c0fa86d --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,30 @@ +# Benchmarks + +During the early days of this plugin, [@fgrehm](https://github.com/fgrehm) wrote +a [blog post](http://fabiorehm.com/blog/2013/05/24/stop-wasting-bandwidth-with-vagrant-cachier#show_me_the_numbers) +with some benchmarks on the time that was cut down by using the plugin. If you +are interested on the numbers only, the projects tested were one of vagrant-lxc's +Ubuntu [dev box](https://github.com/fgrehm/vagrant-lxc/wiki/Development#using-virtualbox-for-development), +[rails-dev-box](https://github.com/rails/rails-dev-box), his own [rails-base-box](https://github.com/fgrehm/rails-base-box) +and Discourse's [dev box](https://github.com/discourse/discourse/blob/master/Vagrantfile) + +
+ +| | First provision | Second provision | Diff. | APT cache | +| --- | --- | --- | --- | --- | +| rails-dev-box | 4m45s | 3m20s | ~29% | 66mb | +| rails-base-box | 11m56s | 7m54s | ~34% | 77mb | +| vagrant-lxc | 10m16s | 5m9s | ~50% | 124mb | +| discourse | 1m41s | 49s | ~51% | 62mb | +
+_Please note that the tests were made on May 24th 2013 and nowadays they might +be a bit different_ + +
+Some people have shared their numbers on Twitter and had experienced even better +results: + +

Holy cow... If you dig Vagrant, and like time - you need Vagrant Cachier. 60% speed increase for me. https://t.co/225jRH7bDa @vagrantup

— Chris Rickard (@chrisrickard) November 12, 2013
+

vagrant-cachier saved 3:20 off my #vagrant #provisioning http://t.co/VzRRu1QEwL

— Joe Ferguson (@svpernova09) November 11, 2013
+

Tested vagrant-cachier. Saved 60% of vagrant up time installing 10 rpms with chef. Pretty awesome. Check it out! github.com/fgrehm/vagrant…

— Miguel. (@miguelcnf) June 9, 2013
+

vagrant-cachier took my vagrant spin up from 30 to 5 minutes and reduced my caffeine intake by 3 cups http://t.co/V0uYpr3U0y

— Russell Cardullo (@russellcardullo) June 7, 2013
From 94ba773ce6479203ae517ca083ce78626283498b Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:26:36 -0200 Subject: [PATCH 05/18] docs: usage --- docs/usage.md | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/usage.md diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..d90db94 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,86 @@ +# Usage + +## Configurations + +### Auto detect supported cache buckets + +This is the easiest way to get started with plugin. By adding the code below to +your `Vagrantfile` you can enable automatic detection of supported cache _buckets_: + +```ruby +Vagrant.configure("2") do |config| + # ... + config.cache.auto_detect = true +end +``` + +This will make vagrant-cachier do its best to find out what is supported on the +guest machine and will set buckets accordingly. + +### Cache scope + +By default downloaded packages will get stored on a folder scoped to base boxes +under your `$HOME/.vagrant.d/cache`. The idea is to leverage the cache by allowing +downloaded packages to be reused across projects. So, if your `Vagrantfile` has +something like: + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box' + config.cache.auto_detect = true +end +``` + +The cached files will be stored under `$HOME/.vagrant.d/cache/some-box`. + +If you are on a [multi VM environment](http://docs.vagrantup.com/v2/multi-machine/index.html), +there is a huge chance that you'll end up having issues by sharing the same bucket +across different machines. For example, if you `apt-get install` from two machines +at "almost the same time" you are probably going to hit a _"SystemError: Failed to +lock /var/cache/apt/archives/lock"_. To work around that, you can set the scope +to be based on machines: + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box' + config.cache.scope = :machine +end +``` + +This will tell vagrant-cachier to download packages to `.vagrant/machines//cache` +on your current project directory. + + +## Finding out disk space used by buckets + +At some point there'll be a `vagrant cache stats` command that will give you that +information, but while that does not get implemented you can run the code below +if you are on a Linux machine: + +``` +# scope = :box (default) +$ du -h -d0 $HOME/.vagrant.d/cache +405M /home/fabio/.vagrant.d/cache/precise64 +1.1G /home/fabio/.vagrant.d/cache/raring64 +448M /home/fabio/.vagrant.d/cache/quantal64 + +# scope = :machine +$ du -h -d0 .vagrant/machines/*/cache +16K .vagrant/machines/precise/cache +90M .vagrant/machines/quantal/cache +210M .vagrant/machines/raring/cache +``` + +## Cleaning up cache buckets + +At some point there'll be a `vagrant cache clean [bucket-name]` command that will +take care of things for you, but while that does not get implemented you can run +the code below if you are on a Linux machine: + +``` +# scope = :box (default) +$ rm -rf $HOME/.vagrant.d/cache// + +# scope = :machine +$ rm -rf .vagrant/cache// +``` From 00f4aaa28ca8c12824cd10e6bdfce26001a9f634 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:43:37 -0200 Subject: [PATCH 06/18] docs for buckets --- docs/buckets/apt-cacher.md | 29 +++++++++++++++++++++++++++++ docs/buckets/apt.md | 14 ++++++++++++++ docs/buckets/chef.md | 11 +++++++++++ docs/buckets/composer.md | 11 +++++++++++ docs/buckets/npm.md | 16 ++++++++++++++++ docs/buckets/pacman.md | 10 ++++++++++ docs/buckets/rubygems.md | 14 ++++++++++++++ docs/buckets/rvm.md | 14 ++++++++++++++ docs/buckets/yum.md | 12 ++++++++++++ docs/buckets/zypper.md | 12 ++++++++++++ docs/template.html | 5 +++++ 11 files changed, 148 insertions(+) create mode 100644 docs/buckets/apt-cacher.md create mode 100644 docs/buckets/apt.md create mode 100644 docs/buckets/chef.md create mode 100644 docs/buckets/composer.md create mode 100644 docs/buckets/npm.md create mode 100644 docs/buckets/pacman.md create mode 100644 docs/buckets/rubygems.md create mode 100644 docs/buckets/rvm.md create mode 100644 docs/buckets/yum.md create mode 100644 docs/buckets/zypper.md diff --git a/docs/buckets/apt-cacher.md b/docs/buckets/apt-cacher.md new file mode 100644 index 0000000..440b2e4 --- /dev/null +++ b/docs/buckets/apt-cacher.md @@ -0,0 +1,29 @@ +# APT-CACHER + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-debian-box' + config.cache.enable :apt_cacher +end +``` + +This is useful, if you are using containers inside your VMs, e.g VirtualBox -> LXC. +This would allow you to reuse packages without sharing folder inside VirtualBox. Only +works with NFS-shared folders (since `vboxsf` is enforcing `vagrant`-user and `apt-cacher` +is running under `apt-cacher-ng` user) + + # install apt-cacher on (Host)-VM + $ sudo apt-get install apt-cacher-ng + + # get the IP for eth0 interface + $ ifconfig eth0 |grep "inet addr"|awk '{print $2}' |cut -c6-20 + + # configure mirror on for your docker/LXC instances: + $ echo 'Acquire::http { Proxy "http://X.X.X.X:3142"; };' > /etc/apt/apt.conf.d/10mirror + + # check, if working by tailing log on (Host)-VM, while installing packages on (Guest)-VMs + $ tail -f /var/log/apt-cacher-ng/apt-cacher.log + + +Used by Debian-like Linux distros, will get configured under guest's `/var/cache/apt-cacher-ng`. + diff --git a/docs/buckets/apt.md b/docs/buckets/apt.md new file mode 100644 index 0000000..780892b --- /dev/null +++ b/docs/buckets/apt.md @@ -0,0 +1,14 @@ +# APT + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-debian-box' + config.cache.enable :apt +end +``` + +Used by Debian-like Linux distros, will get configured under guest's `/var/cache/apt/archives`. + +_Please note that to avoid re-downloading packages, you should avoid `apt-get clean` +as much as possible in order to make a better use of the cache, even if you are +packaging a box_ diff --git a/docs/buckets/chef.md b/docs/buckets/chef.md new file mode 100644 index 0000000..2ead6e9 --- /dev/null +++ b/docs/buckets/chef.md @@ -0,0 +1,11 @@ +# Chef + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box-using-chef-provisioner' + config.cache.enable :chef +end +``` + +When a Chef provisioner is detected, this bucket caches the default +`file_cache_path` directory, `/var/chef/cache`. Requires Vagrant 1.2.4+. diff --git a/docs/buckets/composer.md b/docs/buckets/composer.md new file mode 100644 index 0000000..33f229d --- /dev/null +++ b/docs/buckets/composer.md @@ -0,0 +1,11 @@ +# [Composer](http://getcomposer.org/) + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box-with-php-installed' + config.cache.enable :composer +end +``` + +Compatible with probably any type of linux guest distro, will cache guests' +`$HOME/.composer` if PHP is detected. diff --git a/docs/buckets/npm.md b/docs/buckets/npm.md new file mode 100644 index 0000000..fadca00 --- /dev/null +++ b/docs/buckets/npm.md @@ -0,0 +1,16 @@ +# [npm](https://npmjs.org/) + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box-with-nodejs-installed' + config.cache.enable :npm +end +``` + +Compatible with probably any type of linux guest distro, will hook into npm's +cache directory under the result of running `npm config get cache` as +the default SSH user (usually `vagrant`) on your guest. + +If you use [nvm](https://github.com/creationix/nvm) / [n](https://github.com/visionmedia/n) +on the guest machine, make sure it is already installed before enabling +the bucket, otherwise you won't benefit from this plugin. diff --git a/docs/buckets/pacman.md b/docs/buckets/pacman.md new file mode 100644 index 0000000..6393d63 --- /dev/null +++ b/docs/buckets/pacman.md @@ -0,0 +1,10 @@ +# Pacman + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-arch-linux-box' + config.cache.enable :pacman +end +``` + +Used by Arch Linux, will get configured under guest's `/var/cache/pacman/pkg`. diff --git a/docs/buckets/rubygems.md b/docs/buckets/rubygems.md new file mode 100644 index 0000000..7f58cb4 --- /dev/null +++ b/docs/buckets/rubygems.md @@ -0,0 +1,14 @@ +# RubyGems + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box-with-ruby-installed' + config.cache.enable :gem +end +``` + +Compatible with probably with any type of guest distro, will hook into the `cache` +folder under the result of running `gem env gemdir` as the default SSH user (usualy +`vagrant`) on your guest. If you use rbenv / rvm on the guest machine, make sure +it is already installed before enabling the bucket, otherwise you won't benefit +from this plugin. diff --git a/docs/buckets/rvm.md b/docs/buckets/rvm.md new file mode 100644 index 0000000..e244309 --- /dev/null +++ b/docs/buckets/rvm.md @@ -0,0 +1,14 @@ +# [rvm](https://rvm.io/) + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-box-with-rvm-installed' + config.cache.enable :gem +end +``` + +Compatible with probably with any type of linux guest distro, will hook into the +`cache` folder under the result of running rvm info as the default SSH user (usualy +`vagrant`) on your guest. If you use rvm on the guest machine, make sure it is +already installed before enabling the bucket, otherwise you won't benefit from +this plugin. diff --git a/docs/buckets/yum.md b/docs/buckets/yum.md new file mode 100644 index 0000000..4b3182b --- /dev/null +++ b/docs/buckets/yum.md @@ -0,0 +1,12 @@ +# Yum + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-centos-box' + config.cache.enable :yum +end +``` + +Used by CentOS guests, will get configured under guest's `/var/cache/yum`. It will +also [make sure](lib/vagrant-cachier/bucket/yum.rb#L20) that `keepcache` is set to +`1` on guest's `/etc/yum.conf`. diff --git a/docs/buckets/zypper.md b/docs/buckets/zypper.md new file mode 100644 index 0000000..7c8131f --- /dev/null +++ b/docs/buckets/zypper.md @@ -0,0 +1,12 @@ +# Zypper + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = 'some-suse-box' + config.cache.enable :zypper +end +``` + +Used by SuSE guests, will get configured under guest's `/var/cache/zypp/packages`. It will +also [make sure](lib/vagrant-cachier/bucket/zypper.rb#L20) that `keep-packages` is enabled +for all repositories. diff --git a/docs/template.html b/docs/template.html index 5cc5b97..6c8ee0e 100644 --- a/docs/template.html +++ b/docs/template.html @@ -44,6 +44,11 @@ .navbar-nav>li>iframe { margin-top: 12px; } + + table { + margin-left: auto; + margin-right: auto; + } From 03b2b801219d3ba6c9edb851fff504777f911145 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:50:47 -0200 Subject: [PATCH 07/18] docs on development --- docs/development.md | 47 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/development.md diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..9fb5e15 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,47 @@ +# Development + +## Installing from sources + +If you want to install the plugin from sources: + +```bash +git clone https://github.com/fgrehm/vagrant-cachier.git +cd vagrant-cachier +bundle install +bundle exec rake build +vagrant plugin install pkg/vagrant-cachier-VERSION.gem +``` + + +## Sanity checks + +There are also some [Bats](https://github.com/sstephenson/bats) tests that basically +acts as a [sanity check](https://github.com/fgrehm/vagrant-cachier/blob/master/spec/acceptance/sanity_check.bats) +that you can run with `bats spec/acceptance` in case you are planning to submit a +Pull Request :) Just keep in mind that it might take a while to run if you are +using the default VirtualBox provider. + + +## How to create a new bucket? + +The concept of a cache _bucket_ is pretty simple to understand, we basically symlink a folder under the guest's `/tmp/vagrant-cache` into the folder where other tools keep downloaded packages. For example, in the case of the [apt bucket](https://github.com/fgrehm/vagrant-cachier#apt), we symlink `/var/cache/apt/archives` to `/tmp/vagrant-cache/apt` so that `.deb` packages downloaded with `apt-get install` can be reused when bringing machines up from scratch. + +If you want to see some other package manager supported, you'll first need to know if it has some sort of caching in place, where does it get stored and if it needs some extra configuration. For some managers that path will be fixed (like the canonical apt bucket), for others you might need to read some configuration by [running a command](https://github.com/fgrehm/vagrant-cachier/blob/master/lib/vagrant-cachier/cap/linux/rvm_path.rb#L10) on the guest VM (like [rvm](https://github.com/fgrehm/vagrant-cachier#rvm)) and you might even to [tweak some configs](https://github.com/fgrehm/vagrant-cachier/blob/master/lib/vagrant-cachier/bucket/yum.rb#L20) on the guest (like the [yum](https://github.com/fgrehm/vagrant-cachier#yum) bucket) + +There's currently a lot of duplication around the "installation" of cache buckets so there's plenty of source code for you to read in order to understand how things work under the hood but if you don't feel comfortable reading / writing Ruby code you can provide a high level overview of how to do things using plain old bash. + +For example, if you were to explain how to set up the rvm bucket, the script below should give vagrant-cachier maintainers an overview of how to set things up: + +```bash +# Check is rvm is installed +if ! $(rvm info > /dev/null); then + exit 0 +fi + +# If it is installed, read the cache dir +RVM_CACHE="${rvm_path}/archives" + +# "Install" the bucket! +mkdir -p /tmp/vagrant-cache/rvm/archives +ln -s /tmp/vagrant-cache/rvm/archives $RVM_CACHE +``` From 0340d6e76b7d36e940bded66a7aa9ac43bb9e2e9 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:51:10 -0200 Subject: [PATCH 08/18] docs: Adjust menu --- docs/template.html | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/docs/template.html b/docs/template.html index 6c8ee0e..8e7014f 100644 --- a/docs/template.html +++ b/docs/template.html @@ -71,35 +71,38 @@ From 3c4a20f44d324ff4abbd6c61da0176fa6193a9f5 Mon Sep 17 00:00:00 2001 From: Fabio Rehm Date: Sat, 7 Dec 2013 03:54:03 -0200 Subject: [PATCH 09/18] Hide github badges on smaller screens --- docs/template.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/template.html b/docs/template.html index 8e7014f..4f4f6c4 100644 --- a/docs/template.html +++ b/docs/template.html @@ -100,7 +100,7 @@ -