Ccache Updates

Some great news for ccache users such as myself.

First, the EPEL rpm of ccache was recently updated to 3.2.7 which is great news for users of CentOS-7 and similar distributions. For installation instructions and/or more information about ccache in general I recommend reading my Faster Re-Compiling with ccache post.

Second, ccache 3.3 was just released with substantial new features including support for caching of:

• cuda
• Fortran 77
• multiple arch “fat binaries”
• some assembly code (gcc -S)

This is of course in addition to its existing support of C, C++, Objective-C, and Objective-C++. While the vanilla code appears to have an issue (‘make test’ fails) under CentOS-7, I’m confident this will be resolved in due time (unfortunately I don’t personally have time to investigate further until next week).

Happy trails with your frequent rebuilds!

First Thoughts: Windows Subsystem for Linux – Bash on Ubuntu on Windows

For me the most exciting new feature in the Windows 10 Anniversary Update is the Windows Subsystem for Linux which underpins the Bash on Ubuntu on Windows. Other than the long and confusing branding, this is an exciting move by Microsoft and an impressive piece of technology. WSL allows native Linux ELF binaries to run on the Windows Kernel – let that sink in, not a virtual machine/Linux kernel, and not a recompile like Cygwin (which I have long used). WSL performs real time translation of the Linux systems calls to run on the Windows kernel, enabling practically no CPU or IO performance penalty. While this has been available to Windows Insiders for a few months, I have stuck with the release version of Windows for stability reasons – with today’s Anniversary Update this beta technology is now available to all desktop Windows 10 installations! Microsoft does caution that this particular feature is understandably beta and intended for command line developer usage, not running server software or X11 applications, which fortunately fits my intended usage perfectly.

Installation

Unless you are a Windows Insider, you will need the Aug 2nd Anniversary Update (Build 14393+) installed. I haven’t tested, but you may also need to be in Developer mode (found under Settings ->Update & Security -> For Developers).

Next install Windows Subsystem for Linux from Windows Features.

After the reboot open a regular Windows command prompt and type ‘bash’ and Windows will download and install Ubuntu on Windows. After being guided through creating a Linux user account you will be ready to try it out!

First Thoughts

A few interesting first thoughts after giving it a try.

1. Currently Ubuntu on Windows is based off Ubuntu Trusty Tahr 14.04.4 LTS. This is a venerable and well know, if slightly older distro. I’ve read that in the near future they will be moving to Ubuntu Xenial 16.04 LTS which will be a fresh and welcome update.
   
   
2. Uname reports the kernel as 3.4.0+, so while Microsoft is clear they don’t support all system calls at this time, they appear to be targeting a fairly recent kernel version.
   
   
   
   
   
3. After limited testing apt-get update and apt-get install appear to working normally. Here is a quick example installing iperf. Iperf is a perfect example of a tool you won’t want to run from an VM for IO performance reasons, so prior to this a Windows Binary or Cygwin version were the only options.
   
   

Overall I am very excited and look forward to continuing to explore WSL and Ubuntu on Windows. The only major feature omissions that I can see currently being a detraction for many developers will be the lack of an X11 client for remote Linux work. While I don’t normally make sure of them, for folks that prefer a X11 IDE or remote web browser for testing I can see it being a show stopper. Perhaps/hopefully this will be something coming in the near future as well.

For More Information

• Bash on Ubuntu on Windows
• Issues found within and using Bash on Ubuntu on Windows

Vagrant 1.8.5 and VirtualBox 5.1 Networking Performance

I’m not trying to make this a Vagrant/VirtualBox blog, but as they are currently tools I use almost continuously so new versions, features, and tips to increase their performance are obviously of great interest.

With Vagrant 1.8.5 released Monday and now supporting VirtualBox 5.1, I’ve had some time to conduct a few informal network performance tests. The good news is that network performance with the Virtio Paravirtulized driver appears to be even better, an informal test showed 471Mbits/s versus the previous 361Mbits/s of VirtualBox 5.0.24 – same hardware, both using Xenial.

Additionally, to enable Virtio I’ve also come across an even either method through Vagrant. Simply add the following like to your Vagrantfile inside the VirtualBox provider block:

vb.customize ["modifyvm", :id, "--nictype1", "virtio" ]

VirtualBox 5.1 released - but Vagrant 1.8.4 doesn't support it

UPDATE 7/19/2016: Vagrant 1.8.5 has been released with VirtualBox 5.1 support and other fixes (changelog).


VirtualBox 5.1 was released on Tuesday with several improvements/fixes that *I’m* eager to try out given I use a Windows Host with Linux Guests:

• The new APIC and I/O APIC improvements, which I hope will continue to increase network performance.
• Use of x2APIC for Linux guests and many other related fixes
• No longer relies on DKMS for Linux guest kernel module rebuilding – I haven’t had the opportunity yet to dig into exactly what this entails, but I am excited by the possibility as at least in RHEL/CentOS, DKMS not available by default, although it is in EPEL.
• The new NVMHCI-compatible storage controller

One major note though, Vagrant 1.8.4 does NOT support VirtualBox 5.1.  The good news is this should be coming soon as support was committed to the Vagrant master source on Wednesday. Let’s hope for the next Vagrant release soon!

Virtio Para-virtualized Network Driver with VirtualBox

UPDATE 7/20/2016: I recommend you also read my more recent Vagrant 1.8.5 and VirtualBox 5.1 Networking Performance post with update network performance numbers from VirtualBox 5.1 and a better method to enable Virtio with Vagrant/VirtualBox.


While traditionally emulators, including VirtualBox, have utilized emulated hardware devices today there is another option. Modern Linux kernels (>= 2.6.25 unless your distro backported it) support para-virtualized network devices drivers which are optimized for use in an emulated guest. This avoids much of the performance loss and other compatibility problems with emulating hardware devices. To illustrate the advantages here are some quick iperf numbers from a Xenial image running under Windows 10 in VirtualBox 5.0.24

• Intel 1000 Pro MT Server: 236 Mbits/sec
• Virtio Paravirtualized Network: 361 Mbits/sec

Anecdotally the performance of the Virtio driver also appears more consistent with the hardware emulation mode sometimes "hiccupping", and lower general cpu utilization.

Using the Virtio Para-virtualized Network Driver

Utilizing the Virtio Para-virtualized network driver is quick and easy in the current version of VirtualBox.

• Confirm your Linux guest are configured to use the KVM paravirtualization provider – this should be the default in recent VirtualBox versions. This also enabled paravirtualized clocks and SMP spinlocks.

• Change your network adapter type to ‘Paravirtualized Network (virtio-net)’.

• Reboot and your Linux guest will automatically use the new driver. To confirm ‘lspci’ should now show your network card as ‘Ethernet controller: Red Hat, Inc Virtio network device’.

For more information about the Virtio driver, or VirtualBox networking in general, please see the Chapter 6. Virtual networking of the VirtualBox User Manual. 

Developing locally in Linux boxes using Vagrant, vagrant-vbguest, and VirtualBox

If you have read many of my previous entries, I’m sure you are aware that Linux is my primary development environment. For local development projects I use Vagrant and VirtualBox so that I’m able to quickly and easily make use of a variety of Linux distros (CentOS 7, Ubuntu LTS, and sometimes Debian). While I use these tools almost daily in Windows, they are fully supported under OS X and Linux as well.

Vagrant

Vagrant deserves a longer article of its own, so I’ll keep this brief. Simply put Vagrant creates and configures a variety of virtual development providers using any number of virtualization backend including local hypervisors, the cloud, or even containers. With Vagrant installed I can bring up a new Linux development environment with just two commands.

VirtualBox

Many are probably already familiar with Oracle’s excellent VirtualBox virtualization product. For running local Linux development images, I find that VirtualBox 5 meets all my requirements and provides favorable performance to competitors with the added bonus of being free for all purposes (The separate VirtualBox Extension Pack is only free for non-commercial use, however I do not normally utilize its features regardless).

vagrant-vbguest

The missing “glue” to make using Vagrant and VirtualBox together even easier. The vagrant-vbguest plugin for Vagrant automatically updates the guest additions for Linux guests, making it even simpler to create and update new Linux development boxes and/or update VirtualBox itself. The recent release of vagrant-vbguest version 0.12 was one of the impetuses for this post, and I’m pleased to mention that yours truly is credited with the first fix in the changelog: https://github.com/dotless-de/vagrant-vbguest/blob/v0.12.0/CHANGELOG.md

Getting Started

Installation

Install VirtualBox: https://www.virtualbox.org/wiki/Downloads

Install Vagrant: https://www.vagrantup.com/downloads.html
Install vagrant-vbguest:

‘vagrant plugin install vagrant-vbugest’

Provision and Start your Linux development box

In a new dedicated folder:

vagrant init ubuntu/trusty64
vagrant up --provider virtualbox

While the box is usable now, I normally recommend a box reload on first run to load any new kernels or kernel modules.

vagrant reload

Enter your new Linux development box

vagrant ssh

Congratulations!

You are now using your new Linux Development box. When you are complete exit like normal. The image can be suspended and restart/accessed with ‘vagrant halt’ and then ‘vagrant up && vagrant ssh’. When your done with the image ‘vagrant destroy’ will remove all traces. Additional options to configure your box can be found in the 'Vagrantfile' created in the directory you initialized the box. I frequently increase the memory allocation and number of virtual cpus, but customization to the network or even startup scripts are easily available.

The ‘vagrant’ command has a large number of other useful functions, so I encourage you to check them out. I used trusty64 in the example above, but plenty of additional prebuilt box images can be found at https://atlas.hashicorp.com/boxes/search including popular Debian 8, Ubuntu LTS, and CentOS 7 boxes.

gdistcc v0.9.x released!

After intensive development I’m pleased to announce gdistcc v0.9.x has just been released!

v0.9.x includes some major new features including:

• In addition to CentOS 7, gdistcc now supports Ubuntu 16.04 LTS, Ubuntu 14.04 LTS, and Debian 8.  Additional distros should only require proper entries in the new settings.json file and appropriate startup-script.
• Instance and distribution specific configuration have been moved to their own settings.json file making additions and customizations easy.
• ‘gdistcc {status, make}’ now checks if an instance has been terminated (Google’s preemptible instances) and exclude as appropriate, ‘gdistcc stop’ removes them as normal.
• Pump mode is NOT used to greatly reduce instance setup time as system headers are no longer required on the instances.  This also increases compatibility and makes better use of ccache's pre-processed header caching.
• gdistcc's github page now uses Travis CI (I might make a future post dedicated to this particular topic)
• gdistcc is now published on PYPI and can be easily installed with just ‘pip install gdistcc’