< index

CoreOS Disk Space Management on GCE

· 2 minute read

If running a CoreOS cluster on vanilla Google compute instances you’ll soon run out of disk space. The default GCE boot disk is 9Gb, Docker is disk hungry and doesn’t garbage collect itself. Everyone seems to write their own garbage collector, although Spotify recently open sourced their script. Either way you don’t have to run many containers to run out of space:

core@prod01 ~ $ sudo btrfs fi show
Label: 'ROOT'  uuid: cbb8943a-616d-4b3d-8a01-bceb748b4dfd
	Total devices 1 FS bytes used 3.58GiB
	devid    1 size 6.68GiB used 6.68GiB path /dev/sda9
Btrfs v3.17.1

Why those funky btrfs commands? Docker requires a CoW (Copy on Write) filesystem which at present are immature with limited options: AUFS, BTRFS and OverlayFS. CoreOS originally chose BTRFS, although recently switched to OverlayFS as it is now clearly the future.

It’s possible to save yourself some space by deleting old journal logs (journalctl --vacuum-time=2d), or running your own flavour of Docker garbage collect (docker rmi $(docker ps -q) or something more sophisticated). This might help if you’re out of disk space right now, but beyond immediate mitigation you’ll need to add extra disk storage.

I added an entry in my cloud-config.yaml to format and mount the varlibdocker device at /var/lib/docker:

  units:
    - name: format-ephemeral.service
      command: start
      content: |
        [Unit]
        Description=Formats the ephemeral drive
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/sbin/wipefs -f /dev/disk/by-id/google-varlibdocker
        ExecStart=/usr/sbin/mkfs.btrfs -f /dev/disk/by-id/google-varlibdocker
    - name: var-lib-docker.mount
      command: start
      content: |
        [Unit]
        Description=Mount ephemeral to /var/lib/docker
        Requires=format-ephemeral.service
        After=format-ephemeral.service
        Before=docker.service
        [Mount]
        What=/dev/disk/by-id/google-varlibdocker
        Where=/var/lib/docker
        Type=btrfs

Then when I want to launch a new instance all I need to do is create the disk (in this case a 20Gb SSD):

gcloud compute disks create dockerdisk07 --size 20GB --type pd-ssd

… and then launch the instance with it attached and labelled:

gcloud compute instances create mycoreos07 \
  --disk name=dockerdisk07,device-name=varlibdocker \
  --image coreos --machine-type n1-standard-1 \
  --metadata-from-file user-data=cloud-config.yaml

This results in a CoreOS instance with a more healthy 20Gb disk for Docker usage.

Related...