CoreOS Disk Space Management on GCE
· 2 minute read
If running a CoreOS cluster on vanilla Google compute instances you’ll soon run out of disk space. The default GCE boot disk is 9Gb, Docker is disk hungry and doesn’t garbage collect itself. Everyone seems to write their own garbage collector, although Spotify recently open sourced their script. Either way you don’t have to run many containers to run out of space:
core@prod01 ~ $ sudo btrfs fi show Label: 'ROOT' uuid: cbb8943a-616d-4b3d-8a01-bceb748b4dfd Total devices 1 FS bytes used 3.58GiB devid 1 size 6.68GiB used 6.68GiB path /dev/sda9 Btrfs v3.17.1
Why those funky
btrfs commands? Docker requires a CoW (Copy on Write) filesystem which at present are immature with limited options: AUFS, BTRFS and OverlayFS. CoreOS originally chose BTRFS, although recently switched to OverlayFS as it is now clearly the future.
It’s possible to save yourself some space by deleting old journal logs (
journalctl --vacuum-time=2d), or running your own flavour of Docker garbage collect (
docker rmi $(docker ps -q) or something more sophisticated). This might help if you’re out of disk space right now, but beyond immediate mitigation you’ll need to add extra disk storage.
I added an entry in my
cloud-config.yaml to format and mount the
varlibdocker device at
units: - name: format-ephemeral.service command: start content: | [Unit] Description=Formats the ephemeral drive [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/sbin/wipefs -f /dev/disk/by-id/google-varlibdocker ExecStart=/usr/sbin/mkfs.btrfs -f /dev/disk/by-id/google-varlibdocker - name: var-lib-docker.mount command: start content: | [Unit] Description=Mount ephemeral to /var/lib/docker Requires=format-ephemeral.service After=format-ephemeral.service Before=docker.service [Mount] What=/dev/disk/by-id/google-varlibdocker Where=/var/lib/docker Type=btrfs
Then when I want to launch a new instance all I need to do is create the disk (in this case a 20Gb SSD):
gcloud compute disks create dockerdisk07 --size 20GB --type pd-ssd
… and then launch the instance with it attached and labelled:
gcloud compute instances create mycoreos07 \ --disk name=dockerdisk07,device-name=varlibdocker \ --image coreos --machine-type n1-standard-1 \ --metadata-from-file user-data=cloud-config.yaml
This results in a CoreOS instance with a more healthy 20Gb disk for Docker usage.