When attempting to use tab completion on my ubuntu 16.04 box running on an AWS EC2 instance I was greeted with the following error in the terminal.

cannot create temp file for here-document: No space left on device with plenty of space left

I ran the following command to see what my disk usage was like and was surprised to see that the box was only at 50% capacity.

df -h

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            487M     0  487M   0% /dev
tmpfs           100M  4.3M   95M   5% /run
/dev/xvda1      7.8G  3.7G  3.7G  50% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1000

Next, after a lot of research I was led to check the status of the inodes on the instance.

df -ih

Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             122K   360  122K    1% /dev
tmpfs            124K   443  124K    1% /run
/dev/xvda1       512K  512K    0K  100% /
tmpfs            124K     1  124K    1% /dev/shm
tmpfs            124K     7  124K    1% /run/lock
tmpfs            124K    16  124K    1% /sys/fs/cgroup
tmpfs            124K     4  124K    1% /run/user/1000

Hey! look! Im all out of inodes for this instance. This is caused because there are too many file pointers.

The solution is to remove some files, but which ones? I cant remove the files that are serving my websites, so what can I do?

Chances are, there are tons of old linux header files from previous kernel versions. These can be removed to free up a good chunk of inodes.

First

lets find out what kernel we are using.
uname -a

Linux ip-172-31-55-89 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Next

Lets see what headers are installed
dpkg -l | awk '/-headers-/ {print $2}'

linux-headers-4.4.0-64
linux-headers-4.4.0-64-generic
linux-headers-4.4.0-66
linux-headers-4.4.0-66-generic
linux-headers-4.4.0-70
linux-headers-4.4.0-70-generic
linux-headers-4.4.0-71
linux-headers-4.4.0-71-generic
linux-headers-4.4.0-72
linux-headers-4.4.0-72-generic
linux-headers-4.4.0-75
linux-headers-4.4.0-75-generic
linux-headers-4.4.0-78
linux-headers-4.4.0-78-generic
linux-headers-4.4.0-79
linux-headers-4.4.0-79-generic
linux-headers-4.4.0-81
linux-headers-4.4.0-81-generic
linux-headers-4.4.0-83
linux-headers-4.4.0-83-generic
linux-headers-4.4.0-87
linux-headers-4.4.0-87-generic
linux-headers-4.4.0-89
linux-headers-4.4.0-89-generic
linux-headers-4.4.0-91
linux-headers-4.4.0-91-generic
linux-headers-4.4.0-93
linux-headers-4.4.0-93-generic
linux-headers-4.4.0-96
linux-headers-4.4.0-96-generic
linux-headers-generic
linux-headers-virtual

Then

we probably need to make some space real quick so we can run the next command to purge the old kernel headers.

sudo dpkg -P linux-image-4.4.0-{64,66,70,71,72,75,78,79,81,83,87,89}-generic 

Finally

we can purge the old kernel headers to free up a chunk of inodes and storage.

sudo apt autoremove --purge
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             122K   360  122K    1% /dev
tmpfs            124K   443  124K    1% /run
/dev/xvda1       512K  453K   60K   89% /
tmpfs            124K     1  124K    1% /dev/shm
tmpfs            124K     7  124K    1% /run/lock
tmpfs            124K    16  124K    1% /sys/fs/cgroup
tmpfs            124K     4  124K    1% /run/user/1000

That 11% should be enough to keep this instance happy