Problem

I started having an issue with FTP where users were unable to create new files in their FTP areas. The following error displayed in my Filezilla client when I tested:

553 Could not create file

 

Troubleshooting

 

I examined vsftpd and all permissions on the server but they looked fine. I then tried to copy a file from one partition to the destination and got this error:

cp: cannot create regular file `/data/www/test.png': No space left on device

There was an abundance of space left on that particular partition:

# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       12G  4.2G  6.7G  39% /
tmpfs                 935M     0  935M   0% /dev/shm
/dev/sda1             485M  107M  353M  24% /boot
/dev/mapper/VolGroupData-DataVol
                       40G   13G   25G  34% /data

Then I tried the following command (to show iNodes)

# df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                      758880   71899  686981   10% /
tmpfs                 239335       1  239334    1% /dev/shm
/dev/sda1             128016      58  127958    1% /boot
/dev/mapper/VolGroupData-DataVol
                     2621440 2621440       0  100% /data

Resolution

After investigating it looks like the file system had reached the maximum number of files allowed. The next stage was to track down which folder had all these files. As it was the /data partition all logging was set to the / root partition as per normal. The only data in the /data partition was a WordPress website. I delved further into this and could see there was 4096 folders in the /data/www/wp-content/w3tc/objectcache/ directory with lots of sub directories within these directories. This added up to millions.

The culprit turned out to be W3 Total Cache which is a WordPress extension. The web developers were notified.