Mass delete files in FTP server

Ran into a little problem today, something I don’t really do often and something many would never do! Yeah, I’m talking about FTP .. the old pre-historic service of a thing that was originally RFC’d back in 1971. I had to mass delete files in FTP server, empty a folder and delete that too. Sounds simple, but not really!

So this FTP server had some database and the database got corrupted. Since the database got corrupted, the log directory just kept accumulating logs without deleting as none of the files got processed. Over time, it had about 145789 files sizes between 400mb to 4gb. You know web logs, swells up when people wake up and slows down when everyone off sleeping. Any how, so now I got to delete those 145789 files through old school FTP. It’s supposed to be easy but noooo, I cannot use FileZilla or WinSCP or any GUI client to make my life easier. I have to use CLI FTP. Let’s see:

ubuntu@ftp-client01:~$ ftp
ftp> open ftp-server01
Connected to ftp-server01.
220 Welcome to the Web FTP service.
Name (ftp-server01:ubuntu): admin
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
ftp> ls
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
drwxrwx---    2 ftp      ftp       3878912 Sep 07 00:07 Web_DC1_Log_Dir
drwxrwx---    2 ftp      ftp         36864 Sep 06 07:37 testdatabase
226 Directory send OK.

So far so good, I logged in and can list the directories. It seems there’s a test database folder as well. Let’s check the contents first just to ensure I am looking at the right thing.

ftp> cd Web_DC1_Log_Dir
250 Directory successfully changed.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
-rw-r--r-- 1 ftp ftp 3778547 Jun 15 05:16 webserver.local01-2018-06-15-05_003.log.gz
-rw-r--r-- 1 ftp ftp 4002151 Jun 15 05:21 webserver.local01-2018-06-15-05_004.log.gz
-rw-r--r-- 1 ftp ftp 4139598 Jun 15 05:26 webserver.local01-2018-06-15-05_005.log.gz
-rw-r--r-- 1 ftp ftp 3936480 Jun 15 05:31 webserver.local01-2018-06-15-05_006.log.gz
-rw-r--r-- 1 ftp ftp 3772733 Jun 15 05:36 webserver.local01-2018-06-15-05_007.log.gz
-rw-r--r-- 1 ftp ftp 3763328 Jun 15 05:41 webserver.local01-2018-06-15-05_008.log.gz
-rw-r--r-- 1 ftp ftp 3725765 Jun 15 05:46 webserver.local01-2018-06-15-05_009.log.gz
-rw-r--r-- 1 ftp ftp 3597685 Jun 15 05:51 webserver.local01-2018-06-15-05_010.log.gz

226 Directory send OK.
ftp> cd ..
250 Directory successfully changed.
ftp> cd testdatabase
250 Directory successfully changed.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
-rw-r--r-- 1 ftp ftp 1394829 Apr 14 02:00 Web_Reporter_140.log.gz.done
-rw-r--r-- 1 ftp ftp 1394711 Apr 15 02:00 Web_Reporter_150.log.gz.done
-rw-r--r-- 1 ftp ftp 1386116 Apr 16 02:00 Web_Reporter_160.log.gz.done
-rw-r--r-- 1 ftp ftp 1314330 Apr 17 02:00 Web_Reporter_170.log.gz.done
-rw-r--r-- 1 ftp ftp 1292357 Apr 18 02:00 Web_Reporter_180.log.gz.done
-rw-r--r-- 1 ftp ftp 1289469 Apr 19 02:00 Web_Reporter_190.log.gz.done
226 Directory send OK.
ftp> cd ..
250 Directory successfully changed.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
drwxrwx--- 2 ftp ftp 3878912 Sep 07 00:12 Web_DC1_Log_Dir
drwxrwx--- 2 ftp ftp 36864 Sep 06 07:37 testdatabase
226 Directory send OK.

Looks like all good and ready to go. Let’s try cli-kungfu on that before trying it on the main directory.

ftp> ?
Commands may be abbreviated.  Commands are:

!        debug        mdir        qc        send
$        dir        mget        sendport    site
account        disconnect    mkdir        put        size
append        exit        mls        pwd        status
ascii        form        mode        quit        struct
bell        get        modtime        quote        system
binary        glob        mput        recv        sunique
bye        hash        newer        reget        tenex
case        help        nmap        rstatus        tick
cd        idle        nlist        rhelp        trace
cdup        image        ntrans        rename        type
chmod        lcd        open        reset        user
close        ls        prompt        restart        umask
cr        macdef        passive        rmdir        verbose
delete        mdelete        proxy        runique        ?

mdelete is definitely the way to go as you can delete many files. Let’s try that.

Now hang on, I don’t want to confirm for each file, this is really annoying and time consuming when you have to do it 14000 times! Let’s try something else like rm -r or rmdir -r to do recursive deletion.

ftp>
ftp> rm -r testdatabase
550 Remove directory operation failed.
ftp> rm -r testdatabase/*
550 Remove directory operation failed.
ftp> rmdir -r testdatabase/*
550 Remove directory operation failed.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
drwxrwx---    2 ftp      ftp       3878912 Sep 07 00:12 Web_DC1_Log_Dir
drwxrwx---    2 ftp      ftp         36864 Sep 07 00:14 testdatabase
226 Directory send OK.
ftp> exit
221 Goodbye.

Alright, that was a fail! I searched around and there’s so little info about FTP and how to delete non-empty folder or mass delete files from CLI. Everyone uses some GUI client that allows you to confirm in bulk. Finally I found a way.

Apparently, you can use use -i flag to turn off interactive, so FTP won’t prompt you for every line when using mdelete. Hope it works:

ubuntu@ftp-client01:~$ ftp -i
ftp> open ftp-server01
Connected to ftp-server01.
220 Welcome to the Reporter FTP service.
Name (ftp-server01:ubuntu): admin
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
ftp>
ftp> ls
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
drwxrwx---    2 ftp      ftp       3878912 Sep 07 00:12 Web_DC1_Log_Dir
drwxrwx---    2 ftp      ftp         36864 Sep 07 00:14 testdatabase
226 Directory send OK.
ftp> mdelete -r testdatabase/*
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.

I works! Now to try the same command on the large folder:

ftp> mdelete -r Web_DC1_Log_Dir/*
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
250 Delete operation successful.
ftp> cd Web_DC1_Log_Dir
250 Directory successfully changed.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
226 Directory send OK.
ftp> ls
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
226 Directory send OK.

Looks like the directory is now empty and can be deleted.

ftp> cd ..
250 Directory successfully changed.
ftp> rmdir Web_DC1_Log_Dir
250 Remove directory operation successful.
ftp>
ftp> rmdir testdatabase
250 Remove directory operation successful.
ftp> dir
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
226 Directory send OK.
ftp>
ftp> ls
200 PORT command successful. Consider using PASV.
150 Here comes the directory listing.
226 Directory send OK.
ftp> exit
221 Goodbye.

Well, that was a good one and something I don’t use often. Hence this post. Hope it helps the next person somehow!