Linux: ncdu and mc to manage large directories

Creating, modifying and deleting files are everyday tasks in any operating system, even more so by Sysadmins, Developers and Programmers. For the most part, these tasks are fast enough when managing a handful of files. However, on Linux, and especially with servers, you may sometimes have to manage millions or even billions of files in a single directory.

For example, three weeks ago, I had to delete a directory containing 711,057,408 files because stacklinux vps hosting The customer could not delete them. I made a note to blog about that issue. However, there are already many blog posts and QA’s out there with alternative commands for deleting large numbers of files.

So although we’ll briefly touch on rm Command Options, I would like to focus on two command-line tools for viewing and managing Linux file systems. they are MC And NCDU, More on them later.

“can not execute [Argument list too long]When using “rm .

Have you tried deleting the contents of the directory rm command but received the following error? can not execute [Argument list too long], This limit occurs when rm The command is used to delete a directory containing a large number of files. In short, if the ARG_MAX limit is exceeded, the shell fails to invoke the command.

Check the range with:

getconf ARG_MAX

With Linux 2.6.23+, ARG_MAX is no longer hardcoded. see git entry, It is limited to 1/4 of the stack size (ulimit -s), which ensures that the program can still run. see also git diff of fs/exec.c, This limit will also be affected when using clamshell, rasa, mv, Etcetera. So you can use alternative commands that don’t hit this limit. for example, search execution either find – delete ,faster!,

find . -type f -delete

To delete only files with a specific extension:

find . -name "*.log" -type f -delete

My preferred method, which is about twice as fast as using find-delete, uses rsync. Rs Sync Typically used to synchronize files between two different locations, usually remote, but may also be on the same system. all we want to do is sync target directory (directory with a large # of files) an . with empty directory, in my case, /path/to/var/session/ The directory contained over seven hundred million files (a symptom of session management). So we will first create an empty directory: empty_dir/, (can be named anything),

mkdir empty_dir/

Next, we will use the same -delete Option to be used with find, also with rsync. Example:

rsync -a --delete [empty directory] [target directory]

or, in the case mentioned at the beginning:

rsync -a --delete /path/to/empty_dir/ /path/to/var/session/

-a or -collection is equivalent to using -rlptgody, It’s a quick way to say you want recursion and preserve almost everything (-h with one notable omission). The only exception to the above equivalence is when -files-from specified, in which case -R not rooted. note that -a does not preserve hard links Because finding multi-linked files is expensive. you have to specify separately -h, (see: man rsync)

What usually took hours to complete in about 10 to 15 minutes using rsync.

NCDU (NCurses disk usage)

NCDU

NCDU There is a curse-based version of the famous ‘du’. It provides a fast way to view and manage directories using disk space. Users can navigate using the arrow keys and delete files that take up a lot of space by pressing the ‘d’ key.

To install on Debian or Ubuntu, run:

apt install ncdu

On CentOS, enable the Apple repo, then install:

yum install epel-release
yum install ncdu

To delete a directory or file, select and press D, type , for a list of shortcuts.

MC (Midnight Commander) File Manager

MC (Midnight Commander)

A more powerful and feature packed option would be midnight commander (MC). ‘MC’ is a directory browser/file manager for Unix-like operating systems. Features of Midnight Commander include the ability to view the contents of RPM package files, work with archive formats, and work as an FTP client. Midnight Commander includes a standalone editor called McEdit.

To install on Debian or Ubuntu, run:

apt install mc

On CentOS:

yum install mc

As in the screenshot, press F8 on the keyboard to delete. MC And NCDU There are additional command-line tools for use with already well-known commands such as rasa, Of, And df,

reference:
MC Guide (PDF): http://navaz.org/media/docs/mc/mc.pdf
man mc: https://linux.die.net/man/1/mc
man ncdu: https://linux.die.net/man/1/ncdu
arg_max:

Leave a Comment