DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 538Ncl6x2349266
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 538Ncl6x2349266
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=AEqN8ezh
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 2B9193852768
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744155527;
	bh=faNZXvxCU76pjfTPEh2GBMlGh4l9uysZvVlQKSA6Pk8=;
	h=Date:To:Cc:Subject:References:In-Reply-To:List-Id:
	List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	From:Reply-To:From;
	b=AEqN8ezhXSu5x6RZprOq4MTwMTqgSGhCJAWYFbKufX0xLKzDP5LOxyEPDkFtj+46Q
	bvLYrf/iReKIu45y2sp5SnNOE3+ujL9b7t224hICWAJS8cRWiHF1QvRnd6SoAGjKif
	RpUS1KyxYXneUd/jd4A1H/BGayjs2RY3HqMCeEZU=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 235A1385EC3E
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 235A1385EC3E
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744155504; cv=none;
	b=lrrdd7+WYRx+NnKuVxSn83WemAc2/E+prAzvA9bilYcnPX1ndlHEFl/VKQVu/deYl0JuwN4Gsz5/W/R5QyK4t4kopGLHnIXutukI+CUFI24L+Jf6I6oHlGHtN6d01aV7Gr0JRW95Y1L+VKqoR6ZmPWwWxuyz2//JAVdkv63MmGI=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1744155504; c=relaxed/simple;
	bh=Kuu4c1cPJzh8aqFb/91LIFWskn1444yuBR8gfgqS5Fw=;
	h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version;
	b=CbHwdH+dFrOvEy+R4xItJ7pyG11nwWJ6lLpWlkyqfWmmgWjl5YwVAnfhIU/2WzF2QnzEzEnTF8p2qWGh99rzNxYnmHLzMqeWUX9idc94G51fa/pgFpCscUojVz1GLPZkSwh7kpsOt+RIMBov65PdMAZn5uh4S+ZOt8yPzEL/+U8=
ARC-Authentication-Results:	i=1; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 235A1385EC3E
Date:	Tue, 8 Apr 2025 19:38:20 -0400
To:	Mark Liam Brown <brownmarkliam AT gmail DOT com>
Cc:	cygwin AT cygwin DOT com
Subject:	Re: Tuning ls, ls -l, find ., find . -ls performance for very large
	dirs (60000+ files/dir)
Message-ID:	<Z_WzbNaNgcxiLcE9@xps13>
References:	<CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
MIME-Version:	1.0
In-Reply-To:	<CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A@mail.gmail.com>
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe:	<https://cygwin.com/mailman/options/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Glenn Strauss via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Glenn Strauss <gs-cygwin DOT com AT gluelogic DOT com>
Errors-To:	cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>

On Tue, Apr 08, 2025 at 08:05:21PM +0200, Mark Liam Brown via Cygwin wrote:
> Greetings!
> 
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
> 
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.

`man ls`

This might help a little bit:

`ls -1f`  # one (-1) for single line output, and -f to disable sorting


Of course, please also ask why there are 60000 files in one directory
rather than sharded or stored some other way.  e.g. for log files,
create an archive/ subfolder and rotate all non-recently,
less-frequently accessed files to that subfolder so that the main
folder has (many) fewer entries.

For a folder that size, `ls` and `ls -l` are less appropriate tools.

A web server could serve a single index.html with the directory listing,
and the directory listing could be updated when the folder changes, or
on a periodic basis.

Various solutions depend on context and usage of the folder and its
contents.

Whether or not cmd.exe handles such a large directory better is
immaterial to the assessment that 60000+ entries in a single folder
is lacking better folder organization/file management and access methods

Cheers, Glenn

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -