DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 538Ncl6x2349266
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 538Ncl6x2349266
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=AEqN8ezh
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2B9193852768
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1744155527;
	bh=faNZXvxCU76pjfTPEh2GBMlGh4l9uysZvVlQKSA6Pk8=;
	h=Date:To:Cc:Subject:References:In-Reply-To:List-Id:
	 List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
	 From:Reply-To:From;
	b=AEqN8ezhXSu5x6RZprOq4MTwMTqgSGhCJAWYFbKufX0xLKzDP5LOxyEPDkFtj+46Q
	 bvLYrf/iReKIu45y2sp5SnNOE3+ujL9b7t224hICWAJS8cRWiHF1QvRnd6SoAGjKif
	 RpUS1KyxYXneUd/jd4A1H/BGayjs2RY3HqMCeEZU=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 235A1385EC3E
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 235A1385EC3E
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1744155504; cv=none;
 b=lrrdd7+WYRx+NnKuVxSn83WemAc2/E+prAzvA9bilYcnPX1ndlHEFl/VKQVu/deYl0JuwN4Gsz5/W/R5QyK4t4kopGLHnIXutukI+CUFI24L+Jf6I6oHlGHtN6d01aV7Gr0JRW95Y1L+VKqoR6ZmPWwWxuyz2//JAVdkv63MmGI=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1744155504; c=relaxed/simple;
 bh=Kuu4c1cPJzh8aqFb/91LIFWskn1444yuBR8gfgqS5Fw=;
 h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version;
 b=CbHwdH+dFrOvEy+R4xItJ7pyG11nwWJ6lLpWlkyqfWmmgWjl5YwVAnfhIU/2WzF2QnzEzEnTF8p2qWGh99rzNxYnmHLzMqeWUX9idc94G51fa/pgFpCscUojVz1GLPZkSwh7kpsOt+RIMBov65PdMAZn5uh4S+ZOt8yPzEL/+U8=
ARC-Authentication-Results: i=1; server2.sourceware.org
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 235A1385EC3E
Date: Tue, 8 Apr 2025 19:38:20 -0400
To: Mark Liam Brown <brownmarkliam AT gmail DOT com>
Cc: cygwin AT cygwin DOT com
Subject: Re: Tuning ls, ls -l, find ., find . -ls performance for very large
 dirs (60000+ files/dir)
Message-ID: <Z_WzbNaNgcxiLcE9@xps13>
References: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A AT mail DOT gmail DOT com>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <CAN0SSYywSV5cbJhUFa7TYrSWuqZS-OBpKkJ=8vrcxXxHx2v70A@mail.gmail.com>
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Glenn Strauss via Cygwin <cygwin AT cygwin DOT com>
Reply-To: Glenn Strauss <gs-cygwin DOT com AT gluelogic DOT com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>

On Tue, Apr 08, 2025 at 08:05:21PM +0200, Mark Liam Brown via Cygwin wrote:
> Greetings!
> 
> Are there tuning variables to improve ls, ls -l, find ., find . -ls
> performance for very large dirs?
> 
> If we have a SMB dir with 60000+ entries a simple ls -l can take MANY
> minutes (22+mins), while cmd.exe dir just floods the terminal with
> results immediately.

`man ls`

This might help a little bit:

`ls -1f`  # one (-1) for single line output, and -f to disable sorting


Of course, please also ask why there are 60000 files in one directory
rather than sharded or stored some other way.  e.g. for log files,
create an archive/ subfolder and rotate all non-recently,
less-frequently accessed files to that subfolder so that the main
folder has (many) fewer entries.

For a folder that size, `ls` and `ls -l` are less appropriate tools.

A web server could serve a single index.html with the directory listing,
and the directory listing could be updated when the folder changes, or
on a periodic basis.

Various solutions depend on context and usage of the folder and its
contents.

Whether or not cmd.exe handles such a large directory better is
immaterial to the assessment that 60000+ entries in a single folder
is lacking better folder organization/file management and access methods

Cheers, Glenn

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple