DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 56EK5bxB777534
Authentication-Results: delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results: delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter: OpenDKIM Filter v2.11.0 delorie.com 56EK5bxB777534
Authentication-Results: delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=qRypyuS5
X-Recipient: archive-cygwin AT delorie DOT com
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B451D3858433
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1752523535;
	bh=3B9Ga9DZEwrLAkeNQ1QNDnsJnQ/L7rAF8AsQ9y8rIt8=;
	h=Date:Subject:To:References:In-Reply-To:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=qRypyuS5/pKRzRqPCAtZkAM8gDJ9wsnDTjHSHSKJ7I9wqB0t7z7x+yRoPNOWRov1z
	 sogB2Inm9+gIRZIlAx5HcabPBC8kNdgJ+1PzUMtMCqcDwKHLERfh2F50gNn60dgvA+
	 cOzNWIrosRLma0AJbMtlqtTLWtsctvRqp+sb3QwY=
X-Original-To: cygwin AT cygwin DOT com
Delivered-To: cygwin AT cygwin DOT com
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7E32B3858D32
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 7E32B3858D32
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1752523478; cv=none;
 b=wy3nSBXU91a6uMY7NfA3ournIjAfYLU+Pna41MUxB/3oJTY7EuTqUb9yUD6oCjgnLCZLQURMrs89OfJAdfMeLJKVqZeIWLK25PJjJo1nEuEBHHO5XE8eW0N2XA7ZNEyAomvRVlQCLwge1c3xq1GDBRRsd+5xq9TTsBeQPUdT/Ss=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1752523478; c=relaxed/simple;
 bh=p+fpx/BYXbsNz9n5QWigasrrQR9M6k3R2Mu+4vwDVRU=;
 h=Message-ID:Date:MIME-Version:From:Subject:To:DKIM-Signature;
 b=WYZlNL7v1+ISc4thcpLqWzg118ovJa6bPtQ5wcY1zwSRfUKSSgFA3zeVIBSlXoR9qUbt3UG6weVTx/+2gMUCCZ2B10obOJ9bU+PBWN9KARiYp4ZOBRAbktpKPgKuqQZsrsuj2cNSvBZAP1au7EN3JKEdO3/xRug66XcXDljHNK8=
ARC-Authentication-Results: i=1; server2.sourceware.org
DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7E32B3858D32
Message-ID: <3da288a3-3a70-41ca-b582-67500ae0fb9e@SystematicSW.ab.ca>
Date: Mon, 14 Jul 2025 14:04:35 -0600
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: SLOW ls(1) - cygwin dir lookups with WinNT async requests?
Content-Language: en-CA
To: cygwin AT cygwin DOT com
References: <CA+1jF5rhwmUqe9-gMwDK6FioS13GRkLd0XyxUHeT5=-ycYUpxg AT mail DOT gmail DOT com>
 <45887d0a-17d3-40ce-bea5-13fdf9081edd AT systematicsw DOT ab DOT ca>
 <CA+1jF5psDkbym84tM-=C8VRGfSzLYPrOy-_KMYLBNqwj+ds_Tg AT mail DOT gmail DOT com>
Organization: Systematic Software
In-Reply-To: <CA+1jF5psDkbym84tM-=C8VRGfSzLYPrOy-_KMYLBNqwj+ds_Tg@mail.gmail.com>
X-Stat-Signature: 4fjaumt1pwhajxb1qwk198qxmqpow45z
X-Rspamd-Server: rspamout08
X-Rspamd-Queue-Id: B279519
X-Session-Marker: 427269616E2E496E676C69734053797374656D6174696353572E61622E6361
X-Session-ID: U2FsdGVkX1/IbX9RfndA/mTWYl//7CC6a/V4sAyZEf4=
X-HE-Tag: 1752523476-436612
X-HE-Meta: U2FsdGVkX18nXmnzAoSdfak76KRO22h+lHfEKhUIvqALHjU1dIAw0jtaztskUBAkJWVdVmCcM5KlllTc8UwbNvNEFwqjCSsAnMv/T6qwUYZGdhIEKeJKcpwdVYUyE9N23CMRyO2hukPm6CuPGxMTEeQSGY6sFuIyOGPtNylHBhbT0ssi6FJDgkOWwxLrgiR3V2acgYFniwiMg9uSwoXq96XnGKg2FICJAYzEUMijzarZ+pXIp0tBUhEC3pDVGjPjzUOwlSozQvw2rjEO0UAcSIz3B+D2YRKw2jBox12TbqnrJzNNPJSpR9/9MBTvVU51YuNrRVthgxxdub6R2nzIQ6Fdt9iq9l4i
X-BeenThere: cygwin AT cygwin DOT com
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Unsubscribe: <https://cygwin.com/mailman/options/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=unsubscribe>
List-Archive: <https://cygwin.com/pipermail/cygwin/>
List-Post: <mailto:cygwin AT cygwin DOT com>
List-Help: <mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe: <https://cygwin.com/mailman/listinfo/cygwin>,
 <mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From: Brian Inglis via Cygwin <cygwin AT cygwin DOT com>
Reply-To: cygwin AT cygwin DOT com
Cc: Brian Inglis <Brian DOT Inglis AT SystematicSW DOT ab DOT ca>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from base64 to 8bit by delorie.com id 56EK5bxB777534

On 2025-07-14 04:44, Aurélien Couderc via Cygwin wrote:
> On Sun, Jul 13, 2025 at 11:54 PM Brian Inglis via Cygwin wrote:
>> On 2025-07-12 10:43, Aurélien Couderc via Cygwin wrote:
>>> Follow up to an old thread,
>>> Corinna wrote this:
>>>> Forth, by not trusting Cygwin to do the right thing and adding a lot
>>>> of unnecessary code for each single ACL, you slow down ls(1) even more
>>>> on Cygwin, which already gets a beating for being slow.
>>>
>>> Did anyone consider doing the directory lookups piece by piece, as
>>> async requests?
>>> Each time read and process a bit of the directory listing, while the
>>> filesystem requests for the next bits run as Windows async requests?
>>> That would promote at least parallelism, instead of doing it serially
>>> like now.
>>
>> ISTM the assumption is that any directories accessed are already cached by the
>> filesystem, whether Linux, Unix, or Windows, so I/O is not required, and even in
>> Cygwin's case it only has to return the Windows entries converted to
>> dirent/posix_dent in the directory functions (in a 62KB cache buffer holding 100
>> entries) for use by coreutils ls(1).
> 
> OUCH. That assumption is IMHO very wrong.
> 1. Windows is not good at caching dirs, and it is worse with memory pressure.
> 2. Not everyone is using SSDs
> 3. SMB dir lookups can take multiple 1/10s per file/ACL lookup, so if
> you do not do it in parallel it can take seconds for even small dir
> listings. Even small async requests could be a big win here,
> especially for remote filesystems.

Not much will help if you have memory pressure other than adding memory.
Don't know whether SAMBA can be configured to be an improvement over bare SMB 
under Cygwin?
Some SAM and AD info is cached if you run cygserver

>> Stat and ACL info require additional calls.

> Right, but my proposal is to do parallel/async lookups. The Windows NT
> kernel depends heavily on multithreading and parallelism, of which
> Cygwin uses nothing right now for dir lookups.

Not an option unless it can be cheaply done under winsup/cygwin/fhandler.
Most utilities are GNU/BSD/Linux ports, so custom mods would have to be 
submitted and accepted upstream, as we often already have enough patches to 
maintain, to get them to build and work cleanly under Cygwin.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retrancher  but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple