DMARC-Filter: OpenDMARC Filter v1.4.2 delorie.com 56HIJT291256022
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org BC7763858D26
MIME-Version: 1.0
References: <CA+1jF5rhwmUqe9-gMwDK6FioS13GRkLd0XyxUHeT5=-ycYUpxg AT mail DOT gmail DOT com>
 <45887d0a-17d3-40ce-bea5-13fdf9081edd AT systematicsw DOT ab DOT ca>
 <CA+1jF5psDkbym84tM-=C8VRGfSzLYPrOy-_KMYLBNqwj+ds_Tg AT mail DOT gmail DOT com>
 <3da288a3-3a70-41ca-b582-67500ae0fb9e AT SystematicSW DOT ab DOT ca>
In-Reply-To: <3da288a3-3a70-41ca-b582-67500ae0fb9e@SystematicSW.ab.ca>
Date: Thu, 17 Jul 2025 20:17:47 +0200
Message-ID: <CA+1jF5rJzWh0r3TUWjOA-EqEFW6KMaOqef-=_k530r4ZJSt=gQ@mail.gmail.com>
Subject: Re: SLOW ls(1) - cygwin dir lookups with WinNT async requests?
To: cygwin AT cygwin DOT com
From: =?utf-8?q?Aur=C3=A9lien_Couderc_via_Cygwin?= <cygwin AT cygwin DOT com>
Reply-To: =?UTF-8?Q?Aur=C3=A9lien_Couderc?= <aurelien DOT couderc2002 AT gmail DOT com>
Content-Type: text/plain; charset="utf-8"
Sender: "Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
Content-Transfer-Encoding: 8bit

On Mon, Jul 14, 2025 at 10:05 PM Brian Inglis via Cygwin
<cygwin AT cygwin DOT com> wrote:
>
> On 2025-07-14 04:44, Aurélien Couderc via Cygwin wrote:
> > On Sun, Jul 13, 2025 at 11:54 PM Brian Inglis via Cygwin wrote:
> >> On 2025-07-12 10:43, Aurélien Couderc via Cygwin wrote:
> >>> Follow up to an old thread,
> >>> Corinna wrote this:
> >>>> Forth, by not trusting Cygwin to do the right thing and adding a lot
> >>>> of unnecessary code for each single ACL, you slow down ls(1) even more
> >>>> on Cygwin, which already gets a beating for being slow.
> >>>
> >>> Did anyone consider doing the directory lookups piece by piece, as
> >>> async requests?
> >>> Each time read and process a bit of the directory listing, while the
> >>> filesystem requests for the next bits run as Windows async requests?
> >>> That would promote at least parallelism, instead of doing it serially
> >>> like now.
> >>
> >> ISTM the assumption is that any directories accessed are already cached by the
> >> filesystem, whether Linux, Unix, or Windows, so I/O is not required, and even in
> >> Cygwin's case it only has to return the Windows entries converted to
> >> dirent/posix_dent in the directory functions (in a 62KB cache buffer holding 100
> >> entries) for use by coreutils ls(1).
> >
> > OUCH. That assumption is IMHO very wrong.
> > 1. Windows is not good at caching dirs, and it is worse with memory pressure.
> > 2. Not everyone is using SSDs
> > 3. SMB dir lookups can take multiple 1/10s per file/ACL lookup, so if
> > you do not do it in parallel it can take seconds for even small dir
> > listings. Even small async requests could be a big win here,
> > especially for remote filesystems.
>
> Not much will help if you have memory pressure other than adding memory.
> Don't know whether SAMBA can be configured to be an improvement over bare SMB
> under Cygwin?
> Some SAM and AD info is cached if you run cygserver
>
> >> Stat and ACL info require additional calls.
>
> > Right, but my proposal is to do parallel/async lookups. The Windows NT
> > kernel depends heavily on multithreading and parallelism, of which
> > Cygwin uses nothing right now for dir lookups.
>
> Not an option unless it can be cheaply done under winsup/cygwin/fhandler.
> Most utilities are GNU/BSD/Linux ports, so custom mods would have to be
> submitted and accepted upstream, as we often already have enough patches to
> maintain, to get them to build and work cleanly under Cygwin.

I think you misunderstand me. I am suggesting to improve the Cygwin
implementation of opendir(), readdir() and friends to do Win32/WInNT
calls async to speed up dir listings.

Aurélien
-- 
Aurélien Couderc <aurelien DOT couderc2002 AT gmail DOT com>
Big Data/Data mining expert, chess enthusiast

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple