DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 56EAkw6r255973
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 56EAkw6r255973
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=jrQ4NnTg
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 178833858D37
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1752490016;
	bh=E7731w020yoQkM+D4TPFJiaI2KwLJpRx/P4jyYKBG9w=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=jrQ4NnTg63GwY0e2En73gm8c093eK+SjzZ6dAAGcima9PhOi4PmFoHjcPIoAcMU33
	t8z3Ipn9FoL3OgkAhXLGBASCpW9xxBQ7u4CaFO9xbRAzBdzclr5lksYTMt9+jMnXYW
	v5wGourTe/gz3tgwqaxhy/WfULQH00RimAZvCMgo=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org 047093858C51
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org 047093858C51
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1752489914; cv=none;
	b=sNEBGKQ2rjAuJQL1z//k7WbIY1cYhMVR8n0XjlllqDsIr7KyRccswLs2uRmNuDdfz4OtFtpfpxc+ijm86qyxRE/Q06ws9gBp97yvXq7G6j4GMjftBpobapPJ0Y0d+ZOuTpo5kJSWBcTWI84IrTW2rVjWPa379H3RAg3NvtTznL4=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1752489914; c=relaxed/simple;
	bh=9nr9Q5ckt7WVxLLvztt3lLZ30DaVEUluA5NAPM+vATU=;
	h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To;
	b=v+w1/0PTbGS4ZXS0TUxrRtQTV6BSLc8iSDPljwPW5kuM3KQH6UDrTjCfsajuc/Pa5OKlVgXlpLQIPd4VAxl5Q7lwg67CWhFH0d/JywlWd0dajXmiMtIPkz8jeRHq9Mt3Qn6PegDxbltSFwC+iXQ+nwgWrygDyjFbzAs30qhIEEM=
ARC-Authentication-Results:	i=1; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 047093858C51
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20230601; t=1752489913; x=1753094713;
	h=content-transfer-encoding:to:subject:message-id:date:from
	:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
	:subject:date:message-id:reply-to;
	bh=9nr9Q5ckt7WVxLLvztt3lLZ30DaVEUluA5NAPM+vATU=;
	b=p24uq3AD6Z7DKrKGgZE4nby0Q+YWvBMw+ciwNQBc7Hr4MMkvX4h/SL8ymlgdIrvdnf
	9s6TnZ34NbCohnxruPxq1FJtDke54akvNhbkgYPcksH7iVM38g8hDsN+2NuqwLqafA6c
	+vzVa0i0ecGVZVO+3b/qWbde5x/BMn6kDMmVkd5Ja5QVk2WpVyjpy2gDw4/GuFnoKe7l
	lfa6Aql6DSKKq4mh4iQqrlJzHt4ZZVg1IgpbQ5dIgh0cs/EcYNvHJfrPYrJAEnkezSeO
	I6wU8exxDORDFJQss+A4YKbg1oIk1s3yuK11s2InGLHiAfmym8hx61QcYDfxEwGMoveS
	c3sg==
X-Gm-Message-State:	AOJu0Yxe2p0+hFDiX85i3OPKap96fdaUuidONPBbXM4LY0ASjq2sq4DH
	LHGe3szq3YDB/2b/AJH+3p3qgI0bfcf2BT+jLYBhrn4YXME5CUDe6JgFiQ9PnowqqjlJ7IYgMK6
	4g8qjANSPGSxip6qVUKKVy0rcCNKlUjWZ39Lo
X-Gm-Gg:	ASbGncsMAzQwOCJLN2jg1MmnBVepOB8+yKwtFKxBMoAswc6DtX3xlq+bcBDLTq9d5Xd
	eCA76TKi4gN8+LycwU6pykFtXtbK+TB3PkHBX7RxG2takP4YljQjbPnh7k2hbocwHKVVxGsLMfV
	pK4iwF43R+vWH/d7uITmt/+DEWErHNpoRV4wuCCWu0BNtmsremh4ehWrpTqnSSIuSFl0BQrUYp6
	y314Q==
X-Google-Smtp-Source:	AGHT+IGpHflCPaHtYW26IMnqbZ/kbMwznmdSmn3JgpePmRWxwckULxJ0O0QSWgsZaAbmq8RP2s/ETS8r+Tt2dwPfY8Y=
X-Received:	by 2002:a05:6870:bb16:b0:2c1:6948:d57c with SMTP id
	586e51a60fabf-2ff270740a5mr8979431fac.28.1752489913030; Mon, 14 Jul 2025
	03:45:13 -0700 (PDT)
MIME-Version:	1.0
References:	<CA+1jF5rhwmUqe9-gMwDK6FioS13GRkLd0XyxUHeT5=-ycYUpxg AT mail DOT gmail DOT com>
	<45887d0a-17d3-40ce-bea5-13fdf9081edd AT systematicsw DOT ab DOT ca>
In-Reply-To:	<45887d0a-17d3-40ce-bea5-13fdf9081edd@systematicsw.ab.ca>
Date:	Mon, 14 Jul 2025 12:44:37 +0200
X-Gm-Features:	Ac12FXzrOMr1A4ohENi2U-JmovTZNmL69yA0kv4uny5SVQqgD5iaocYHfqPgrDI
Message-ID:	<CA+1jF5psDkbym84tM-=C8VRGfSzLYPrOy-_KMYLBNqwj+ds_Tg@mail.gmail.com>
Subject:	Re: SLOW ls(1) - cygwin dir lookups with WinNT async requests?
To:	cygwin AT cygwin DOT com
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	=?utf-8?q?Aur=C3=A9lien_Couderc_via_Cygwin?= <cygwin AT cygwin DOT com>
Reply-To:	=?UTF-8?Q?Aur=C3=A9lien_Couderc?= <aurelien DOT couderc2002 AT gmail DOT com>
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 56EAkw6r255973

On Sun, Jul 13, 2025 at 11:54 PM Brian Inglis via Cygwin
<cygwin AT cygwin DOT com> wrote:
>
> On 2025-07-12 10:43, Aurélien Couderc via Cygwin wrote:
> > Follow up to an old thread,
> > Corinna wrote this:
> >> Forth, by not trusting Cygwin to do the right thing and adding a lot
> >> of unnecessary code for each single ACL, you slow down ls(1) even more
> >> on Cygwin, which already gets a beating for being slow.
> >
> > Did anyone consider doing the directory lookups piece by piece, as
> > async requests?
> > Each time read and process a bit of the directory listing, while the
> > filesystem requests for the next bits run as Windows async requests?
> > That would promote at least parallelism, instead of doing it serially
> > like now.
>
> ISTM the assumption is that any directories accessed are already cached by the
> filesystem, whether Linux, Unix, or Windows, so I/O is not required, and even in
> Cygwin's case it only has to return the Windows entries converted to
> dirent/posix_dent in the directory functions (in a 62KB cache buffer holding 100
> entries) for use by coreutils ls(1).

OUCH. That assumption is IMHO very wrong.
1. Windows is not good at caching dirs, and it is worse with memory pressure.
2. Not everyone is using SSDs
3. SMB dir lookups can take multiple 1/10s per file/ACL lookup, so if
you do not do it in parallel it can take seconds for even small dir
listings. Even small async requests could be a big win here,
especially for remote filesystems.

> Stat and ACL info require additional calls.

Right, but my proposal is to do parallel/async lookups. The Windows NT
kernel depends heavily on multithreading and parallelism, of which
Cygwin uses nothing right now for dir lookups.

Aurélien
-- 
Aurélien Couderc <aurelien DOT couderc2002 AT gmail DOT com>
Big Data/Data mining expert, chess enthusiast

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -