DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 56HIJT291256022
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 56HIJT291256022
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=TaZDlX+Y
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 29542385EC41
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1752776366;
	bh=EWo3S+N8AtRxtJ3fGw0up+skCwa57IYcWhwI8l4iwkI=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=TaZDlX+YSrgkEIFDTJHbqR1LwIbl15LEsmN5z9Od4x8clkeM6XoqaUBNjGW+3bPRG
	MnJlmUaDMzGDyffIZLIzlZaceCAPNoEwx86SM+RmwtypvWKKlCT84FS2sQ2CjULcor
	A4ycZZaXp3fD4xFaX1USNaZND4N3Vci7OoBOltY4=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org BC7763858D26
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org BC7763858D26
ARC-Seal:	i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1752776305; cv=none;
	b=npk8eN5LXEz3qN4obDOkYiTLhZorSaIstqU/7M3qMBTFWsSBs/k6+qlvwSW6jpat2yx0DkCgQFFnE9/KIQkC/gXJaKMCrqjkYpfoOHTyOfzdewXCOZ/64K8SXSWRZC006s5l8HXCzm8GzPpTr26VM018H0nd6yAyycgT0xnFwRI=
ARC-Message-Signature:	i=1; a=rsa-sha256; d=sourceware.org; s=key;
	t=1752776305; c=relaxed/simple;
	bh=uacc6VDJc12AeFDbogR1wj1vqYpV9X1V0MRCA0bv8y0=;
	h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To;
	b=PUlxDTaPrxDNl03ONKqjlJ3wCZmITymH0gB3rxWWofS0N7EZXtUxOk0SvZ9JRaQtHnCAaXS5zCeeTDUweLj1RiTvwjC7Z9Kd8WP23FatfLJDuVFyMzJwxDVv6YoobNMtTTau/ZUvyvqZHPmJzUqKSQsDx/jvWZQM1UQMbNSoPHg=
ARC-Authentication-Results:	i=1; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org BC7763858D26
X-Google-DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20230601; t=1752776304; x=1753381104;
	h=content-transfer-encoding:to:subject:message-id:date:from
	:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
	:subject:date:message-id:reply-to;
	bh=uacc6VDJc12AeFDbogR1wj1vqYpV9X1V0MRCA0bv8y0=;
	b=W/PCPzkWVyue5iIsPrEuAcM80o/lHB8elDQHT91mgsUoh9CFNnLHVIed3HCRwzjmng
	5IN+3i7BNsj4nd1R079vOKfZxUhzZrnWs7x60qYRtJNxTHNMcAI9p459ni5nbHcDkCoU
	hRTL1XFuuACRLTPGD5FXJEqZhg1knEJ1jzbaL8xmoZdt+nIDWoyFT9AgVB3GtbO8XMfE
	BglxhMqSnSCK+CpavXbGp14JhVDdIz3b60II1ogQl4k6os+QfAJ0iEi08JRXEuwJJr8C
	XPt37X8pF94h9fqLmKEHCoZGAlsOHsis7hMziEia1LpHt4F2iFYFVxh8BP/RpeqESixh
	Mrzg==
X-Gm-Message-State:	AOJu0Yw6bBNlGxCcsg9wB6AEuSnmGhAe5gsP3khNHbzgg5r4eQvJi3lw
	wMqdHHr5OoD4odwkjfKT3p1Hwu28gTMUDeF+c7uW9HUnYdD4TALRrmzwRgTs0g4TaAaF99q4Pk4
	Z0umbTyvt3ejtPsB7FCW8BMugqGVqAdltmsQwl98=
X-Gm-Gg:	ASbGncvKgsCZBZvtmrR4fwkaJCzCYhA8YEyVGI4UWWR9pJRjXnYIgPs3Ocq/XeFGDGl
	zLjVUIpGBuLfi8gQhc+WxIExCBoS1dT9s3TkjElEc6YkEuXzsno/Vjoru7qjOx6rnoBIAVDGbcA
	x/z/G/2eXrGXIeZnuYFyMrvSQhhTSGDTPC8V44SItQ55LSFZXkEG7+wfFzHhX/aqM3+T3WfLxQq
	hQCr44=
X-Google-Smtp-Source:	AGHT+IFZN8Jm+o+TGC32Qf4uXsxEKxcISFprC7/IxtOxouPfSISNOcA2AwziT2VF31bPjvcpw7pr98B0eRFTUjDp3+0=
X-Received:	by 2002:a05:6871:e787:b0:2ff:9b13:f690 with SMTP id
	586e51a60fabf-2ffd1f1864bmr3194777fac.23.1752776303848; Thu, 17 Jul 2025
	11:18:23 -0700 (PDT)
MIME-Version:	1.0
References:	<CA+1jF5rhwmUqe9-gMwDK6FioS13GRkLd0XyxUHeT5=-ycYUpxg AT mail DOT gmail DOT com>
	<45887d0a-17d3-40ce-bea5-13fdf9081edd AT systematicsw DOT ab DOT ca>
	<CA+1jF5psDkbym84tM-=C8VRGfSzLYPrOy-_KMYLBNqwj+ds_Tg AT mail DOT gmail DOT com>
	<3da288a3-3a70-41ca-b582-67500ae0fb9e AT SystematicSW DOT ab DOT ca>
In-Reply-To:	<3da288a3-3a70-41ca-b582-67500ae0fb9e@SystematicSW.ab.ca>
Date:	Thu, 17 Jul 2025 20:17:47 +0200
X-Gm-Features:	Ac12FXzsVTIgLLszjwZUK5xe2eg0OR0JHypAVk-hwmzKDkOBfQyG0Gww-my0kdE
Message-ID:	<CA+1jF5rJzWh0r3TUWjOA-EqEFW6KMaOqef-=_k530r4ZJSt=gQ@mail.gmail.com>
Subject:	Re: SLOW ls(1) - cygwin dir lookups with WinNT async requests?
To:	cygwin AT cygwin DOT com
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	=?utf-8?q?Aur=C3=A9lien_Couderc_via_Cygwin?= <cygwin AT cygwin DOT com>
Reply-To:	=?UTF-8?Q?Aur=C3=A9lien_Couderc?= <aurelien DOT couderc2002 AT gmail DOT com>
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 56HIJT291256022

On Mon, Jul 14, 2025 at 10:05 PM Brian Inglis via Cygwin
<cygwin AT cygwin DOT com> wrote:
>
> On 2025-07-14 04:44, Aurélien Couderc via Cygwin wrote:
> > On Sun, Jul 13, 2025 at 11:54 PM Brian Inglis via Cygwin wrote:
> >> On 2025-07-12 10:43, Aurélien Couderc via Cygwin wrote:
> >>> Follow up to an old thread,
> >>> Corinna wrote this:
> >>>> Forth, by not trusting Cygwin to do the right thing and adding a lot
> >>>> of unnecessary code for each single ACL, you slow down ls(1) even more
> >>>> on Cygwin, which already gets a beating for being slow.
> >>>
> >>> Did anyone consider doing the directory lookups piece by piece, as
> >>> async requests?
> >>> Each time read and process a bit of the directory listing, while the
> >>> filesystem requests for the next bits run as Windows async requests?
> >>> That would promote at least parallelism, instead of doing it serially
> >>> like now.
> >>
> >> ISTM the assumption is that any directories accessed are already cached by the
> >> filesystem, whether Linux, Unix, or Windows, so I/O is not required, and even in
> >> Cygwin's case it only has to return the Windows entries converted to
> >> dirent/posix_dent in the directory functions (in a 62KB cache buffer holding 100
> >> entries) for use by coreutils ls(1).
> >
> > OUCH. That assumption is IMHO very wrong.
> > 1. Windows is not good at caching dirs, and it is worse with memory pressure.
> > 2. Not everyone is using SSDs
> > 3. SMB dir lookups can take multiple 1/10s per file/ACL lookup, so if
> > you do not do it in parallel it can take seconds for even small dir
> > listings. Even small async requests could be a big win here,
> > especially for remote filesystems.
>
> Not much will help if you have memory pressure other than adding memory.
> Don't know whether SAMBA can be configured to be an improvement over bare SMB
> under Cygwin?
> Some SAM and AD info is cached if you run cygserver
>
> >> Stat and ACL info require additional calls.
>
> > Right, but my proposal is to do parallel/async lookups. The Windows NT
> > kernel depends heavily on multithreading and parallelism, of which
> > Cygwin uses nothing right now for dir lookups.
>
> Not an option unless it can be cheaply done under winsup/cygwin/fhandler.
> Most utilities are GNU/BSD/Linux ports, so custom mods would have to be
> submitted and accepted upstream, as we often already have enough patches to
> maintain, to get them to build and work cleanly under Cygwin.

I think you misunderstand me. I am suggesting to improve the Cygwin
implementation of opendir(), readdir() and friends to do Win32/WInNT
calls async to speed up dir listings.

Aurélien
-- 
Aurélien Couderc <aurelien DOT couderc2002 AT gmail DOT com>
Big Data/Data mining expert, chess enthusiast

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

- Raw text -