Mail Archives: cygwin/2025/02/26/23:59:03

Roland Mainz via Cygwin <cygwin AT cygwin DOT com> · Thu, 27 Feb 2025 05:58:00 +0100

On Wed, Feb 26, 2025 at 9:26 PM Corinna Vinschen via Cygwin
<cygwin AT cygwin DOT com> wrote:
> On Feb 26 20:16, Roland Mainz via Cygwin wrote:
> > Something is wrong with UNC paths in Cygwin 3.6 build
> > 3.6.0-0.404.g323729f654ae.x86_64 and ms-nfs41-client 2025-02-21:
> >
> > Example:
> > ---- snip ----
> > $ cd //derfwnb4966_ipv4 DOT global DOT loc AT 2049/nfs4/bigdisk/builds/bash_build1
> > $ rm -Rfv bash
> > removed 'bash/builtins/hash.o'
> > removed 'bash/builtins/fc.o'
> > removed 'bash/builtins/jobs.o'
> > removed 'bash/builtins/help.o'
> > removed 'bash/builtins/history.o'
> > removed 'bash/builtins/let.o'
> > removed 'bash/builtins/kill.o'
> > removed 'bash/builtins/mapfile.o'
> > removed 'bash/builtins/pushd.o'
> > removed 'bash/builtins/read.o'
> > removed 'bash/builtins/return.o'
> > removed 'bash/builtins/setattr.o'
> > ---- snip ----
> >
> > Each line from rm -Rfv now needs around 3-5 seconds. Switching to CWD
> > //derfwnb4966_ipv4 AT 2049/nfs4/bigdisk/builds/bash_build1 (e.g. non-FQDN
> > for share host) does not fix the problem. Going to
> > /cygdrive/l/builds/bash_build1 and the rm -Rfv deletes around 70-90
> > files per second.
> > Last tested version was Cygwin commit
> > #4bcc6adec765ee8354bfc9e6c060c9d1c0c7bf46 (and same ms-nfs41-client
> > 2025-02-21 release), rm -Rfv performance there was normal (same as on
> > /cygdrive/l/...).
>
> I can't reproduce this on MS NFS and SMB.
>
> You should try this with the latest 406 build.

Done, works perfectly - see below, the issue was something different...

> If this doesn't fix
> it, I need to know which test release introduced this problem.

It turns out this is something very NFSv4.x-specific: Delegations!

A NFSv4.x server can grant a client a "read" or "write" delegation. A
client can cache reads or read/writes locally if the server grants
that delegation. If another client touches the file for which the
server has granted a delegation to another client, the delegation must
be recalled first and the caller gets a |NFS4ERR_DELAY|, and has to
re-try after a delay.

And this happens in this case, because the mounts I used for testing
are (intentionally [1]) treated as separate clients.

[1]=First it's my test setup, and secondly the ms-nfs41-client is
highly threaded to deal with the async nature of the Windows kernel
and NFSv4.x's nature of being able to handle requests in parallel, the
only limiting factor being TCP and the RPC implementations's locking.

Wireshark documented this nicely:
---- snip ----
No. Time Source Destination Protocol Length Info
1143 85.561066 10.49.202.232 10.49.202.230 NFS 318 V4 Call (Reply In
1145) remove REMOVE DH: 0x4b1e488f/caller.def
1144 85.561419 10.49.202.230 10.49.202.232 RPC 222 Continuation
1145 85.591706 10.49.202.230 10.49.202.232 NFS 174 V4 Reply (Call In
1143) remove REMOVE Status: NFS4ERR_DELAY
1146 85.615056 10.49.202.232 10.49.202.230 TCP 66 604 → 2049 [ACK]
Seq=6277 Ack=4741 Win=32763 Len=0 TSval=2579336 TSecr=3137316992
1147 85.646291 10.49.202.232 10.49.202.230 TCP 66 603 → 2049 [ACK]
Seq=22273 Ack=14733 Win=32768 Len=0 TSval=2579368 TSecr=3137317023
1149 86.099396 10.49.202.232 10.49.202.230 NFS 318 V4 Call remove
REMOVE DH: 0x4b1e488f/caller.def
1150 86.131692 10.49.202.230 10.49.202.232 NFS 174 V4 Reply (Call In
1149) remove REMOVE Status: NFS4ERR_DELAY
1151 86.177304 10.49.202.232 10.49.202.230 TCP 66 603 → 2049 [ACK]
Seq=22525 Ack=14841 Win=32767 Len=0 TSval=2579899 TSecr=3137317563
1158 87.146588 10.49.202.232 10.49.202.230 NFS 318 V4 Call (Reply In
1159) remove REMOVE DH: 0x4b1e488f/caller.def
1159 87.175843 10.49.202.230 10.49.202.232 NFS 174 V4 Reply (Call In
1158) remove REMOVE Status: NFS4ERR_DELAY
1160 87.224483 10.49.202.232 10.49.202.230 TCP 66 603 → 2049 [ACK]
Seq=22777 Ack=14949 Win=32767 Len=0 TSval=2580946 TSecr=3137318607
1170 88.677605 10.49.202.232 10.49.202.230 NFS 318 V4 Call (Reply In
1171) remove REMOVE DH: 0x4b1e488f/caller.def
1171 88.707859 10.49.202.230 10.49.202.232 NFS 174 V4 Reply (Call In
1170) remove REMOVE Status: NFS4ERR_DELAY
1172 88.755661 10.49.202.232 10.49.202.230 TCP 66 603 → 2049 [ACK]
Seq=23029 Ack=15057 Win=32766 Len=0 TSval=2582477 TSecr=3137320139
1179 89.271735 10.49.202.232 10.49.202.230 RPC 174 Continuation
1180 89.272124 10.49.202.232 10.49.202.230 NFS 306 V4 Call (Reply In
1182) delegreturn DELEGRETURN StateID: 0xa200
1181 89.272396 10.49.202.230 10.49.202.232 TCP 66 2049 → 604 [ACK]
Seq=4741 Ack=6625 Win=11624 Len=0 TSval=3137320703 TSecr=2582993
1182 89.272493 10.49.202.230 10.49.202.232 NFS 178 V4 Reply (Call In
1180) delegreturn DELEGRETURN
1183 89.318145 10.49.202.232 10.49.202.230 TCP 66 604 → 2049 [ACK]
Seq=6625 Ack=4853 Win=32763 Len=0 TSval=2583040 TSecr=3137320703
1204 90.708871 10.49.202.232 10.49.202.230 NFS 318 V4 Call remove
REMOVE DH: 0x4b1e488f/caller.def
1205 90.709587 10.49.202.230 10.49.202.232 NFS 314 V4 Reply (Call In
1204) remove REMOVE
1206 90.725892 10.49.202.232 10.49.202.230 NFS 314 V4 Call (Reply In
1208) remove REMOVE DH: 0x4b1e488f/cd.def
---- snip ----

So everything is working as expected, but I really need a FSCTL_* to
flush all delegations for accurate performance benchmarking.

Does the Win32 have a "flush cache"-|FSCTL_*|/|IOCTL_*| ?

----

Bye,
Roland
-- 
  __ .  . __
 (o.\ \/ /.o) roland DOT mainz AT nrubsig DOT org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

-- 
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

DMARC-Filter:	OpenDMARC Filter v1.4.2 delorie.com 51R4x2IU494368
Authentication-Results:	delorie.com; dmarc=pass (p=none dis=none) header.from=cygwin.com
Authentication-Results:	delorie.com; spf=pass smtp.mailfrom=cygwin.com
DKIM-Filter:	OpenDKIM Filter v2.11.0 delorie.com 51R4x2IU494368
Authentication-Results:	delorie.com;
	dkim=pass (1024-bit key, unprotected) header.d=cygwin.com header.i=@cygwin.com header.a=rsa-sha256 header.s=default header.b=JGW3jKbm
X-Recipient:	archive-cygwin AT delorie DOT com
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org 7E47A3858C2D
DKIM-Signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=cygwin.com;
	s=default; t=1740632340;
	bh=uRJialV+yW7mKURMZMRofNVVruqQekj7es6l2B4LT94=;
	h=References:In-Reply-To:Date:Subject:To:List-Id:List-Unsubscribe:
	List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:
	From;
	b=JGW3jKbmo5SzIQ6Be3RgmCMf5VCOQ3QDI0opbzMtzaqx2DENpWV/bYVLYG2BNa4eD
	Zk4YsQkc+NwGTdj364UANkm3NBAP+9JlGd/YkKShRrDfzLp2F7dzfibGhvGLH6ND6H
	bbjNiQvsA37NjEJDaFfYTn2hUVfRpoAMm6DxfxMk=
X-Original-To:	cygwin AT cygwin DOT com
Delivered-To:	cygwin AT cygwin DOT com
DMARC-Filter:	OpenDMARC Filter v1.4.2 sourceware.org A82ED3858D26
ARC-Filter:	OpenARC Filter v1.0.0 sourceware.org A82ED3858D26
ARC-Seal:	i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1740632316; cv=pass;
	b=ASm3BUJlTF9ctd2RFMuEcILLP+1HxN6sheikiNkbk8YM0fZJLhewOgsm26e9gB4t50BYVvNTXSkZ2xqkD67SLNNtBMmmX59q3gK1jntvNdLs4zSQECf6VsBcsyEHeFyAF7RGv4HGMJ1VS4B5PRVqIYCTb6mC2Wm1HlIvy8sYE6M=
ARC-Message-Signature:	i=2; a=rsa-sha256; d=sourceware.org; s=key;
	t=1740632316; c=relaxed/simple;
	bh=ylDOEHydorKehJxecXlMQLG006fDyPDOqISmnMpLCGI=;
	h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To;
	b=F63+ob44x7rJWem4KUJGc9MrE5shF6rFG/vuJ6QctTKjtWt7c+H/1PWnVzy9jYPh3PBKFdp8R8xrK1Q/dmQAvkyxFewp8yT+t667kHov6pFFdYx4o0G7ltj2uYO284BC7v5G2gimfFKS/SEANQ1VYjMXLTQ3tkp4ECn03QTUi3Q=
ARC-Authentication-Results:	i=2; server2.sourceware.org
DKIM-Filter:	OpenDKIM Filter v2.11.0 sourceware.org A82ED3858D26
X-Sender-Id:	dreamhost\|x-authsender\|gisburn AT nrubsig DOT org
ARC-Seal:	i=1; s=arc-2022; d=mailchannels.net; t=1740632313; a=rsa-sha256;
	cv=none;
	b=KyYgyjLutp4IGil/Nh1ZRJbOS/Cswj2nUoIbdpaixNV6uVQcEtBtOpBGms03bpYTbgRe78
	Ggxc3fJ9Gbkvaqi/YeJ587bjn++Dx5GDErwrvcDPd6UaJKvkm3DLzcB+6QCK3Bt5ery+C1
	Kf1FUOd6xKJdNUJG+o3yWCyMpzOaHAHwXRfQUWEEKhJittORVPyHRHbFZ6oGrmkyQNw3rE
	oishN1MN8U6GbhAFZmxth2F8YBMuCnJGyYss2hJBdBfieRVxTOtSamYzsoWub9ZCpW678S
	+AEYQex7tMybndhO9pWHMHJ6Rt7RbdydCUrbi43f2XN3pJGYiATcNCXKF2WYnw==
ARC-Message-Signature:	i=1; a=rsa-sha256; c=relaxed/relaxed;
	d=mailchannels.net; s=arc-2022; t=1740632313;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	to:to:cc:mime-version:mime-version:content-type:content-type:
	content-transfer-encoding:content-transfer-encoding:
	in-reply-to:in-reply-to:references:references:dkim-signature;
	bh=rnf+X2ayAhxFikvGOswn4fjuMS99B0A36bxzefhRFc4=;
	b=JSxJg3olz9wJ0TYgOSWDF44zyI/ohSCjRL/P6ucfh9L8QbVmSUpVNRcOiqU/pNPYYTuiS6
	XI0uwUvsj7DrgQ0nTDjAoN0pzMiy5MxEpNT9Z9H2zg0VerWBf1euEH2RGCIGp+9uE3Z7mI
	j5KDOwa29OSrkewoqNOpw8s65k/zLQrwoMPViCfSQoBW+Y/NomGfo+0pxYgnBmowGF75ea
	dkwidixJGxfR8n7LaNgaabFzXVQ65tg78+9pqrliadFnjKjMXTjJg9YzXIoOwglkwxZ6vO
	Gw5UHiJAcQ0BzEI/mOESFQZSWXH+ZR3wo/jtjlutT3Idg3WWv1djB+wjAD4Mtg==
ARC-Authentication-Results:	i=1; rspamd-79f778f6bc-cn65g;
	auth=pass smtp.auth=dreamhost smtp.mailfrom=roland DOT mainz AT nrubsig DOT org
X-Sender-Id:	dreamhost\|x-authsender\|gisburn AT nrubsig DOT org
X-MC-Relay:	Neutral
X-MailChannels-SenderId:	dreamhost\|x-authsender\|gisburn AT nrubsig DOT org
X-MailChannels-Auth-Id:	dreamhost
X-Cure-Illegal:	279a332e2d2a0c9f_1740632313669_1747148637
X-MC-Loop-Signature:	1740632313669:197476308
X-MC-Ingress-Time:	1740632313668
X-Gm-Message-State:	AOJu0Yxz5iUntzUwfFIAX1WDtrHxAAu7935dbFITJSKZDQ7TOaR8ceZB
	fZPxv1T7oYpZibmcdArk5FQ17H4K75h6tyv0gzl1LWFa+idD4nQZVxjZhJt6dYrklWHvMhm7Y3Y
	2DDBzzz57fhYcaUvGT0/2aHwVehQ=
X-Google-Smtp-Source:	AGHT+IGaSSlTnuojN6gZGj4IoVLZHJ5lS26ckTa8JFh1IyIl05DBMIQynsadVn61f4dK+mfqMWwsKzU2K58QKzpM8pk=
X-Received:	by 2002:a5d:47a9:0:b0:38f:2211:e628 with SMTP id
	ffacd0b85a97d-390d4f3ced2mr4333971f8f.20.1740632311678; Wed, 26 Feb 2025
	20:58:31 -0800 (PST)
MIME-Version:	1.0
References:	<CAKAoaQniATLryoaKnpFwYXcLOU4VViHE+fUHA_nTz6omxTZc4w AT mail DOT gmail DOT com>
	<Z795BWjJd7bPNgGj AT calimero DOT vinschen DOT de>
In-Reply-To:	<Z795BWjJd7bPNgGj@calimero.vinschen.de>
Date:	Thu, 27 Feb 2025 05:58:00 +0100
X-Gmail-Original-Message-ID:	<CAKAoaQkmBT-FNvLUCv=wNy6MXGiMxvkWnS74NB9h2U8RZ_wuTQ AT mail DOT gmail DOT com>
X-Gm-Features:	AQ5f1JrBAeg-05ncsBFft7tWNugz-W88EmZjhnaJ-xzxELZ8JprdUsME6Q01Rsw
Message-ID:	<CAKAoaQkmBT-FNvLUCv=wNy6MXGiMxvkWnS74NB9h2U8RZ_wuTQ@mail.gmail.com>
Subject:	Re: rm -Rfv on UNC path horribly slow in Cygwin
	3.6.0-0.404.g323729f654ae.x86_64 ...
To:	cygwin AT cygwin DOT com, ms-nfs41-client-devel AT lists DOT sourceforge DOT net
X-BeenThere:	cygwin AT cygwin DOT com
X-Mailman-Version:	2.1.30
List-Id:	General Cygwin discussions and problem reports <cygwin.cygwin.com>
List-Archive:	<https://cygwin.com/pipermail/cygwin/>
List-Post:	<mailto:cygwin AT cygwin DOT com>
List-Help:	<mailto:cygwin-request AT cygwin DOT com?subject=help>
List-Subscribe:	<https://cygwin.com/mailman/listinfo/cygwin>,
	<mailto:cygwin-request AT cygwin DOT com?subject=subscribe>
From:	Roland Mainz via Cygwin <cygwin AT cygwin DOT com>
Reply-To:	Roland Mainz <roland DOT mainz AT nrubsig DOT org>
Sender:	"Cygwin" <cygwin-bounces~archive-cygwin=delorie DOT com AT cygwin DOT com>
X-MIME-Autoconverted:	from base64 to 8bit by delorie.com id 51R4x2IU494368

webmaster	delorie software privacy
Copyright � 2019 by DJ Delorie	Updated Jul 2019