Fwd: [EXTERNAL] Re: [nfsv4] NFS over QUIC

Fri Sep 4 03:03:11 UTC 2020

---------- Forwarded message ---------
From: Steven French <Steven.French at microsoft.com>
Date: Thu, Sep 3, 2020 at 10:01 PM
Subject: RE: [EXTERNAL] Re: [nfsv4] NFS over QUIC
To: Chuck Lever <chucklever at gmail.com>, Bruce Fields <bfields at fieldses.org>
Cc: linux-nfs at vger.kernel.org <linux-nfs at vger.kernel.org>, NFSv4
<nfsv4 at ietf.org>, linux-cifs at vger.kernel.org
<linux-cifs at vger.kernel.org>, Ronnie Sahlberg <lsahlber at redhat.com>

We (open source SMB3 developers) have been thinking a lot about
SMB3.1.1 over QUIC especially after the interesting talks on this at
the last few SDC conferences e.g.

https://www.snia.org/sites/default/files/SDC/2019/presentations/SMB/Li_Frank_Surfing_the_World_Wide_File_SMB3_Improvements_for_Safe_and_Efficient_Internet_Access.pdf
(slides 8 and following).

And some of the broader coverage:
https://redmondmag.com/articles/2020/03/02/microsoft-smb-over-quic-to-windows.aspx

I have gotten a chance to talk with some of the developers of the
recently open sourced quic libraries
(https://github.com/microsoft/msquic) and there are various code style
issues that would prevent it being merged into Linux kernel without
significant changes (not just to fix camelCase) but ... it is a very
very promising place to start.

One of the ideas being bounced around is to do this in stages - first
try to cleanup the kernel style issues with the open sourced msquic
project and merge it experimentally into the Linux client (cifs.ko)
since there should be SMB3.1.1. servers to test it against reasonably
soon (presumably we will know a lot more after the 2020 SNIA SDC talk
on SMB3.1.1 over QUIC in a few weeks updating us on status
https://www.snia.org/events/storage-developer/2020/abstracts#smb-Dantuluri)
after we are convinced that the quic driver interoperates ok then move
it into the Linux networking stack and let the networking experts
clean it up more and remove the 'experimental' tag from it when
satisfied.

More discussion is needed on this so we should add samba-technical and
some of the network experts as well.

I am very excited about the possibilities opened up by a decent quic
kernel driver, but the current cross platform open sourced driver is
fairly large - ~90KLOC so will have to be trimmed down (and a lot of
boring changes to convert to kernel style needed).

-----Original Message-----
From: Chuck Lever <chucklever at gmail.com>
Sent: Thursday, September 3, 2020 7:53 PM
To: Bruce Fields <bfields at fieldses.org>
Cc: Linux NFS Mailing List <linux-nfs at vger.kernel.org>; NFSv4
<nfsv4 at ietf.org>; Steven French <Steven.French at microsoft.com>
Subject: [EXTERNAL] Re: [nfsv4] NFS over QUIC

> On Sep 3, 2020, at 8:32 PM, Bruce Fields <bfields at fieldses.org> wrote:
>
> On Thu, Sep 03, 2020 at 07:48:19PM -0400, Chuck Lever wrote:
>> Hi Bruce-
>>
>>> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <bfields at fieldses.org> wrote:
>>>
>>> I've been thinking about what might be required for NFS to run over
>>> QUIC.
>>>
>>> Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
>>>
>>> I don't have real plans.  For Linux, I don't even know if there's a
>>> kernel QUIC implementation planned yet.
>>>
>>> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fda
>>> tatracker.ietf.org%2Fdoc%2Fdraft-cel-nfsv4-rpc-tls%2F&data=02%7C
>>> 01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee6
>>> 3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&am
>>> p;sdata=pOnlPJcTkyJiqQm%2BCbo7dbOAqHMKFIi01qrShVZTH%2Bk%3D&reser
>>> ved=0
>>
>> The link to the latest version of that document is
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdat
>> atracker.ietf.org%2Fdoc%2Fdraft-ietf-nfsv4-rpc-tls%2F&data=02%7C0
>> 1%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%
>> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&s
>> data=YYBPryN5sFJApWm%2BkQvmjYzGXxjxI1fXRUWVXvVGE24%3D&reserved=0
>>
>>> For example, section 4.3, which explains how to authenticate on top
>>> of an already-encrypted session, should also apply to QUIC.
>>
>> Most of the document's content will be re-used for defining
>> RPC-over-QUIC, for example the ALPN defined in Section 8.2.
>> Lars Eggert, a chair of the QUIC WG, has been helping guide the
>> RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC
>> becomes more mature.
>>
>> I thought the plan was to write a specification of RPC-over- QUIC as
>> a new RPC transport type with a netid and uaddr along with a
>> definition of the transport semantics (a la TI-RPC).
>> The document would need to explain record marking, peer
>> authentication, how to use multi-path and multi-stream support, and
>> so on.
>>
>> Making NFS work on that transport should then be straightforward
>> enough that perhaps additional standards work wouldn't be necessary.
>
> Oh, OK, good.  Sounds like you're way ahead of me, then, I didn't know
> there was a plan.

That's all there is for the moment! :-)

> --b.
>
>>> QUIC runs over UDP, so I think all that would be required to
>>> negotiate support would be to attempt a QUIC connection to port 2049.
>>>
>>> The "Transport Layers" section in the NFS RFCs:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Frfc5661%23section-2.9&data=02%7C01%7CSteve
>>> n.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%7C72f988
>>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=ej
>>> gN6OuHrcDMzejeEKbaP5UiD1GAxHfQcPyLZZ45vbo%3D&reserved=0
>>>
>>> requires transports support reliable and in-order transmission,
>>> forbids clients from retrying a request unless a connection is lost,
>>> and forbids servers from dropping a request without closing a
>>> connection.  I'm still vague on how those requirements interact with
>>> QUIC's connection management and 0-RTT reconnection.
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.ietf.org%2Fid%2Fdraft-ietf-quic-applicability-07.txt&data=02%7
>>> C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee
>>> 63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=TZeoR%2FT22hg%2BkG0LllDEttlgdzv%2FAsH8PeXbdEHQ9Mk%3D&reserved=0 looks useful, as a guide for applications running over QUIC.  It warns that connections can time out fairly quickly.  For timely callbacks over NFS sessions, that means we need the client to ping the server regularly.
>>> Sounds like that's what they do for HTTP/QUIC to make server push
>>> notifications work:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Fdraft-ietf-quic-http-09%23section-5&data=0
>>> 2%7C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506
>>> cee63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63734777607589360
>>> 4&sdata=rF91Tq8OXHeN7W5iuSbtYotq1XBC4Zc8DGEGzwPUuTg%3D&reser
>>> ved=0
>>>
>>>     HTTP clients are expected to use QUIC PING frames to keep
>>>     connections open.  Servers SHOULD NOT use PING frames to keep a
>>>     connection open.  A client SHOULD NOT use PING frames for this
>>>     purpose unless there are responses outstanding for requests or
>>>     server pushes.
>>>
>>> QUIC allows multiple streams per connection--I wonder how we might
>>> use that.  RFC 5661 justifies the requirement for an ordered transport with:
>>>
>>>     Ordered delivery simplifies detection of transmit errors, and
>>>     simplifies the sending of arbitrary sized requests and responses
>>>     via the record marking protocol.
>>>
>>> So as long as we don't try to split a single RPC among streams, I
>>> think we're OK.  Would a stream per session slot be reasonable?  I'm
>>> not sure what the cost of a stream is.
>>>
>>> Do we need to add a new universal address type so the protocol can
>>> specify QUIC endpoints when necessary?  (For server-to-server-copy,
>>> pnfs file layouts, fs_locations, etc.)  All QUIC needs is an IP
>>> address and maybe a port, so maybe the existing UDP/TCP addresses are enough?
>>
>> --
>> Chuck Lever
>> chucklever at gmail.com

--
Chuck Lever
chucklever at gmail.com

-- 
Thanks,

Steve