[linux-cifs-client] Re: sequential read performance - read ahead ?

[James Roper]

Hi Alexander,

It's a long time since I've had a look at cifs, vfs, or done any kernel development, I'm now a Java developer, so some of what I say here will probably be wrong, but I'll give my thoughts anyway.

It sounds like you're willing to implement some changes to cifs vfs just for your needs, regardless of whether anyone else will ever use them or whether they'll be included in the kernel (which they probably won't, as the response would be if you want read ahead, then use a 2.6 kernel).  Taking this into account, and assuming that you're only ever going to sequentially access files with CIFS, you can implement a very simple read ahead solution that would work great for sequential access, but at a huge cost to random access.

Now I can't remember anything about the VFS or CIFS data structures, but I think that there is a structure somewhere that represents an open file in VFS, this gets passed in some form to cifs_readpage(), and there is a pointer in there that points to a CIFS structure.  To this structure you would add a pointer to a linked list, lets called it readahead_list, and its items are a structure that contains the number of the page, and the page data.  Here would be your logic in cifs_readpage():

if readahead_list != null:
   loop through readahead_list:
      if the current item is for the page we're looking for:
         copy it into the return buffer
         free the memory the item took
         remove the page from the list
         break out of the loop
      else
         free the memory the item took
         remove the page from the list
      end if
   end loop
end if
if we haven't found the page yet
   issue multi page read
   place all bar the first page into the readahead_list
   copy the first page into the return buffer
end if

Like I said, this will work fine for sequential access, but incredibly poorly for random access.  Note there are all sorts of other things you'll need to take into account, partial page reads, concurrency etc.

Now, about that line that says "issue multi page read".  There are two possibilities for that.  One is to use my patch to issue asynchronous 4KB requests, the other is to issue requests greater than 4KB, CIFS I think supports reads up to 128KB.  The second is simpler, though it requires some modifications to the core of CIFS because it can only receive 4KB responses.  I think in the current 2.6 kernel CIFS is able to handle larger read requests, so that would be the first place to look on how to do that.

Regards,

James

----- Original Message -----
From: Alexander Indenbaum <alexander.indenbaum at gmail.com>
Date: Wednesday, September 17, 2008 2:43 am
Subject: Re: sequential read performance - read ahead ?
To: James Roper <u3205097 at anu.edu.au>
Cc: "sfrench at samba.org" <sfrench at samba.org>, "linux-cifs-client at lists.samba.org" <linux-cifs-client at lists.samba.org>

> Hello James,
> 
> Thank you for your response! See my comments bellow.
> 
> On Tue, Sep 16, 2008 at 2:55 AM, James Roper 
> <u3205097 at anu.edu.au> wrote:
> > Hi Alexander,
> >
> > The concept behind read ahead is that you can minimise the 
> penalty caused by the in flight time as the data is traveling 
> between the client and the server, by doing other things while 
> you're waiting, eg by sending more requests. If it's the clients 
> CPU speed that is slowing you down, then it's likely that the 
> client isn't spending much time waiting in comparison to the 
> amount of time it's spending processing, so the in flight time 
> would not be that large a penalty.  So reading ahead 
> probably won't help you that much. Additionally, it would be no 
> trivial task to implement read ahead in the cifs client in a 2.4 
> kernel, you would have to do your own sequential read detection, 
> caching of read ahead pages between vfs reads, and cleanup of 
> unread pages.
> >
> 
> In my test environment I do not see that slow CPU is fully utilized
> during CIFS transfers. Actually the CPU has a lot of spare 
> cycles. I
> suppose that we see degradation of sequential read performance on
> slow CPU because there is no TCP like "sliding window" and
> request<->response cycle takes longer due to slugish CPU. 
> After all
> one slow CPU tick takes as long as modern CPU 10 ticks :).  
> So this
> way neither CPU nor network link are fully utilized. That's why I
> still hope "read-ahead" could improve the read throughput. I have
> almost no experience in Linux VM/VFS so I'm not really sure how to
> implement file's memory page "pre-fetch". In case I get the
> "pre-fetched" page from the server that application has not yet issued
> cifs_readpage() for it. How can I allocate new page and 
> associate with
> the file, so VM/VFS will not try to fetch it over network again? Any
> pointer to similar code will be appreciated.
> 
> > I'm not sure about other performance improvements, it's been a 
> long time since I've looked at cifs vfs, but there may be some 
> memcopys that can be eliminated, that has the potential to make 
> a significant difference.
> 
> Thank you for this idea. Please see the attached "very raw and
> untested" patch against 1.20c. It should save memcpy of data on
> receive path. Unfortunately, I did not notice any significant
> performance improvement :)
> 
> >
> > Regards,
> >
> > James Roper
> >
> > Sent from my iPhone
> 
> Thank you for your help,
> Alexander Indenbaum
> 
> >
> > On 16/09/2008, at 2:17, Alexander Indenbaum  wrote:
> >
> >> Hello,
> >>
> >> I'm working with two kinds of embedded Linux boxes powered by 2.4
> >> kernel and cifs  1.20c connected to Windows CIFS server 
> via 100Mbps
> >> Ethernet switch. Our usage pattern is sequential read of 
> large files
> >> from CIFS shares ("streaming"). I'm using  simple "time dd
> >> if=CIFS_FILE of=/dev/null" benchmark to measure read 
> performance. Here
> >> are results:
> >>
> >> CPU 
> strength/BogoMIPS        Read 
> Throughput/Mbps>> 
> 266                                       5
> >> 
> 2988                                     14.5
> >>
> >> As you see, read performance is heavily influenced by CPU 
> strength :)
> >> I'm looking for a ways to improve read performance especially 
> on  weak
> >> 266 BogoMIPS box. After doing some reading I started to 
> believe the
> >> best way to do so is to implement some kind of read-ahead 
> mechanism. I
> >> saw 2.5 kernel patch by James Roper doing so over vfs_readpages
> >> interface - http://marc.info/?l=linux-cifs-client&m=112665626721358
> >> I've tried to integrate the patch but 2.4 kernel does not have
> >> vfs_readpages interface :(.
> >>
> >> So what do you think? Is it possible to implement CIFS read-
> ahead over
> >> 2.4 VFS? How?
> >> Maybe other ideas for performance optimization?
> >>
> >> Thank you for your help,
> >> Alexander Indenbaum