Using sendfile for reducing CPU utilization

[Jeremy Allison]

On Sat, Jul 20, 2002 at 03:57:27AM +0930, Richard Sharpe wrote:
> Hi,
> 
> I did some testing a couple of days ago with sendfile under FreeBSD and 
> Samba, and observed that in pulling 500MB of data from files on a server I 
> could achieve about 45MB/s over GigE (I have a problem in my switch, I 
> think) but that CPU utilization was at 100% without sendfile and 50% with 
> sendfile.
> 
> This is a big improvement, but there are potential problems. These 
> problems are due to fact that once you start sending, if anything changes, 
> you cannot stop and say oops, I screwed up. If you promised to send 64kB, 
> you have to send that or drop the connection.
> 
> The problem is exacerbated by the fact that the Linux sendfile call does 
> not seem to allow you to specify the header on the call, so you are forced 
> to send the header from userspace and the data from the kernel, and you 
> therefor introduce a window during which things can go wrong. For example, 
> the file can be truncated or deleted, and I don't think SMB allows you to 
> send zeros for parts of the file that are not there.
> 
> The way Samba does this normally is that it assembles the data in 
> userspace and then sends the response. If a problem occurs, it can send an 
> error response and does not need to drop the connection.
> 
> However, FreeBSD's sendfile implementation allows you to specify the 
> header to be sent, and I believe that it also locks the vnode prior to 
> trying anything so you are protected against the file changing under you. 
> The only other thing that it should do is to pin all the pages before 
> trying to send anything. Thus, if any error occurs, it can return to the 
> user saying, sorry, I could not do this, you send an error message.
> 
> A useful extension for memory constrained situations might be that the 
> sendfile call should also take a timeout field that says, if you cannot 
> start sending before this time expires (ie, you cannot assemble all the 
> data before this time) please don't send anything and let me deal with it 
> the slow way.
>  
> So, I think I can get the benefits of sendfile and recvfile, which is, a 
> very large reduction in CPU (caused by copying the data to and from 
> userland) for clients that do lots of reading and writing), and thus 
> support more clients.
> 
> I will send more info as implementation proceeds.

Richard - can you get this patch to me asap ? If so we can add it
with an --enable-sendfile option to allow people to experiment with
it for 2.2.6.

Thanks,

	Jeremy.