Using sendfile for reducing CPU utilization

[Richard Sharpe]

Hi,

I did some testing a couple of days ago with sendfile under FreeBSD and 
Samba, and observed that in pulling 500MB of data from files on a server I 
could achieve about 45MB/s over GigE (I have a problem in my switch, I 
think) but that CPU utilization was at 100% without sendfile and 50% with 
sendfile.

This is a big improvement, but there are potential problems. These 
problems are due to fact that once you start sending, if anything changes, 
you cannot stop and say oops, I screwed up. If you promised to send 64kB, 
you have to send that or drop the connection.

The problem is exacerbated by the fact that the Linux sendfile call does 
not seem to allow you to specify the header on the call, so you are forced 
to send the header from userspace and the data from the kernel, and you 
therefor introduce a window during which things can go wrong. For example, 
the file can be truncated or deleted, and I don't think SMB allows you to 
send zeros for parts of the file that are not there.

The way Samba does this normally is that it assembles the data in 
userspace and then sends the response. If a problem occurs, it can send an 
error response and does not need to drop the connection.

However, FreeBSD's sendfile implementation allows you to specify the 
header to be sent, and I believe that it also locks the vnode prior to 
trying anything so you are protected against the file changing under you. 
The only other thing that it should do is to pin all the pages before 
trying to send anything. Thus, if any error occurs, it can return to the 
user saying, sorry, I could not do this, you send an error message.

A useful extension for memory constrained situations might be that the 
sendfile call should also take a timeout field that says, if you cannot 
start sending before this time expires (ie, you cannot assemble all the 
data before this time) please don't send anything and let me deal with it 
the slow way.
 
So, I think I can get the benefits of sendfile and recvfile, which is, a 
very large reduction in CPU (caused by copying the data to and from 
userland) for clients that do lots of reading and writing), and thus 
support more clients.

I will send more info as implementation proceeds.

Regards
-----
Richard Sharpe, rsharpe at ns.aus.com, rsharpe at samba.org, 
sharpe at ethereal.com