[PATCH] two Systemtap scripts that instrument gencache and winbindd (was: RFC: maybe add two Systemtap scripts that instrument gencache and winbindd?)

Ralph Böhme slow at samba.org
Fri Jul 14 11:48:18 UTC 2017


Hi all,

On Fri, Jul 07, 2017 at 06:59:46PM +0200, Ralph Böhme via samba-technical wrote:
> Over the last few days, after realizing how damn simple it is to instrument code
> with Systemtap with DWARF debug symbols present, I quickly implemented the
> attached two Systemtap scripts.
>
> ...

updated patchset attached that moves the scripts from script/ to
examples/systemtap/. Seems like a better place for them to live.

Cheerio!
-slow
-------------- next part --------------
From 185a3d265c79b63814109417fd1ab50ea10a629a Mon Sep 17 00:00:00 2001
From: Ralph Boehme <slow at samba.org>
Date: Fri, 30 Jun 2017 12:59:37 +0200
Subject: [PATCH 1/2] examples: add gencache.stp

Add a Systemtap script to profile gencache.

Usage:

- profile a single smbd process:
  # stap -x 22225 gencache.stp smbd

- profile all winbindd proceses:
  # stap gencache.stp winbindd

Signed-off-by: Ralph Boehme <slow at samba.org>
---
 examples/systemtap/gencache.stp | 124 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100755 examples/systemtap/gencache.stp

diff --git a/examples/systemtap/gencache.stp b/examples/systemtap/gencache.stp
new file mode 100755
index 0000000..c0c5163
--- /dev/null
+++ b/examples/systemtap/gencache.stp
@@ -0,0 +1,124 @@
+#!/usr/bin/stap
+#
+# Systemtap script to instrument the Samba gencache subsystem
+#
+# Usage:
+#
+# Instrument all smbd processes:
+# # stap gencache.stp smbd
+#
+# Instrument all winbindd processes:
+# # stap gencache.stp winbindd
+#
+# Instrument a specific smbd process:
+# # stap -x PID gencache.stp smbd
+#
+# Instrument a specific winbindd process:
+# # stap -x PID gencache.stp winbindd
+#
+
+global running, intervals
+
+probe begin {
+        printf("Collecting data, press ctrl-C to stop... ")
+}
+
+probe process(@1).library("*").function("gencache_parse") {
+        running["gencache_parse", tid()] = gettimeofday_us()
+}
+
+probe process(@1).library("*").function("gencache_parse").return {
+	if (!(["gencache_parse", tid()] in running))
+		next
+
+        end = gettimeofday_us()
+        begin = running["gencache_parse", tid()]
+        delete running["gencache_parse", tid()]
+
+	duration = end - begin
+	intervals["gencache_parse"] <<< duration
+}
+
+probe process(@1).library("*").function("gencache_get_data_blob") {
+        running["gencache_get_data_blob", tid()] = gettimeofday_us()
+}
+
+probe process(@1).library("*").function("gencache_get_data_blob").return {
+	if (!(["gencache_get_data_blob", tid()] in running))
+		next
+
+        end = gettimeofday_us()
+        begin = running["gencache_get_data_blob", tid()]
+        delete running["gencache_get_data_blob", tid()]
+
+	duration = end - begin
+	intervals["gencache_get_data_blob"] <<< duration
+}
+
+probe process(@1).library("*").function("gencache_set_data_blob") {
+        running["gencache_set_data_blob", tid()] = gettimeofday_us()
+}
+
+probe process(@1).library("*").function("gencache_set_data_blob").return {
+	if (!(["gencache_set_data_blob", tid()] in running))
+		next
+
+        end = gettimeofday_us()
+        begin = running["gencache_set_data_blob", tid()]
+        delete running["gencache_set_data_blob", tid()]
+
+	duration = end - begin
+	intervals["gencache_set_data_blob"] <<< duration
+}
+
+probe process(@1).library("*").function("gencache_del") {
+        running["gencache_del", tid()] = gettimeofday_us()
+}
+
+probe process(@1).library("*").function("gencache_del").return {
+	if (!(["gencache_del", tid()] in running))
+		next
+
+        end = gettimeofday_us()
+        begin = running["gencache_del", tid()]
+        delete running["gencache_del", tid()]
+
+	duration = end - begin
+	intervals["gencache_del"] <<< duration
+}
+
+probe process(@1).library("*").function("gencache_stabilize") {
+        running["gencache_stabilize", tid()] = gettimeofday_us()
+}
+
+probe process(@1).library("*").function("gencache_stabilize").return {
+	if (!(["gencache_stabilize", tid()] in running))
+		next
+
+        end = gettimeofday_us()
+        begin = running["gencache_stabilize", tid()]
+        delete running["gencache_stabilize", tid()]
+
+	duration = end - begin
+	intervals["gencache_stabilize"] <<< duration
+}
+
+probe end {
+        printf("\n\n")
+
+        foreach ([name] in intervals) {
+                printf("%-30s count: %d sum: %d us (min: %d us avg: %d us max: %d us)\n",
+                       name,
+                       @count(intervals[name]),
+                       @sum(intervals[name]),
+                       @min(intervals[name]),
+                       @avg(intervals[name]),
+                       @max(intervals[name]))
+        }
+
+        printf("\n")
+        foreach ([name] in intervals) {
+                printf("%s time distribution histogram:\n", name)
+                println(@hist_log(intervals[name]))
+        }
+}
-- 
2.9.4


From 19d6ea7eda9e71f8ce4684a90a72a1f16c4fe44c Mon Sep 17 00:00:00 2001
From: Ralph Boehme <slow at samba.org>
Date: Fri, 30 Jun 2017 19:37:03 +0200
Subject: [PATCH 2/2] examples: add winbindd.stp and a shell script to generate
 it

Usage:

  Instrument all winbindd processes:
  # stap winbindd.stp

  Instrument a specific winbindd process:
  # stap -x PID winbindd.stp

Regenerate winbindd.stp:

  $ examples/generate-winbindd.stp.sh

Example output:

  # stap winbindd.stp
  Collecting data, press ctrl-C to stop... ^C

  Winbind request service time
  ============================
  winbindd_getpwnam_send                   count:    99, sum:   6229 ms (min:   2669 us, avg:  62921 us, max: 157907 us)

  Winbind request runtime
  =======================
  winbindd_getpwnam_send                   count:    99, sum:      3 ms (min:     21 us, avg:     36 us, max:     77 us)

  Winbind domain-child request service time
  =========================================
  _wbint_LookupName                        count:    99, sum:   1403 ms (min:    619 us, avg:  14181 us, max: 136613 us)
  _wbint_GetNssInfo                        count:    99, sum:      0 ms (min:      2 us, avg:      3 us, max:      6 us)
  _wbint_LookupSid                         count:   102, sum:     49 ms (min:     13 us, avg:    481 us, max:   6315 us)
  _wbint_Sids2UnixIDs                      count:   101, sum:      2 ms (min:     18 us, avg:     29 us, max:     49 us)
  _wbint_LookupSids                        count:   101, sum:     84 ms (min:    411 us, avg:    838 us, max:   3524 us)

  Winbind domain-child AD-backend service time
  ============================================
  sid_to_name                              count:    56, sum:     45 ms (min:    431 us, avg:    816 us, max:   6275 us)
  sequence_number                          count:    12, sum:   1209 ms (min:  46618 us, avg: 100803 us, max: 131439 us)
  name_to_sid                              count:    99, sum:    176 ms (min:    547 us, avg:   1781 us, max:   9866 us)

  ...

Signed-off-by: Ralph Boehme <slow at samba.org>
---
 examples/systemtap/generate-winbindd.stp.sh |  310 +++
 examples/systemtap/winbindd.stp             | 2872 +++++++++++++++++++++++++++
 2 files changed, 3182 insertions(+)
 create mode 100755 examples/systemtap/generate-winbindd.stp.sh
 create mode 100644 examples/systemtap/winbindd.stp

diff --git a/examples/systemtap/generate-winbindd.stp.sh b/examples/systemtap/generate-winbindd.stp.sh
new file mode 100755
index 0000000..377e474
--- /dev/null
+++ b/examples/systemtap/generate-winbindd.stp.sh
@@ -0,0 +1,310 @@
+#!/bin/sh
+
+outfile="$(dirname $0)/winbindd.stp"
+
+child_funcs="winbindd_dual_ping
+winbindd_dual_list_trusted_domains
+winbindd_dual_init_connection
+winbindd_dual_pam_auth
+winbindd_dual_pam_auth_crap
+winbindd_dual_pam_logoff
+winbindd_dual_pam_chng_pswd_auth_crap
+winbindd_dual_pam_chauthtok
+_wbint_LookupSid
+_wbint_LookupSids
+_wbint_LookupName
+_wbint_Sids2UnixIDs
+_wbint_UnixIDs2Sids
+_wbint_AllocateUid
+_wbint_AllocateGid
+_wbint_GetNssInfo
+_wbint_LookupUserAliases
+_wbint_LookupUserGroups
+_wbint_QuerySequenceNumber
+_wbint_LookupGroupMembers
+_wbint_QueryGroupList
+_wbint_QueryUserRidList
+_wbint_DsGetDcName
+_wbint_LookupRids
+_wbint_CheckMachineAccount
+_wbint_ChangeMachineAccount
+_wbint_PingDc"
+
+async_funcs="wb_ping
+winbindd_lookupsid
+winbindd_lookupsids
+winbindd_lookupname
+winbindd_sids_to_xids
+winbindd_xids_to_sids
+winbindd_getpwsid
+winbindd_getpwnam
+winbindd_getpwuid
+winbindd_getsidaliases
+winbindd_getuserdomgroups
+winbindd_getgroups
+winbindd_show_sequence
+winbindd_getgrgid
+winbindd_getgrnam
+winbindd_getusersids
+winbindd_lookuprids
+winbindd_setpwent
+winbindd_getpwent
+winbindd_endpwent
+winbindd_dsgetdcname
+winbindd_getdcname
+winbindd_setgrent
+winbindd_getgrent
+winbindd_endgrent
+winbindd_list_users
+winbindd_list_groups
+winbindd_check_machine_acct
+winbindd_ping_dc
+winbindd_pam_auth
+winbindd_pam_logoff
+winbindd_pam_chauthtok
+winbindd_pam_chng_pswd_auth_crap
+winbindd_wins_byip
+winbindd_wins_byname
+winbindd_allocate_uid
+winbindd_allocate_gid
+winbindd_change_machine_acct
+winbindd_pam_auth_crap"
+
+backend_funcs="query_user_list
+enum_dom_groups
+enum_local_groups
+name_to_sid
+sid_to_name
+rids_to_names
+lookup_usergroups
+lookup_useraliases
+lookup_groupmem
+sequence_number
+lockout_policy
+password_policy
+trusted_domains"
+
+header='#!/usr/bin/stap
+#
+# Systemtap script to instrument winbindd
+#
+'"# Generated by script/$(basename $0) on $(date), do not edit
+#"'
+# Usage:
+#
+# Instrument all winbindd processes:
+# # stap winbindd.stp
+#
+# Instrument a specific winbindd process:
+# # stap -x PID winbindd.stp
+#
+
+global dc_running, dc_svctime
+global backend_running, backend_svctime
+global send_running, recv_running
+global start_time, idle_time
+global async_svctime, async_runtime
+
+probe begin {
+	printf("Collecting data, press ctrl-C to stop... ")
+}'
+
+domchild_req_template='
+#
+# winbind domain child function XXX
+#
+
+probe process("winbindd").function("XXX") {
+	dc_running[tid(), "XXX"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("XXX").return {
+	if (!([tid(), "XXX"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "XXX"]
+	delete dc_running[tid(), "XXX"]
+
+	duration = end - begin
+	dc_svctime["XXX"] <<< duration
+}'
+
+backend_req_template='
+#
+# winbind domain child backend function XXX
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("XXX at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "XXX"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("XXX at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "XXX"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "XXX"]
+	delete backend_running[tid(), "XXX"]
+
+	duration = end - begin
+	backend_svctime["XXX"] <<< duration
+}'
+
+async_req_template='
+#
+# winbind async function XXX
+#
+
+probe process("winbindd").function("XXX_send") {
+	send_running["XXX_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("XXX_send").return {
+	if (!(["XXX_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["XXX_send"]
+	delete send_running["XXX_send"]
+
+	start_time["XXX_send", $return] = start
+	idle_time["XXX_send", $return] = end
+}
+
+probe process("winbindd").function("XXX_recv") {
+	if (!(["XXX_send", $req] in start_time))
+		next
+
+	recv_running["XXX_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("XXX_recv").return {
+	if (!(["XXX_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["XXX_recv"]
+	delete recv_running["XXX_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["XXX_send", req]
+	delete start_time["XXX_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["XXX_send", req]
+	delete idle_time["XXX_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["XXX_send"] <<< svctime
+	async_runtime["XXX_send"] <<< runtime
+}'
+
+footer='
+probe end {
+	printf("\n\n")
+
+	printf("Winbind request service time\n")
+	printf("============================\n")
+	foreach ([name] in async_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(async_svctime[name]),
+		       @sum(async_svctime[name]) / 1000,
+		       @min(async_svctime[name]),
+		       @avg(async_svctime[name]),
+		       @max(async_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request runtime\n")
+	printf("=======================\n")
+	foreach ([name] in async_runtime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(async_runtime[name]),
+		       @sum(async_runtime[name]) / 1000,
+		       @min(async_runtime[name]),
+		       @avg(async_runtime[name]),
+		       @max(async_runtime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind domain-child request service time\n")
+	printf("=========================================\n")
+	foreach ([name] in dc_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(dc_svctime[name]),
+		       @sum(dc_svctime[name]) / 1000,
+		       @min(dc_svctime[name]),
+		       @avg(dc_svctime[name]),
+		       @max(dc_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind domain-child AD-backend service time\n")
+	printf("============================================\n")
+	foreach ([name] in backend_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(backend_svctime[name]),
+		       @sum(backend_svctime[name]) / 1000,
+		       @min(backend_svctime[name]),
+		       @avg(backend_svctime[name]),
+		       @max(backend_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request service time distributions (us)\n")
+	printf("===============================================\n")
+	foreach ([name] in async_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(async_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request runtime distributions (us)\n")
+	printf("==========================================\n")
+	foreach ([name] in async_runtime) {
+		printf("%s:\n", name);
+		println(@hist_log(async_runtime[name]))
+	}
+
+	printf("Winbind domain-child request service time distributions (us)\n")
+	printf("============================================================\n")
+	foreach ([name] in dc_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(dc_svctime[name]))
+	}
+
+	printf("Winbind domain-child backend service time distributions (us)\n")
+	printf("============================================================\n")
+	foreach ([name] in backend_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(backend_svctime[name]))
+	}
+}'
+
+cat <<EOF > $outfile
+$header
+EOF
+
+printf "$child_funcs\n" | while read func ; do
+    printf "$domchild_req_template\n" | sed -e s/XXX/$func/g >> $outfile
+done
+
+printf "$backend_funcs\n" | while read func ; do
+    printf "$backend_req_template\n" | sed -e "s|XXX|$func|g" >> $outfile
+done
+
+printf "$async_funcs\n" | while read func ; do
+    printf "$async_req_template\n" | sed -e s/XXX/$func/g >> $outfile
+done
+
+cat <<EOF >>$outfile
+$footer
+EOF
diff --git a/examples/systemtap/winbindd.stp b/examples/systemtap/winbindd.stp
new file mode 100644
index 0000000..c420f88
--- /dev/null
+++ b/examples/systemtap/winbindd.stp
@@ -0,0 +1,2872 @@
+#!/usr/bin/stap
+#
+# Systemtap script to instrument winbindd
+#
+# Generated by script/generate-winbindd.stp.sh on Fri Jul  7 18:27:30 CEST 2017, do not edit
+#
+# Usage:
+#
+# Instrument all winbindd processes:
+# # stap winbindd.stp
+#
+# Instrument a specific winbindd process:
+# # stap -x PID winbindd.stp
+#
+
+global dc_running, dc_svctime
+global backend_running, backend_svctime
+global send_running, recv_running
+global start_time, idle_time
+global async_svctime, async_runtime
+
+probe begin {
+	printf("Collecting data, press ctrl-C to stop... ")
+}
+
+#
+# winbind domain child function winbindd_dual_ping
+#
+
+probe process("winbindd").function("winbindd_dual_ping") {
+	dc_running[tid(), "winbindd_dual_ping"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_ping").return {
+	if (!([tid(), "winbindd_dual_ping"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_ping"]
+	delete dc_running[tid(), "winbindd_dual_ping"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_ping"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_list_trusted_domains
+#
+
+probe process("winbindd").function("winbindd_dual_list_trusted_domains") {
+	dc_running[tid(), "winbindd_dual_list_trusted_domains"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_list_trusted_domains").return {
+	if (!([tid(), "winbindd_dual_list_trusted_domains"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_list_trusted_domains"]
+	delete dc_running[tid(), "winbindd_dual_list_trusted_domains"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_list_trusted_domains"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_init_connection
+#
+
+probe process("winbindd").function("winbindd_dual_init_connection") {
+	dc_running[tid(), "winbindd_dual_init_connection"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_init_connection").return {
+	if (!([tid(), "winbindd_dual_init_connection"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_init_connection"]
+	delete dc_running[tid(), "winbindd_dual_init_connection"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_init_connection"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_pam_auth
+#
+
+probe process("winbindd").function("winbindd_dual_pam_auth") {
+	dc_running[tid(), "winbindd_dual_pam_auth"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_pam_auth").return {
+	if (!([tid(), "winbindd_dual_pam_auth"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_pam_auth"]
+	delete dc_running[tid(), "winbindd_dual_pam_auth"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_pam_auth"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_pam_auth_crap
+#
+
+probe process("winbindd").function("winbindd_dual_pam_auth_crap") {
+	dc_running[tid(), "winbindd_dual_pam_auth_crap"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_pam_auth_crap").return {
+	if (!([tid(), "winbindd_dual_pam_auth_crap"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_pam_auth_crap"]
+	delete dc_running[tid(), "winbindd_dual_pam_auth_crap"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_pam_auth_crap"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_pam_logoff
+#
+
+probe process("winbindd").function("winbindd_dual_pam_logoff") {
+	dc_running[tid(), "winbindd_dual_pam_logoff"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_pam_logoff").return {
+	if (!([tid(), "winbindd_dual_pam_logoff"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_pam_logoff"]
+	delete dc_running[tid(), "winbindd_dual_pam_logoff"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_pam_logoff"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_pam_chng_pswd_auth_crap
+#
+
+probe process("winbindd").function("winbindd_dual_pam_chng_pswd_auth_crap") {
+	dc_running[tid(), "winbindd_dual_pam_chng_pswd_auth_crap"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_pam_chng_pswd_auth_crap").return {
+	if (!([tid(), "winbindd_dual_pam_chng_pswd_auth_crap"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_pam_chng_pswd_auth_crap"]
+	delete dc_running[tid(), "winbindd_dual_pam_chng_pswd_auth_crap"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_pam_chng_pswd_auth_crap"] <<< duration
+}
+
+#
+# winbind domain child function winbindd_dual_pam_chauthtok
+#
+
+probe process("winbindd").function("winbindd_dual_pam_chauthtok") {
+	dc_running[tid(), "winbindd_dual_pam_chauthtok"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dual_pam_chauthtok").return {
+	if (!([tid(), "winbindd_dual_pam_chauthtok"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "winbindd_dual_pam_chauthtok"]
+	delete dc_running[tid(), "winbindd_dual_pam_chauthtok"]
+
+	duration = end - begin
+	dc_svctime["winbindd_dual_pam_chauthtok"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupSid
+#
+
+probe process("winbindd").function("_wbint_LookupSid") {
+	dc_running[tid(), "_wbint_LookupSid"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupSid").return {
+	if (!([tid(), "_wbint_LookupSid"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupSid"]
+	delete dc_running[tid(), "_wbint_LookupSid"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupSid"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupSids
+#
+
+probe process("winbindd").function("_wbint_LookupSids") {
+	dc_running[tid(), "_wbint_LookupSids"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupSids").return {
+	if (!([tid(), "_wbint_LookupSids"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupSids"]
+	delete dc_running[tid(), "_wbint_LookupSids"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupSids"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupName
+#
+
+probe process("winbindd").function("_wbint_LookupName") {
+	dc_running[tid(), "_wbint_LookupName"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupName").return {
+	if (!([tid(), "_wbint_LookupName"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupName"]
+	delete dc_running[tid(), "_wbint_LookupName"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupName"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_Sids2UnixIDs
+#
+
+probe process("winbindd").function("_wbint_Sids2UnixIDs") {
+	dc_running[tid(), "_wbint_Sids2UnixIDs"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_Sids2UnixIDs").return {
+	if (!([tid(), "_wbint_Sids2UnixIDs"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_Sids2UnixIDs"]
+	delete dc_running[tid(), "_wbint_Sids2UnixIDs"]
+
+	duration = end - begin
+	dc_svctime["_wbint_Sids2UnixIDs"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_UnixIDs2Sids
+#
+
+probe process("winbindd").function("_wbint_UnixIDs2Sids") {
+	dc_running[tid(), "_wbint_UnixIDs2Sids"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_UnixIDs2Sids").return {
+	if (!([tid(), "_wbint_UnixIDs2Sids"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_UnixIDs2Sids"]
+	delete dc_running[tid(), "_wbint_UnixIDs2Sids"]
+
+	duration = end - begin
+	dc_svctime["_wbint_UnixIDs2Sids"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_AllocateUid
+#
+
+probe process("winbindd").function("_wbint_AllocateUid") {
+	dc_running[tid(), "_wbint_AllocateUid"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_AllocateUid").return {
+	if (!([tid(), "_wbint_AllocateUid"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_AllocateUid"]
+	delete dc_running[tid(), "_wbint_AllocateUid"]
+
+	duration = end - begin
+	dc_svctime["_wbint_AllocateUid"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_AllocateGid
+#
+
+probe process("winbindd").function("_wbint_AllocateGid") {
+	dc_running[tid(), "_wbint_AllocateGid"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_AllocateGid").return {
+	if (!([tid(), "_wbint_AllocateGid"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_AllocateGid"]
+	delete dc_running[tid(), "_wbint_AllocateGid"]
+
+	duration = end - begin
+	dc_svctime["_wbint_AllocateGid"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_GetNssInfo
+#
+
+probe process("winbindd").function("_wbint_GetNssInfo") {
+	dc_running[tid(), "_wbint_GetNssInfo"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_GetNssInfo").return {
+	if (!([tid(), "_wbint_GetNssInfo"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_GetNssInfo"]
+	delete dc_running[tid(), "_wbint_GetNssInfo"]
+
+	duration = end - begin
+	dc_svctime["_wbint_GetNssInfo"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupUserAliases
+#
+
+probe process("winbindd").function("_wbint_LookupUserAliases") {
+	dc_running[tid(), "_wbint_LookupUserAliases"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupUserAliases").return {
+	if (!([tid(), "_wbint_LookupUserAliases"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupUserAliases"]
+	delete dc_running[tid(), "_wbint_LookupUserAliases"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupUserAliases"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupUserGroups
+#
+
+probe process("winbindd").function("_wbint_LookupUserGroups") {
+	dc_running[tid(), "_wbint_LookupUserGroups"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupUserGroups").return {
+	if (!([tid(), "_wbint_LookupUserGroups"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupUserGroups"]
+	delete dc_running[tid(), "_wbint_LookupUserGroups"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupUserGroups"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_QuerySequenceNumber
+#
+
+probe process("winbindd").function("_wbint_QuerySequenceNumber") {
+	dc_running[tid(), "_wbint_QuerySequenceNumber"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_QuerySequenceNumber").return {
+	if (!([tid(), "_wbint_QuerySequenceNumber"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_QuerySequenceNumber"]
+	delete dc_running[tid(), "_wbint_QuerySequenceNumber"]
+
+	duration = end - begin
+	dc_svctime["_wbint_QuerySequenceNumber"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupGroupMembers
+#
+
+probe process("winbindd").function("_wbint_LookupGroupMembers") {
+	dc_running[tid(), "_wbint_LookupGroupMembers"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupGroupMembers").return {
+	if (!([tid(), "_wbint_LookupGroupMembers"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupGroupMembers"]
+	delete dc_running[tid(), "_wbint_LookupGroupMembers"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupGroupMembers"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_QueryGroupList
+#
+
+probe process("winbindd").function("_wbint_QueryGroupList") {
+	dc_running[tid(), "_wbint_QueryGroupList"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_QueryGroupList").return {
+	if (!([tid(), "_wbint_QueryGroupList"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_QueryGroupList"]
+	delete dc_running[tid(), "_wbint_QueryGroupList"]
+
+	duration = end - begin
+	dc_svctime["_wbint_QueryGroupList"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_QueryUserRidList
+#
+
+probe process("winbindd").function("_wbint_QueryUserRidList") {
+	dc_running[tid(), "_wbint_QueryUserRidList"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_QueryUserRidList").return {
+	if (!([tid(), "_wbint_QueryUserRidList"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_QueryUserRidList"]
+	delete dc_running[tid(), "_wbint_QueryUserRidList"]
+
+	duration = end - begin
+	dc_svctime["_wbint_QueryUserRidList"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_DsGetDcName
+#
+
+probe process("winbindd").function("_wbint_DsGetDcName") {
+	dc_running[tid(), "_wbint_DsGetDcName"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_DsGetDcName").return {
+	if (!([tid(), "_wbint_DsGetDcName"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_DsGetDcName"]
+	delete dc_running[tid(), "_wbint_DsGetDcName"]
+
+	duration = end - begin
+	dc_svctime["_wbint_DsGetDcName"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_LookupRids
+#
+
+probe process("winbindd").function("_wbint_LookupRids") {
+	dc_running[tid(), "_wbint_LookupRids"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_LookupRids").return {
+	if (!([tid(), "_wbint_LookupRids"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_LookupRids"]
+	delete dc_running[tid(), "_wbint_LookupRids"]
+
+	duration = end - begin
+	dc_svctime["_wbint_LookupRids"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_CheckMachineAccount
+#
+
+probe process("winbindd").function("_wbint_CheckMachineAccount") {
+	dc_running[tid(), "_wbint_CheckMachineAccount"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_CheckMachineAccount").return {
+	if (!([tid(), "_wbint_CheckMachineAccount"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_CheckMachineAccount"]
+	delete dc_running[tid(), "_wbint_CheckMachineAccount"]
+
+	duration = end - begin
+	dc_svctime["_wbint_CheckMachineAccount"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_ChangeMachineAccount
+#
+
+probe process("winbindd").function("_wbint_ChangeMachineAccount") {
+	dc_running[tid(), "_wbint_ChangeMachineAccount"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_ChangeMachineAccount").return {
+	if (!([tid(), "_wbint_ChangeMachineAccount"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_ChangeMachineAccount"]
+	delete dc_running[tid(), "_wbint_ChangeMachineAccount"]
+
+	duration = end - begin
+	dc_svctime["_wbint_ChangeMachineAccount"] <<< duration
+}
+
+#
+# winbind domain child function _wbint_PingDc
+#
+
+probe process("winbindd").function("_wbint_PingDc") {
+	dc_running[tid(), "_wbint_PingDc"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("_wbint_PingDc").return {
+	if (!([tid(), "_wbint_PingDc"] in dc_running))
+		next
+
+	end = gettimeofday_us()
+	begin = dc_running[tid(), "_wbint_PingDc"]
+	delete dc_running[tid(), "_wbint_PingDc"]
+
+	duration = end - begin
+	dc_svctime["_wbint_PingDc"] <<< duration
+}
+
+#
+# winbind domain child backend function query_user_list
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("query_user_list at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "query_user_list"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("query_user_list at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "query_user_list"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "query_user_list"]
+	delete backend_running[tid(), "query_user_list"]
+
+	duration = end - begin
+	backend_svctime["query_user_list"] <<< duration
+}
+
+#
+# winbind domain child backend function enum_dom_groups
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("enum_dom_groups at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "enum_dom_groups"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("enum_dom_groups at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "enum_dom_groups"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "enum_dom_groups"]
+	delete backend_running[tid(), "enum_dom_groups"]
+
+	duration = end - begin
+	backend_svctime["enum_dom_groups"] <<< duration
+}
+
+#
+# winbind domain child backend function enum_local_groups
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("enum_local_groups at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "enum_local_groups"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("enum_local_groups at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "enum_local_groups"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "enum_local_groups"]
+	delete backend_running[tid(), "enum_local_groups"]
+
+	duration = end - begin
+	backend_svctime["enum_local_groups"] <<< duration
+}
+
+#
+# winbind domain child backend function name_to_sid
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("name_to_sid at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "name_to_sid"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("name_to_sid at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "name_to_sid"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "name_to_sid"]
+	delete backend_running[tid(), "name_to_sid"]
+
+	duration = end - begin
+	backend_svctime["name_to_sid"] <<< duration
+}
+
+#
+# winbind domain child backend function sid_to_name
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("sid_to_name at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "sid_to_name"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("sid_to_name at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "sid_to_name"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "sid_to_name"]
+	delete backend_running[tid(), "sid_to_name"]
+
+	duration = end - begin
+	backend_svctime["sid_to_name"] <<< duration
+}
+
+#
+# winbind domain child backend function rids_to_names
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("rids_to_names at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "rids_to_names"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("rids_to_names at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "rids_to_names"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "rids_to_names"]
+	delete backend_running[tid(), "rids_to_names"]
+
+	duration = end - begin
+	backend_svctime["rids_to_names"] <<< duration
+}
+
+#
+# winbind domain child backend function lookup_usergroups
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("lookup_usergroups at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "lookup_usergroups"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("lookup_usergroups at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "lookup_usergroups"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "lookup_usergroups"]
+	delete backend_running[tid(), "lookup_usergroups"]
+
+	duration = end - begin
+	backend_svctime["lookup_usergroups"] <<< duration
+}
+
+#
+# winbind domain child backend function lookup_useraliases
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("lookup_useraliases at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "lookup_useraliases"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("lookup_useraliases at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "lookup_useraliases"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "lookup_useraliases"]
+	delete backend_running[tid(), "lookup_useraliases"]
+
+	duration = end - begin
+	backend_svctime["lookup_useraliases"] <<< duration
+}
+
+#
+# winbind domain child backend function lookup_groupmem
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("lookup_groupmem at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "lookup_groupmem"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("lookup_groupmem at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "lookup_groupmem"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "lookup_groupmem"]
+	delete backend_running[tid(), "lookup_groupmem"]
+
+	duration = end - begin
+	backend_svctime["lookup_groupmem"] <<< duration
+}
+
+#
+# winbind domain child backend function sequence_number
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("sequence_number at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "sequence_number"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("sequence_number at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "sequence_number"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "sequence_number"]
+	delete backend_running[tid(), "sequence_number"]
+
+	duration = end - begin
+	backend_svctime["sequence_number"] <<< duration
+}
+
+#
+# winbind domain child backend function lockout_policy
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("lockout_policy at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "lockout_policy"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("lockout_policy at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "lockout_policy"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "lockout_policy"]
+	delete backend_running[tid(), "lockout_policy"]
+
+	duration = end - begin
+	backend_svctime["lockout_policy"] <<< duration
+}
+
+#
+# winbind domain child backend function password_policy
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("password_policy at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "password_policy"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("password_policy at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "password_policy"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "password_policy"]
+	delete backend_running[tid(), "password_policy"]
+
+	duration = end - begin
+	backend_svctime["password_policy"] <<< duration
+}
+
+#
+# winbind domain child backend function trusted_domains
+#
+# @../source3/winbindd/winbindd_ads.c:*
+
+probe process("winbindd").function("trusted_domains at ../source3/winbindd/winbindd_ads.c") {
+	backend_running[tid(), "trusted_domains"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("trusted_domains at ../source3/winbindd/winbindd_ads.c").return {
+	if (!([tid(), "trusted_domains"] in backend_running))
+		next
+
+	end = gettimeofday_us()
+	begin = backend_running[tid(), "trusted_domains"]
+	delete backend_running[tid(), "trusted_domains"]
+
+	duration = end - begin
+	backend_svctime["trusted_domains"] <<< duration
+}
+
+#
+# winbind async function wb_ping
+#
+
+probe process("winbindd").function("wb_ping_send") {
+	send_running["wb_ping_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("wb_ping_send").return {
+	if (!(["wb_ping_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["wb_ping_send"]
+	delete send_running["wb_ping_send"]
+
+	start_time["wb_ping_send", $return] = start
+	idle_time["wb_ping_send", $return] = end
+}
+
+probe process("winbindd").function("wb_ping_recv") {
+	if (!(["wb_ping_send", $req] in start_time))
+		next
+
+	recv_running["wb_ping_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("wb_ping_recv").return {
+	if (!(["wb_ping_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["wb_ping_recv"]
+	delete recv_running["wb_ping_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["wb_ping_send", req]
+	delete start_time["wb_ping_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["wb_ping_send", req]
+	delete idle_time["wb_ping_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["wb_ping_send"] <<< svctime
+	async_runtime["wb_ping_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_lookupsid
+#
+
+probe process("winbindd").function("winbindd_lookupsid_send") {
+	send_running["winbindd_lookupsid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupsid_send").return {
+	if (!(["winbindd_lookupsid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_lookupsid_send"]
+	delete send_running["winbindd_lookupsid_send"]
+
+	start_time["winbindd_lookupsid_send", $return] = start
+	idle_time["winbindd_lookupsid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_lookupsid_recv") {
+	if (!(["winbindd_lookupsid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_lookupsid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupsid_recv").return {
+	if (!(["winbindd_lookupsid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_lookupsid_recv"]
+	delete recv_running["winbindd_lookupsid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_lookupsid_send", req]
+	delete start_time["winbindd_lookupsid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_lookupsid_send", req]
+	delete idle_time["winbindd_lookupsid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_lookupsid_send"] <<< svctime
+	async_runtime["winbindd_lookupsid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_lookupsids
+#
+
+probe process("winbindd").function("winbindd_lookupsids_send") {
+	send_running["winbindd_lookupsids_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupsids_send").return {
+	if (!(["winbindd_lookupsids_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_lookupsids_send"]
+	delete send_running["winbindd_lookupsids_send"]
+
+	start_time["winbindd_lookupsids_send", $return] = start
+	idle_time["winbindd_lookupsids_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_lookupsids_recv") {
+	if (!(["winbindd_lookupsids_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_lookupsids_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupsids_recv").return {
+	if (!(["winbindd_lookupsids_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_lookupsids_recv"]
+	delete recv_running["winbindd_lookupsids_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_lookupsids_send", req]
+	delete start_time["winbindd_lookupsids_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_lookupsids_send", req]
+	delete idle_time["winbindd_lookupsids_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_lookupsids_send"] <<< svctime
+	async_runtime["winbindd_lookupsids_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_lookupname
+#
+
+probe process("winbindd").function("winbindd_lookupname_send") {
+	send_running["winbindd_lookupname_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupname_send").return {
+	if (!(["winbindd_lookupname_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_lookupname_send"]
+	delete send_running["winbindd_lookupname_send"]
+
+	start_time["winbindd_lookupname_send", $return] = start
+	idle_time["winbindd_lookupname_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_lookupname_recv") {
+	if (!(["winbindd_lookupname_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_lookupname_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookupname_recv").return {
+	if (!(["winbindd_lookupname_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_lookupname_recv"]
+	delete recv_running["winbindd_lookupname_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_lookupname_send", req]
+	delete start_time["winbindd_lookupname_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_lookupname_send", req]
+	delete idle_time["winbindd_lookupname_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_lookupname_send"] <<< svctime
+	async_runtime["winbindd_lookupname_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_sids_to_xids
+#
+
+probe process("winbindd").function("winbindd_sids_to_xids_send") {
+	send_running["winbindd_sids_to_xids_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_sids_to_xids_send").return {
+	if (!(["winbindd_sids_to_xids_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_sids_to_xids_send"]
+	delete send_running["winbindd_sids_to_xids_send"]
+
+	start_time["winbindd_sids_to_xids_send", $return] = start
+	idle_time["winbindd_sids_to_xids_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_sids_to_xids_recv") {
+	if (!(["winbindd_sids_to_xids_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_sids_to_xids_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_sids_to_xids_recv").return {
+	if (!(["winbindd_sids_to_xids_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_sids_to_xids_recv"]
+	delete recv_running["winbindd_sids_to_xids_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_sids_to_xids_send", req]
+	delete start_time["winbindd_sids_to_xids_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_sids_to_xids_send", req]
+	delete idle_time["winbindd_sids_to_xids_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_sids_to_xids_send"] <<< svctime
+	async_runtime["winbindd_sids_to_xids_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_xids_to_sids
+#
+
+probe process("winbindd").function("winbindd_xids_to_sids_send") {
+	send_running["winbindd_xids_to_sids_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_xids_to_sids_send").return {
+	if (!(["winbindd_xids_to_sids_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_xids_to_sids_send"]
+	delete send_running["winbindd_xids_to_sids_send"]
+
+	start_time["winbindd_xids_to_sids_send", $return] = start
+	idle_time["winbindd_xids_to_sids_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_xids_to_sids_recv") {
+	if (!(["winbindd_xids_to_sids_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_xids_to_sids_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_xids_to_sids_recv").return {
+	if (!(["winbindd_xids_to_sids_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_xids_to_sids_recv"]
+	delete recv_running["winbindd_xids_to_sids_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_xids_to_sids_send", req]
+	delete start_time["winbindd_xids_to_sids_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_xids_to_sids_send", req]
+	delete idle_time["winbindd_xids_to_sids_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_xids_to_sids_send"] <<< svctime
+	async_runtime["winbindd_xids_to_sids_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getpwsid
+#
+
+probe process("winbindd").function("winbindd_getpwsid_send") {
+	send_running["winbindd_getpwsid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwsid_send").return {
+	if (!(["winbindd_getpwsid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getpwsid_send"]
+	delete send_running["winbindd_getpwsid_send"]
+
+	start_time["winbindd_getpwsid_send", $return] = start
+	idle_time["winbindd_getpwsid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getpwsid_recv") {
+	if (!(["winbindd_getpwsid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getpwsid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwsid_recv").return {
+	if (!(["winbindd_getpwsid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getpwsid_recv"]
+	delete recv_running["winbindd_getpwsid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getpwsid_send", req]
+	delete start_time["winbindd_getpwsid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getpwsid_send", req]
+	delete idle_time["winbindd_getpwsid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getpwsid_send"] <<< svctime
+	async_runtime["winbindd_getpwsid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getpwnam
+#
+
+probe process("winbindd").function("winbindd_getpwnam_send") {
+	send_running["winbindd_getpwnam_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwnam_send").return {
+	if (!(["winbindd_getpwnam_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getpwnam_send"]
+	delete send_running["winbindd_getpwnam_send"]
+
+	start_time["winbindd_getpwnam_send", $return] = start
+	idle_time["winbindd_getpwnam_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getpwnam_recv") {
+	if (!(["winbindd_getpwnam_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getpwnam_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwnam_recv").return {
+	if (!(["winbindd_getpwnam_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getpwnam_recv"]
+	delete recv_running["winbindd_getpwnam_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getpwnam_send", req]
+	delete start_time["winbindd_getpwnam_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getpwnam_send", req]
+	delete idle_time["winbindd_getpwnam_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getpwnam_send"] <<< svctime
+	async_runtime["winbindd_getpwnam_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getpwuid
+#
+
+probe process("winbindd").function("winbindd_getpwuid_send") {
+	send_running["winbindd_getpwuid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwuid_send").return {
+	if (!(["winbindd_getpwuid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getpwuid_send"]
+	delete send_running["winbindd_getpwuid_send"]
+
+	start_time["winbindd_getpwuid_send", $return] = start
+	idle_time["winbindd_getpwuid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getpwuid_recv") {
+	if (!(["winbindd_getpwuid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getpwuid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwuid_recv").return {
+	if (!(["winbindd_getpwuid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getpwuid_recv"]
+	delete recv_running["winbindd_getpwuid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getpwuid_send", req]
+	delete start_time["winbindd_getpwuid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getpwuid_send", req]
+	delete idle_time["winbindd_getpwuid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getpwuid_send"] <<< svctime
+	async_runtime["winbindd_getpwuid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getsidaliases
+#
+
+probe process("winbindd").function("winbindd_getsidaliases_send") {
+	send_running["winbindd_getsidaliases_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getsidaliases_send").return {
+	if (!(["winbindd_getsidaliases_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getsidaliases_send"]
+	delete send_running["winbindd_getsidaliases_send"]
+
+	start_time["winbindd_getsidaliases_send", $return] = start
+	idle_time["winbindd_getsidaliases_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getsidaliases_recv") {
+	if (!(["winbindd_getsidaliases_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getsidaliases_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getsidaliases_recv").return {
+	if (!(["winbindd_getsidaliases_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getsidaliases_recv"]
+	delete recv_running["winbindd_getsidaliases_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getsidaliases_send", req]
+	delete start_time["winbindd_getsidaliases_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getsidaliases_send", req]
+	delete idle_time["winbindd_getsidaliases_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getsidaliases_send"] <<< svctime
+	async_runtime["winbindd_getsidaliases_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getuserdomgroups
+#
+
+probe process("winbindd").function("winbindd_getuserdomgroups_send") {
+	send_running["winbindd_getuserdomgroups_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getuserdomgroups_send").return {
+	if (!(["winbindd_getuserdomgroups_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getuserdomgroups_send"]
+	delete send_running["winbindd_getuserdomgroups_send"]
+
+	start_time["winbindd_getuserdomgroups_send", $return] = start
+	idle_time["winbindd_getuserdomgroups_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getuserdomgroups_recv") {
+	if (!(["winbindd_getuserdomgroups_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getuserdomgroups_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getuserdomgroups_recv").return {
+	if (!(["winbindd_getuserdomgroups_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getuserdomgroups_recv"]
+	delete recv_running["winbindd_getuserdomgroups_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getuserdomgroups_send", req]
+	delete start_time["winbindd_getuserdomgroups_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getuserdomgroups_send", req]
+	delete idle_time["winbindd_getuserdomgroups_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getuserdomgroups_send"] <<< svctime
+	async_runtime["winbindd_getuserdomgroups_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getgroups
+#
+
+probe process("winbindd").function("winbindd_getgroups_send") {
+	send_running["winbindd_getgroups_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgroups_send").return {
+	if (!(["winbindd_getgroups_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getgroups_send"]
+	delete send_running["winbindd_getgroups_send"]
+
+	start_time["winbindd_getgroups_send", $return] = start
+	idle_time["winbindd_getgroups_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getgroups_recv") {
+	if (!(["winbindd_getgroups_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getgroups_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgroups_recv").return {
+	if (!(["winbindd_getgroups_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getgroups_recv"]
+	delete recv_running["winbindd_getgroups_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getgroups_send", req]
+	delete start_time["winbindd_getgroups_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getgroups_send", req]
+	delete idle_time["winbindd_getgroups_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getgroups_send"] <<< svctime
+	async_runtime["winbindd_getgroups_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_show_sequence
+#
+
+probe process("winbindd").function("winbindd_show_sequence_send") {
+	send_running["winbindd_show_sequence_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_show_sequence_send").return {
+	if (!(["winbindd_show_sequence_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_show_sequence_send"]
+	delete send_running["winbindd_show_sequence_send"]
+
+	start_time["winbindd_show_sequence_send", $return] = start
+	idle_time["winbindd_show_sequence_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_show_sequence_recv") {
+	if (!(["winbindd_show_sequence_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_show_sequence_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_show_sequence_recv").return {
+	if (!(["winbindd_show_sequence_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_show_sequence_recv"]
+	delete recv_running["winbindd_show_sequence_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_show_sequence_send", req]
+	delete start_time["winbindd_show_sequence_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_show_sequence_send", req]
+	delete idle_time["winbindd_show_sequence_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_show_sequence_send"] <<< svctime
+	async_runtime["winbindd_show_sequence_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getgrgid
+#
+
+probe process("winbindd").function("winbindd_getgrgid_send") {
+	send_running["winbindd_getgrgid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrgid_send").return {
+	if (!(["winbindd_getgrgid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getgrgid_send"]
+	delete send_running["winbindd_getgrgid_send"]
+
+	start_time["winbindd_getgrgid_send", $return] = start
+	idle_time["winbindd_getgrgid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getgrgid_recv") {
+	if (!(["winbindd_getgrgid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getgrgid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrgid_recv").return {
+	if (!(["winbindd_getgrgid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getgrgid_recv"]
+	delete recv_running["winbindd_getgrgid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getgrgid_send", req]
+	delete start_time["winbindd_getgrgid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getgrgid_send", req]
+	delete idle_time["winbindd_getgrgid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getgrgid_send"] <<< svctime
+	async_runtime["winbindd_getgrgid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getgrnam
+#
+
+probe process("winbindd").function("winbindd_getgrnam_send") {
+	send_running["winbindd_getgrnam_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrnam_send").return {
+	if (!(["winbindd_getgrnam_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getgrnam_send"]
+	delete send_running["winbindd_getgrnam_send"]
+
+	start_time["winbindd_getgrnam_send", $return] = start
+	idle_time["winbindd_getgrnam_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getgrnam_recv") {
+	if (!(["winbindd_getgrnam_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getgrnam_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrnam_recv").return {
+	if (!(["winbindd_getgrnam_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getgrnam_recv"]
+	delete recv_running["winbindd_getgrnam_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getgrnam_send", req]
+	delete start_time["winbindd_getgrnam_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getgrnam_send", req]
+	delete idle_time["winbindd_getgrnam_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getgrnam_send"] <<< svctime
+	async_runtime["winbindd_getgrnam_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getusersids
+#
+
+probe process("winbindd").function("winbindd_getusersids_send") {
+	send_running["winbindd_getusersids_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getusersids_send").return {
+	if (!(["winbindd_getusersids_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getusersids_send"]
+	delete send_running["winbindd_getusersids_send"]
+
+	start_time["winbindd_getusersids_send", $return] = start
+	idle_time["winbindd_getusersids_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getusersids_recv") {
+	if (!(["winbindd_getusersids_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getusersids_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getusersids_recv").return {
+	if (!(["winbindd_getusersids_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getusersids_recv"]
+	delete recv_running["winbindd_getusersids_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getusersids_send", req]
+	delete start_time["winbindd_getusersids_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getusersids_send", req]
+	delete idle_time["winbindd_getusersids_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getusersids_send"] <<< svctime
+	async_runtime["winbindd_getusersids_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_lookuprids
+#
+
+probe process("winbindd").function("winbindd_lookuprids_send") {
+	send_running["winbindd_lookuprids_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookuprids_send").return {
+	if (!(["winbindd_lookuprids_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_lookuprids_send"]
+	delete send_running["winbindd_lookuprids_send"]
+
+	start_time["winbindd_lookuprids_send", $return] = start
+	idle_time["winbindd_lookuprids_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_lookuprids_recv") {
+	if (!(["winbindd_lookuprids_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_lookuprids_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_lookuprids_recv").return {
+	if (!(["winbindd_lookuprids_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_lookuprids_recv"]
+	delete recv_running["winbindd_lookuprids_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_lookuprids_send", req]
+	delete start_time["winbindd_lookuprids_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_lookuprids_send", req]
+	delete idle_time["winbindd_lookuprids_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_lookuprids_send"] <<< svctime
+	async_runtime["winbindd_lookuprids_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_setpwent
+#
+
+probe process("winbindd").function("winbindd_setpwent_send") {
+	send_running["winbindd_setpwent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_setpwent_send").return {
+	if (!(["winbindd_setpwent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_setpwent_send"]
+	delete send_running["winbindd_setpwent_send"]
+
+	start_time["winbindd_setpwent_send", $return] = start
+	idle_time["winbindd_setpwent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_setpwent_recv") {
+	if (!(["winbindd_setpwent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_setpwent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_setpwent_recv").return {
+	if (!(["winbindd_setpwent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_setpwent_recv"]
+	delete recv_running["winbindd_setpwent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_setpwent_send", req]
+	delete start_time["winbindd_setpwent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_setpwent_send", req]
+	delete idle_time["winbindd_setpwent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_setpwent_send"] <<< svctime
+	async_runtime["winbindd_setpwent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getpwent
+#
+
+probe process("winbindd").function("winbindd_getpwent_send") {
+	send_running["winbindd_getpwent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwent_send").return {
+	if (!(["winbindd_getpwent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getpwent_send"]
+	delete send_running["winbindd_getpwent_send"]
+
+	start_time["winbindd_getpwent_send", $return] = start
+	idle_time["winbindd_getpwent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getpwent_recv") {
+	if (!(["winbindd_getpwent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getpwent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getpwent_recv").return {
+	if (!(["winbindd_getpwent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getpwent_recv"]
+	delete recv_running["winbindd_getpwent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getpwent_send", req]
+	delete start_time["winbindd_getpwent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getpwent_send", req]
+	delete idle_time["winbindd_getpwent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getpwent_send"] <<< svctime
+	async_runtime["winbindd_getpwent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_endpwent
+#
+
+probe process("winbindd").function("winbindd_endpwent_send") {
+	send_running["winbindd_endpwent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_endpwent_send").return {
+	if (!(["winbindd_endpwent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_endpwent_send"]
+	delete send_running["winbindd_endpwent_send"]
+
+	start_time["winbindd_endpwent_send", $return] = start
+	idle_time["winbindd_endpwent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_endpwent_recv") {
+	if (!(["winbindd_endpwent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_endpwent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_endpwent_recv").return {
+	if (!(["winbindd_endpwent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_endpwent_recv"]
+	delete recv_running["winbindd_endpwent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_endpwent_send", req]
+	delete start_time["winbindd_endpwent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_endpwent_send", req]
+	delete idle_time["winbindd_endpwent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_endpwent_send"] <<< svctime
+	async_runtime["winbindd_endpwent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_dsgetdcname
+#
+
+probe process("winbindd").function("winbindd_dsgetdcname_send") {
+	send_running["winbindd_dsgetdcname_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dsgetdcname_send").return {
+	if (!(["winbindd_dsgetdcname_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_dsgetdcname_send"]
+	delete send_running["winbindd_dsgetdcname_send"]
+
+	start_time["winbindd_dsgetdcname_send", $return] = start
+	idle_time["winbindd_dsgetdcname_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_dsgetdcname_recv") {
+	if (!(["winbindd_dsgetdcname_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_dsgetdcname_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_dsgetdcname_recv").return {
+	if (!(["winbindd_dsgetdcname_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_dsgetdcname_recv"]
+	delete recv_running["winbindd_dsgetdcname_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_dsgetdcname_send", req]
+	delete start_time["winbindd_dsgetdcname_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_dsgetdcname_send", req]
+	delete idle_time["winbindd_dsgetdcname_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_dsgetdcname_send"] <<< svctime
+	async_runtime["winbindd_dsgetdcname_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getdcname
+#
+
+probe process("winbindd").function("winbindd_getdcname_send") {
+	send_running["winbindd_getdcname_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getdcname_send").return {
+	if (!(["winbindd_getdcname_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getdcname_send"]
+	delete send_running["winbindd_getdcname_send"]
+
+	start_time["winbindd_getdcname_send", $return] = start
+	idle_time["winbindd_getdcname_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getdcname_recv") {
+	if (!(["winbindd_getdcname_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getdcname_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getdcname_recv").return {
+	if (!(["winbindd_getdcname_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getdcname_recv"]
+	delete recv_running["winbindd_getdcname_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getdcname_send", req]
+	delete start_time["winbindd_getdcname_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getdcname_send", req]
+	delete idle_time["winbindd_getdcname_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getdcname_send"] <<< svctime
+	async_runtime["winbindd_getdcname_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_setgrent
+#
+
+probe process("winbindd").function("winbindd_setgrent_send") {
+	send_running["winbindd_setgrent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_setgrent_send").return {
+	if (!(["winbindd_setgrent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_setgrent_send"]
+	delete send_running["winbindd_setgrent_send"]
+
+	start_time["winbindd_setgrent_send", $return] = start
+	idle_time["winbindd_setgrent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_setgrent_recv") {
+	if (!(["winbindd_setgrent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_setgrent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_setgrent_recv").return {
+	if (!(["winbindd_setgrent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_setgrent_recv"]
+	delete recv_running["winbindd_setgrent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_setgrent_send", req]
+	delete start_time["winbindd_setgrent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_setgrent_send", req]
+	delete idle_time["winbindd_setgrent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_setgrent_send"] <<< svctime
+	async_runtime["winbindd_setgrent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_getgrent
+#
+
+probe process("winbindd").function("winbindd_getgrent_send") {
+	send_running["winbindd_getgrent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrent_send").return {
+	if (!(["winbindd_getgrent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_getgrent_send"]
+	delete send_running["winbindd_getgrent_send"]
+
+	start_time["winbindd_getgrent_send", $return] = start
+	idle_time["winbindd_getgrent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_getgrent_recv") {
+	if (!(["winbindd_getgrent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_getgrent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_getgrent_recv").return {
+	if (!(["winbindd_getgrent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_getgrent_recv"]
+	delete recv_running["winbindd_getgrent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_getgrent_send", req]
+	delete start_time["winbindd_getgrent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_getgrent_send", req]
+	delete idle_time["winbindd_getgrent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_getgrent_send"] <<< svctime
+	async_runtime["winbindd_getgrent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_endgrent
+#
+
+probe process("winbindd").function("winbindd_endgrent_send") {
+	send_running["winbindd_endgrent_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_endgrent_send").return {
+	if (!(["winbindd_endgrent_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_endgrent_send"]
+	delete send_running["winbindd_endgrent_send"]
+
+	start_time["winbindd_endgrent_send", $return] = start
+	idle_time["winbindd_endgrent_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_endgrent_recv") {
+	if (!(["winbindd_endgrent_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_endgrent_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_endgrent_recv").return {
+	if (!(["winbindd_endgrent_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_endgrent_recv"]
+	delete recv_running["winbindd_endgrent_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_endgrent_send", req]
+	delete start_time["winbindd_endgrent_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_endgrent_send", req]
+	delete idle_time["winbindd_endgrent_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_endgrent_send"] <<< svctime
+	async_runtime["winbindd_endgrent_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_list_users
+#
+
+probe process("winbindd").function("winbindd_list_users_send") {
+	send_running["winbindd_list_users_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_list_users_send").return {
+	if (!(["winbindd_list_users_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_list_users_send"]
+	delete send_running["winbindd_list_users_send"]
+
+	start_time["winbindd_list_users_send", $return] = start
+	idle_time["winbindd_list_users_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_list_users_recv") {
+	if (!(["winbindd_list_users_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_list_users_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_list_users_recv").return {
+	if (!(["winbindd_list_users_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_list_users_recv"]
+	delete recv_running["winbindd_list_users_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_list_users_send", req]
+	delete start_time["winbindd_list_users_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_list_users_send", req]
+	delete idle_time["winbindd_list_users_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_list_users_send"] <<< svctime
+	async_runtime["winbindd_list_users_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_list_groups
+#
+
+probe process("winbindd").function("winbindd_list_groups_send") {
+	send_running["winbindd_list_groups_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_list_groups_send").return {
+	if (!(["winbindd_list_groups_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_list_groups_send"]
+	delete send_running["winbindd_list_groups_send"]
+
+	start_time["winbindd_list_groups_send", $return] = start
+	idle_time["winbindd_list_groups_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_list_groups_recv") {
+	if (!(["winbindd_list_groups_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_list_groups_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_list_groups_recv").return {
+	if (!(["winbindd_list_groups_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_list_groups_recv"]
+	delete recv_running["winbindd_list_groups_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_list_groups_send", req]
+	delete start_time["winbindd_list_groups_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_list_groups_send", req]
+	delete idle_time["winbindd_list_groups_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_list_groups_send"] <<< svctime
+	async_runtime["winbindd_list_groups_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_check_machine_acct
+#
+
+probe process("winbindd").function("winbindd_check_machine_acct_send") {
+	send_running["winbindd_check_machine_acct_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_check_machine_acct_send").return {
+	if (!(["winbindd_check_machine_acct_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_check_machine_acct_send"]
+	delete send_running["winbindd_check_machine_acct_send"]
+
+	start_time["winbindd_check_machine_acct_send", $return] = start
+	idle_time["winbindd_check_machine_acct_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_check_machine_acct_recv") {
+	if (!(["winbindd_check_machine_acct_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_check_machine_acct_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_check_machine_acct_recv").return {
+	if (!(["winbindd_check_machine_acct_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_check_machine_acct_recv"]
+	delete recv_running["winbindd_check_machine_acct_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_check_machine_acct_send", req]
+	delete start_time["winbindd_check_machine_acct_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_check_machine_acct_send", req]
+	delete idle_time["winbindd_check_machine_acct_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_check_machine_acct_send"] <<< svctime
+	async_runtime["winbindd_check_machine_acct_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_ping_dc
+#
+
+probe process("winbindd").function("winbindd_ping_dc_send") {
+	send_running["winbindd_ping_dc_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_ping_dc_send").return {
+	if (!(["winbindd_ping_dc_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_ping_dc_send"]
+	delete send_running["winbindd_ping_dc_send"]
+
+	start_time["winbindd_ping_dc_send", $return] = start
+	idle_time["winbindd_ping_dc_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_ping_dc_recv") {
+	if (!(["winbindd_ping_dc_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_ping_dc_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_ping_dc_recv").return {
+	if (!(["winbindd_ping_dc_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_ping_dc_recv"]
+	delete recv_running["winbindd_ping_dc_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_ping_dc_send", req]
+	delete start_time["winbindd_ping_dc_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_ping_dc_send", req]
+	delete idle_time["winbindd_ping_dc_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_ping_dc_send"] <<< svctime
+	async_runtime["winbindd_ping_dc_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_pam_auth
+#
+
+probe process("winbindd").function("winbindd_pam_auth_send") {
+	send_running["winbindd_pam_auth_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_auth_send").return {
+	if (!(["winbindd_pam_auth_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_pam_auth_send"]
+	delete send_running["winbindd_pam_auth_send"]
+
+	start_time["winbindd_pam_auth_send", $return] = start
+	idle_time["winbindd_pam_auth_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_pam_auth_recv") {
+	if (!(["winbindd_pam_auth_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_pam_auth_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_auth_recv").return {
+	if (!(["winbindd_pam_auth_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_pam_auth_recv"]
+	delete recv_running["winbindd_pam_auth_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_pam_auth_send", req]
+	delete start_time["winbindd_pam_auth_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_pam_auth_send", req]
+	delete idle_time["winbindd_pam_auth_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_pam_auth_send"] <<< svctime
+	async_runtime["winbindd_pam_auth_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_pam_logoff
+#
+
+probe process("winbindd").function("winbindd_pam_logoff_send") {
+	send_running["winbindd_pam_logoff_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_logoff_send").return {
+	if (!(["winbindd_pam_logoff_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_pam_logoff_send"]
+	delete send_running["winbindd_pam_logoff_send"]
+
+	start_time["winbindd_pam_logoff_send", $return] = start
+	idle_time["winbindd_pam_logoff_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_pam_logoff_recv") {
+	if (!(["winbindd_pam_logoff_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_pam_logoff_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_logoff_recv").return {
+	if (!(["winbindd_pam_logoff_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_pam_logoff_recv"]
+	delete recv_running["winbindd_pam_logoff_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_pam_logoff_send", req]
+	delete start_time["winbindd_pam_logoff_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_pam_logoff_send", req]
+	delete idle_time["winbindd_pam_logoff_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_pam_logoff_send"] <<< svctime
+	async_runtime["winbindd_pam_logoff_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_pam_chauthtok
+#
+
+probe process("winbindd").function("winbindd_pam_chauthtok_send") {
+	send_running["winbindd_pam_chauthtok_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_chauthtok_send").return {
+	if (!(["winbindd_pam_chauthtok_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_pam_chauthtok_send"]
+	delete send_running["winbindd_pam_chauthtok_send"]
+
+	start_time["winbindd_pam_chauthtok_send", $return] = start
+	idle_time["winbindd_pam_chauthtok_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_pam_chauthtok_recv") {
+	if (!(["winbindd_pam_chauthtok_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_pam_chauthtok_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_chauthtok_recv").return {
+	if (!(["winbindd_pam_chauthtok_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_pam_chauthtok_recv"]
+	delete recv_running["winbindd_pam_chauthtok_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_pam_chauthtok_send", req]
+	delete start_time["winbindd_pam_chauthtok_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_pam_chauthtok_send", req]
+	delete idle_time["winbindd_pam_chauthtok_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_pam_chauthtok_send"] <<< svctime
+	async_runtime["winbindd_pam_chauthtok_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_pam_chng_pswd_auth_crap
+#
+
+probe process("winbindd").function("winbindd_pam_chng_pswd_auth_crap_send") {
+	send_running["winbindd_pam_chng_pswd_auth_crap_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_chng_pswd_auth_crap_send").return {
+	if (!(["winbindd_pam_chng_pswd_auth_crap_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_pam_chng_pswd_auth_crap_send"]
+	delete send_running["winbindd_pam_chng_pswd_auth_crap_send"]
+
+	start_time["winbindd_pam_chng_pswd_auth_crap_send", $return] = start
+	idle_time["winbindd_pam_chng_pswd_auth_crap_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_pam_chng_pswd_auth_crap_recv") {
+	if (!(["winbindd_pam_chng_pswd_auth_crap_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_pam_chng_pswd_auth_crap_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_chng_pswd_auth_crap_recv").return {
+	if (!(["winbindd_pam_chng_pswd_auth_crap_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_pam_chng_pswd_auth_crap_recv"]
+	delete recv_running["winbindd_pam_chng_pswd_auth_crap_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_pam_chng_pswd_auth_crap_send", req]
+	delete start_time["winbindd_pam_chng_pswd_auth_crap_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_pam_chng_pswd_auth_crap_send", req]
+	delete idle_time["winbindd_pam_chng_pswd_auth_crap_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_pam_chng_pswd_auth_crap_send"] <<< svctime
+	async_runtime["winbindd_pam_chng_pswd_auth_crap_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_wins_byip
+#
+
+probe process("winbindd").function("winbindd_wins_byip_send") {
+	send_running["winbindd_wins_byip_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_wins_byip_send").return {
+	if (!(["winbindd_wins_byip_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_wins_byip_send"]
+	delete send_running["winbindd_wins_byip_send"]
+
+	start_time["winbindd_wins_byip_send", $return] = start
+	idle_time["winbindd_wins_byip_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_wins_byip_recv") {
+	if (!(["winbindd_wins_byip_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_wins_byip_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_wins_byip_recv").return {
+	if (!(["winbindd_wins_byip_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_wins_byip_recv"]
+	delete recv_running["winbindd_wins_byip_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_wins_byip_send", req]
+	delete start_time["winbindd_wins_byip_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_wins_byip_send", req]
+	delete idle_time["winbindd_wins_byip_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_wins_byip_send"] <<< svctime
+	async_runtime["winbindd_wins_byip_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_wins_byname
+#
+
+probe process("winbindd").function("winbindd_wins_byname_send") {
+	send_running["winbindd_wins_byname_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_wins_byname_send").return {
+	if (!(["winbindd_wins_byname_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_wins_byname_send"]
+	delete send_running["winbindd_wins_byname_send"]
+
+	start_time["winbindd_wins_byname_send", $return] = start
+	idle_time["winbindd_wins_byname_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_wins_byname_recv") {
+	if (!(["winbindd_wins_byname_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_wins_byname_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_wins_byname_recv").return {
+	if (!(["winbindd_wins_byname_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_wins_byname_recv"]
+	delete recv_running["winbindd_wins_byname_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_wins_byname_send", req]
+	delete start_time["winbindd_wins_byname_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_wins_byname_send", req]
+	delete idle_time["winbindd_wins_byname_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_wins_byname_send"] <<< svctime
+	async_runtime["winbindd_wins_byname_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_allocate_uid
+#
+
+probe process("winbindd").function("winbindd_allocate_uid_send") {
+	send_running["winbindd_allocate_uid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_allocate_uid_send").return {
+	if (!(["winbindd_allocate_uid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_allocate_uid_send"]
+	delete send_running["winbindd_allocate_uid_send"]
+
+	start_time["winbindd_allocate_uid_send", $return] = start
+	idle_time["winbindd_allocate_uid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_allocate_uid_recv") {
+	if (!(["winbindd_allocate_uid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_allocate_uid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_allocate_uid_recv").return {
+	if (!(["winbindd_allocate_uid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_allocate_uid_recv"]
+	delete recv_running["winbindd_allocate_uid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_allocate_uid_send", req]
+	delete start_time["winbindd_allocate_uid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_allocate_uid_send", req]
+	delete idle_time["winbindd_allocate_uid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_allocate_uid_send"] <<< svctime
+	async_runtime["winbindd_allocate_uid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_allocate_gid
+#
+
+probe process("winbindd").function("winbindd_allocate_gid_send") {
+	send_running["winbindd_allocate_gid_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_allocate_gid_send").return {
+	if (!(["winbindd_allocate_gid_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_allocate_gid_send"]
+	delete send_running["winbindd_allocate_gid_send"]
+
+	start_time["winbindd_allocate_gid_send", $return] = start
+	idle_time["winbindd_allocate_gid_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_allocate_gid_recv") {
+	if (!(["winbindd_allocate_gid_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_allocate_gid_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_allocate_gid_recv").return {
+	if (!(["winbindd_allocate_gid_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_allocate_gid_recv"]
+	delete recv_running["winbindd_allocate_gid_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_allocate_gid_send", req]
+	delete start_time["winbindd_allocate_gid_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_allocate_gid_send", req]
+	delete idle_time["winbindd_allocate_gid_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_allocate_gid_send"] <<< svctime
+	async_runtime["winbindd_allocate_gid_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_change_machine_acct
+#
+
+probe process("winbindd").function("winbindd_change_machine_acct_send") {
+	send_running["winbindd_change_machine_acct_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_change_machine_acct_send").return {
+	if (!(["winbindd_change_machine_acct_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_change_machine_acct_send"]
+	delete send_running["winbindd_change_machine_acct_send"]
+
+	start_time["winbindd_change_machine_acct_send", $return] = start
+	idle_time["winbindd_change_machine_acct_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_change_machine_acct_recv") {
+	if (!(["winbindd_change_machine_acct_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_change_machine_acct_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_change_machine_acct_recv").return {
+	if (!(["winbindd_change_machine_acct_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_change_machine_acct_recv"]
+	delete recv_running["winbindd_change_machine_acct_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_change_machine_acct_send", req]
+	delete start_time["winbindd_change_machine_acct_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_change_machine_acct_send", req]
+	delete idle_time["winbindd_change_machine_acct_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_change_machine_acct_send"] <<< svctime
+	async_runtime["winbindd_change_machine_acct_send"] <<< runtime
+}
+
+#
+# winbind async function winbindd_pam_auth_crap
+#
+
+probe process("winbindd").function("winbindd_pam_auth_crap_send") {
+	send_running["winbindd_pam_auth_crap_send"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_auth_crap_send").return {
+	if (!(["winbindd_pam_auth_crap_send"] in send_running))
+		next
+
+	end = gettimeofday_us()
+	start = send_running["winbindd_pam_auth_crap_send"]
+	delete send_running["winbindd_pam_auth_crap_send"]
+
+	start_time["winbindd_pam_auth_crap_send", $return] = start
+	idle_time["winbindd_pam_auth_crap_send", $return] = end
+}
+
+probe process("winbindd").function("winbindd_pam_auth_crap_recv") {
+	if (!(["winbindd_pam_auth_crap_send", $req] in start_time))
+		next
+
+	recv_running["winbindd_pam_auth_crap_recv"] = gettimeofday_us()
+}
+
+probe process("winbindd").function("winbindd_pam_auth_crap_recv").return {
+	if (!(["winbindd_pam_auth_crap_recv"] in recv_running))
+		next
+
+	recv_end = gettimeofday_us()
+	recv_start = recv_running["winbindd_pam_auth_crap_recv"]
+	delete recv_running["winbindd_pam_auth_crap_recv"]
+	recv_runtime = recv_end - recv_start
+
+	req = @entry($req)
+
+	send_begin = start_time["winbindd_pam_auth_crap_send", req]
+	delete start_time["winbindd_pam_auth_crap_send", req]
+	svctime = recv_end - send_begin
+
+	idle = idle_time["winbindd_pam_auth_crap_send", req]
+	delete idle_time["winbindd_pam_auth_crap_send", req]
+	runtime = (idle - send_begin) + recv_runtime
+
+	async_svctime["winbindd_pam_auth_crap_send"] <<< svctime
+	async_runtime["winbindd_pam_auth_crap_send"] <<< runtime
+}
+
+probe end {
+	printf("\n\n")
+
+	printf("Winbind request service time\n")
+	printf("============================\n")
+	foreach ([name] in async_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(async_svctime[name]),
+		       @sum(async_svctime[name]) / 1000,
+		       @min(async_svctime[name]),
+		       @avg(async_svctime[name]),
+		       @max(async_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request runtime\n")
+	printf("=======================\n")
+	foreach ([name] in async_runtime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(async_runtime[name]),
+		       @sum(async_runtime[name]) / 1000,
+		       @min(async_runtime[name]),
+		       @avg(async_runtime[name]),
+		       @max(async_runtime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind domain-child request service time\n")
+	printf("=========================================\n")
+	foreach ([name] in dc_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(dc_svctime[name]),
+		       @sum(dc_svctime[name]) / 1000,
+		       @min(dc_svctime[name]),
+		       @avg(dc_svctime[name]),
+		       @max(dc_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind domain-child AD-backend service time\n")
+	printf("============================================\n")
+	foreach ([name] in backend_svctime) {
+		printf("%-40s count: %5d, sum: %6d ms (min: %6d us, avg: %6d us, max: %6d us)\n",
+		       name,
+		       @count(backend_svctime[name]),
+		       @sum(backend_svctime[name]) / 1000,
+		       @min(backend_svctime[name]),
+		       @avg(backend_svctime[name]),
+		       @max(backend_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request service time distributions (us)\n")
+	printf("===============================================\n")
+	foreach ([name] in async_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(async_svctime[name]))
+	}
+	printf("\n")
+
+	printf("Winbind request runtime distributions (us)\n")
+	printf("==========================================\n")
+	foreach ([name] in async_runtime) {
+		printf("%s:\n", name);
+		println(@hist_log(async_runtime[name]))
+	}
+
+	printf("Winbind domain-child request service time distributions (us)\n")
+	printf("============================================================\n")
+	foreach ([name] in dc_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(dc_svctime[name]))
+	}
+
+	printf("Winbind domain-child backend service time distributions (us)\n")
+	printf("============================================================\n")
+	foreach ([name] in backend_svctime) {
+		printf("%s:\n", name);
+		println(@hist_log(backend_svctime[name]))
+	}
+}
-- 
2.9.4



More information about the samba-technical mailing list