[Samba] Problems encountered when building on SmartOS/Illumos

[Martin Clemons]

Hi everyone,

I spent some time yesterday trying to get samba 4.23.4 to build on SmartOS (image base-64-lts 25.4.0).
In the end I managed to get a working binary but not before working through some bugs blocking the build with the help of Codex.

I’ve attached a summary of the issues below (also available as a gist <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd>).
Would it be worth it to submit each of these issues to the bug tracker? Is anyone else building for SmartOS/Illumos these days?

Thanks!

Martin

Samba 4.23.4 on SmartOS: Bug Report Drafts

Bug 1: auxv_t redefinition in embedded Heimdal on SmartOS

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-1-auxv_t-redefinition-in-embedded-heimdal-on-smartos>
Component: embedded Heimdal (roken)
Affected file: third_party/heimdal/lib/roken/getauxval.h
Build failure signal:
error: conflicting types for 'auxv_t'; have 'struct rk_auxv'
previous declaration from /usr/include/sys/auxv.h
Observed impact: build stops in Heimdal getauxval path.
Expected behavior: build should detect system auxv_t and avoid redefining it.
Local workaround used: guard/restrict local auxv_t typedef on Solaris.
Suggested fix direction: add robust configure/header guard for Solaris auxv_t.
Bug 2: msghdr.msg_flags portability break in messages_dgm.c

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-2-msghdrmsg_flags-portability-break-in-messages_dgmc>
Component: messaging
Affected file: lib/messaging/messages_dgm.c
Build failure signal:
error: 'struct msghdr' has no member named 'msg_flags'
Observed impact: build fails in messaging_dgm_read_handler.
Expected behavior: code should be gated by platform capability detection.
Local workaround used: conditionally skip msg.msg_flags |= MSG_CMSG_CLOEXEC; on SmartOS.
Suggested fix direction: gate on HAVE_STRUCT_MSGHDR_MSG_FLAGS (or equivalent autoconf result).
Bug 3: macro collision on s_addr identifier in tsocket_bsd.c

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-3-macro-collision-on-s_addr-identifier-in-tsocket_bsdc>
Component: tsocket
Affected file: lib/tsocket/tsocket_bsd.c
Build failure signal:
error: expected ... before '.' token
error: 'S_un' undeclared
failure at local declaration char *s_addr = NULL;
Observed impact: SmartOS macro expansion collides with local variable name.
Expected behavior: avoid variable names that collide with system networking macros.
Local workaround used: rename local variable from s_addr to addr_part only within the function scope.
Suggested fix direction: rename local identifier upstream in _tsocket_address_inet_from_hostport_strings.
Bug 4: Solaris quota RPC include/GSS type break

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-4-solaris-quota-rpc-includegss-type-break>
Component: source3 quota support
Affected file: source3/smbd/quotas.c (through RPC includes)
Build failure signal:
from /usr/include/rpc/rpcsec_gss.h: unknown types gss_cred_id_t, gss_ctx_id_t, etc.
Observed impact: quota-related build path fails on SmartOS headers/toolchain.
Expected behavior: quota path should compile or be auto-disabled when required RPC/GSS types are unavailable.
Local workaround used: configure with --without-quotas.
Suggested fix direction: tighten configure detection / conditional compilation for Solaris RPC/GSS variants.
Bug 5: vfs_solarisacl.c declaration/signature/typo regressions

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-5-vfs_solarisaclc-declarationsignaturetypo-regressions>
Component: vfs_solarisacl
Affected file: source3/modules/vfs_solarisacl.c (and declaration in vfs_solarisacl.h)
Build failure signal:
error: static declaration ... follows non-static declaration
incompatible pointer argument (char * passed where const struct smb_filename * expected)
typo call DBG_DEBG (implicit declaration warning)
Observed impact: module fails to compile on SmartOS.
Expected behavior: declaration/definition linkage and signatures should match; debug macro should be valid.
Local workaround used:
remove static mismatch,
pass fsp->fsp_name instead of fsp->fsp_name->base_name,
DBG_DEBG -> DBG_DEBUG.
Suggested fix direction: upstream consistency fix in solarisacl module.
Bug 6: ctime_r arity mismatch in libsmbclient example on SmartOS

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-6-ctime_r-arity-mismatch-in-libsmbclient-example-on-smartos>
Component: examples/libsmbclient
Affected file: examples/libsmbclient/teststat.c
Build failure signal:
error: too few arguments to function 'ctime_r'
SmartOS prototype expects 3 args.
Observed impact: examples target fails to compile.
Expected behavior: use portable wrapper or platform-specific ctime_r call signature.
Local workaround used: changed calls to include buffer length (sizeof(...)).
Suggested fix direction: portability macro for 2-arg vs 3-arg ctime_r.
Bug 7: smbtorture link failure with system Heimdal on Solaris linker

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-7-smbtorture-link-failure-with-system-heimdal-on-solaris-linker>
Component: source4/torture (smbtorture)
Affected target: source4/torture/smbtorture
Link failure signal:
undefined krb5_* symbols (e.g., krb5_get_creds, krb5_c_decrypt)
Solaris ld reports symbols belong to implicit dependency /opt/local/heimdal/lib/libkrb5.so.26
Observed impact: install/build blocked when smbtorture is included in install graph.
Expected behavior: target should link direct Kerberos libs when required (not rely on implicit deps).
Local workaround used: disable smbtorture target to complete installation.
Suggested fix direction: add explicit krb5/hx509/etc link dependencies to smbtorture (or its dependent modules) for Solaris linkers.
Bug 8: Embedded Heimdal libhcrypto-private-samba.so unresolved PKCS#11 symbols on SmartOS

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#bug-8-embedded-heimdal-libhcrypto-private-sambaso-unresolved-pkcs11-symbols-on-smartos>
Component: embedded Heimdal / hcrypto in Samba build graph
Affected artifact: bin/default/third_party/heimdal_build/libhcrypto-private-samba.so
Link failure signal:
undefined symbols hc_EVP_pkcs11_hcrypto_* (e.g. hc_EVP_pkcs11_hcrypto_sha1, ..._aes_256_cbc, ..._md5, etc.)
breaks linking of core targets including source3/smbd/smbd and source3/utils/smbpasswd
Observed impact: file-server binaries fail to link, so build cannot complete with embedded Heimdal.
Expected behavior: embedded Heimdal should either:
compile/link all required PKCS#11 provider symbols, or
disable those call sites when provider symbols are not available.
Local workaround used: built/installed Heimdal separately and switched Samba to system Heimdal via --with-system-heimdalkrb5.
Suggested fix direction:
fix embedded Heimdal feature detection and symbol export/link ordering for SmartOS/Solaris, and
ensure libhcrypto-private-samba.so does not retain unresolved hc_EVP_pkcs11_hcrypto_* references in this configuration.
Optional Global Context

 <https://gist.github.com/martin-c/fae6934b129edde02468d1df7427f3dd#optional-global-context-attach-once-across-reports>
Build profile included:
--without-ad-dc --without-ads --without-ldap --without-quotas --without-gpgme
--with-system-heimdalkrb5
--with-shared-modules=vfs_fruit,vfs_streams_xattr,vfs_catia,vfs_zfsacl
Platform: SmartOS / Solaris userland.