[SCM] The rsync repository. - branch master updated
Rsync CVS commit messages
rsync-cvs at lists.samba.org
Sat Sep 10 20:15:58 UTC 2022
The branch, master has been updated
via 0efa63f2 Use JSON output if --version (-V) is repeated (client side only).
via ae16850d Add support for various SHA checksum digests
via 7e2711bb Improve various things in the checksum code
via b8c2fde3 Try freebsd-13-1 to fix weird wget issue.
via 1f12b196 When deleting a tag, del in the patches dir too.
via bafe73dd Start 3.2.7dev going.
from db5bfe67 Preparing for release of 3.2.6
https://git.samba.org/?p=rsync.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 0efa63f2e601808cae02313311a7331af66a2313
Author: Wayne Davison <wayne at opencoder.net>
Date: Sat Sep 10 11:07:01 2022 -0700
Use JSON output if --version (-V) is repeated (client side only).
commit ae16850dc58e884eb9f5cb7f772342b2db28f471
Author: Wayne Davison <wayne at opencoder.net>
Date: Sat Sep 10 10:23:36 2022 -0700
Add support for various SHA checksum digests
The main purpose of the SHA checksums are to allow the daemon auth code
to pick a stonger digest method when negotiating the auth digest to use.
However, the SHA digests are also available for use in file checksums,
should someon really want to use one of them.
The new digests are listed from strongest to weakest at the start of the
daemon auth list, giving them the highest priority.
The new digests are listed from weakest to strongest near the end of the
checksum list, giving them the lowest priority of use for file
checksums.
commit 7e2711bb2b4b30bc842dd8670c34a87e2ca0c2df
Author: Wayne Davison <wayne at opencoder.net>
Date: Sat Sep 10 09:43:47 2022 -0700
Improve various things in the checksum code
- Size flist checksum data to hold the active size, not the max.
- Add a negotiated hash method to the daemon auth code.
- Use EVP for all openssl digests. This makes it easy to add more
openssl digest methods and avoids deprecation warnings.
- Support a way to re-enable deprecated digests via openssl conf
file and allow a default file to be configured.
- Supply a simple openssl-rsync.cnf file to enable legacy digests.
commit b8c2fde3a54abc1ebef6d375b687395b798412f5
Author: Wayne Davison <wayne at opencoder.net>
Date: Fri Sep 9 13:16:27 2022 -0700
Try freebsd-13-1 to fix weird wget issue.
commit 1f12b196fd049058e1d12bdf8e9b01dd05f656f0
Author: Wayne Davison <wayne at opencoder.net>
Date: Fri Sep 9 12:59:22 2022 -0700
When deleting a tag, del in the patches dir too.
commit bafe73dd5ce57f56b7d67e1f1eec0cafd1db9a7b
Author: Wayne Davison <wayne at opencoder.net>
Date: Fri Sep 9 12:58:22 2022 -0700
Start 3.2.7dev going.
-----------------------------------------------------------------------
Summary of changes:
.cirrus.yml | 2 +-
NEWS.md | 47 +++++
authenticate.c | 11 +-
checksum.c | 431 +++++++++++++++++++++++++++++---------------
clientserver.c | 19 +-
compat.c | 190 +++++++++++++------
configure.ac | 10 +
flist.c | 6 +-
lib/md-defines.h | 20 ++
lib/md5.c | 2 -
lib/mdigest.h | 12 +-
log.c | 10 +-
main.c | 11 ++
match.c | 18 +-
options.c | 2 +-
packaging/openssl-rsync.cnf | 18 ++
packaging/release-rsync | 2 +
receiver.c | 13 +-
rsync.1.md | 14 +-
rsync.h | 11 +-
support/json-rsync-version | 59 ++++++
usage.c | 118 +++++++++---
version.h | 2 +-
xattrs.c | 29 +--
24 files changed, 759 insertions(+), 298 deletions(-)
create mode 100644 packaging/openssl-rsync.cnf
create mode 100755 support/json-rsync-version
Changeset truncated at 500 lines:
diff --git a/.cirrus.yml b/.cirrus.yml
index 16489f8f..33e2685e 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,7 +1,7 @@
freebsd_task:
name: FreeBSD
freebsd_instance:
- image_family: freebsd-13-0
+ image_family: freebsd-13-1
env:
PATH: /usr/local/bin:$PATH
prep_script:
diff --git a/NEWS.md b/NEWS.md
index 02cc2fc7..81d66b63 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,3 +1,49 @@
+# NEWS for rsync 3.2.7 (UNRELEASED)
+
+## Changes in this version:
+
+### BUG FIXES:
+
+- ...
+
+### ENHANCEMENTS:
+
+- Added negotiated daemon-auth support that allows a stronger checksum digest
+ to be used. Added SHA512, SHA256, and SHA1 digests to MD5 & MD4. These new
+ digests are at the highest priority in the new negotiation list.
+
+- Added support for SHA1, SHA256, and SHA512 digests in file checksums. While
+ This tends to be overkill, if someone needs it, it is available. These
+ overly-long checksums are at the lowest priority in the normal checksum
+ negotation list.
+
+- If the `--version` option is repeated (e.g. `-VV`) then the information is
+ output in a (still human-readable) JSON format (client side only).
+
+- The script `support/json-rsync-version` is available to get the JSON style
+ version output from any rsync. The script accepts the version output on
+ stdin **or** the name of an rsync to run as an arg. If the text isn't
+ already in JSON format, the text is translated into equivalent JSON.
+
+### PACKAGING RELATED:
+
+- The checksum code now uses openssl's EVP methods, which gets rid of various
+ deprecation warnings and makes it easy to support more digest methods. On
+ newer systems, the MD4 digest is marked as legacy in the openssl code, which
+ makes openssl refuse to support it via EVP. You can just ignore this and
+ allow the included MD4 code to be used for older rsync connections (when
+ talking to an rsync prior to 3.0.0) or you can configure rsync to tell
+ openssl to enable legacy algorithms (see below).
+
+- A simple openssl config file is supplied that can be optionally installed for
+ rsync to use. If you install packaging/openssl-rsync.cnf to a public spot
+ (such as ` /etc/ssl/openssl-rsync.cnf` or similar) and then configure rsync
+ using `--with-openssl-conf=/path/name.cnf`, this will cause rsync to export
+ the configured path in the OPENSSL_CONF environment variable (when it is not
+ already set). This will enable openssl's MD4 code for rsync to use.
+
+------------------------------------------------------------------------------
+
# NEWS for rsync 3.2.6 (9 Sep 2022)
## Changes in this version:
@@ -4592,6 +4638,7 @@
| RELEASE DATE | VER. | DATE OF COMMIT\* | PROTOCOL |
|--------------|--------|------------------|-------------|
+| ?? Dec 2022 | 3.2.7 | | 31 |
| 09 Sep 2022 | 3.2.6 | | 31 |
| 14 Aug 2022 | 3.2.5 | | 31 |
| 15 Apr 2022 | 3.2.4 | | 31 |
diff --git a/authenticate.c b/authenticate.c
index 4306d167..01b26c08 100644
--- a/authenticate.c
+++ b/authenticate.c
@@ -24,6 +24,7 @@
extern int read_only;
extern char *password_file;
+extern struct name_num_obj valid_auth_checksums;
/***************************************************************************
encode a buffer using base64 - simple and slow algorithm. null terminates
@@ -72,9 +73,9 @@ static void gen_challenge(const char *addr, char *challenge)
SIVAL(input, 20, tv.tv_usec);
SIVAL(input, 24, getpid());
- sum_init(-1, 0);
+ len = sum_init(valid_auth_checksums.negotiated_nni, 0);
sum_update(input, sizeof input);
- len = sum_end(digest);
+ sum_end(digest);
base64_encode(digest, len, challenge, 0);
}
@@ -86,10 +87,10 @@ static void generate_hash(const char *in, const char *challenge, char *out)
char buf[MAX_DIGEST_LEN];
int len;
- sum_init(-1, 0);
+ len = sum_init(valid_auth_checksums.negotiated_nni, 0);
sum_update(in, strlen(in));
sum_update(challenge, strlen(challenge));
- len = sum_end(buf);
+ sum_end(buf);
base64_encode(buf, len, out, 0);
}
@@ -238,6 +239,7 @@ char *auth_server(int f_in, int f_out, int module, const char *host,
if (!users || !*users)
return "";
+ negotiate_daemon_auth(f_out, 0);
gen_challenge(addr, challenge);
io_printf(f_out, "%s%s\n", leader, challenge);
@@ -350,6 +352,7 @@ void auth_client(int fd, const char *user, const char *challenge)
if (!user || !*user)
user = "nobody";
+ negotiate_daemon_auth(-1, 1);
if (!(pass = getpassf(password_file))
&& !(pass = getenv("RSYNC_PASSWORD"))) {
diff --git a/checksum.c b/checksum.c
index 7eb50f17..68ea0fa0 100644
--- a/checksum.c
+++ b/checksum.c
@@ -42,50 +42,101 @@ extern int protocol_version;
extern int proper_seed_order;
extern const char *checksum_choice;
+#define NNI_BUILTIN (1<<0)
+#define NNI_EVP (1<<1)
+#define NNI_EVP_OK (1<<2)
+
struct name_num_item valid_checksums_items[] = {
#ifdef SUPPORT_XXH3
- { CSUM_XXH3_128, "xxh128", NULL },
- { CSUM_XXH3_64, "xxh3", NULL },
+ { CSUM_XXH3_128, 0, "xxh128", NULL },
+ { CSUM_XXH3_64, 0, "xxh3", NULL },
#endif
#ifdef SUPPORT_XXHASH
- { CSUM_XXH64, "xxh64", NULL },
- { CSUM_XXH64, "xxhash", NULL },
+ { CSUM_XXH64, 0, "xxh64", NULL },
+ { CSUM_XXH64, 0, "xxhash", NULL },
+#endif
+ { CSUM_MD5, NNI_BUILTIN|NNI_EVP, "md5", NULL },
+ { CSUM_MD4, NNI_BUILTIN|NNI_EVP, "md4", NULL },
+#ifdef SHA_DIGEST_LENGTH
+ { CSUM_SHA1, NNI_EVP, "sha1", NULL },
+#endif
+#ifdef SHA256_DIGEST_LENGTH
+ { CSUM_SHA256, NNI_EVP, "sha256", NULL },
+#endif
+#ifdef SHA512_DIGEST_LENGTH
+ { CSUM_SHA512, NNI_EVP, "sha512", NULL },
#endif
- { CSUM_MD5, "md5", NULL },
- { CSUM_MD4, "md4", NULL },
- { CSUM_NONE, "none", NULL },
- { 0, NULL, NULL }
+ { CSUM_NONE, 0, "none", NULL },
+ { 0, 0, NULL, NULL }
};
struct name_num_obj valid_checksums = {
- "checksum", NULL, NULL, 0, 0, valid_checksums_items
+ "checksum", NULL, 0, 0, valid_checksums_items
+};
+
+struct name_num_item valid_auth_checksums_items[] = {
+#ifdef SHA512_DIGEST_LENGTH
+ { CSUM_SHA512, NNI_EVP, "sha512", NULL },
+#endif
+#ifdef SHA256_DIGEST_LENGTH
+ { CSUM_SHA256, NNI_EVP, "sha256", NULL },
+#endif
+#ifdef SHA_DIGEST_LENGTH
+ { CSUM_SHA1, NNI_EVP, "sha1", NULL },
+#endif
+ { CSUM_MD5, NNI_BUILTIN|NNI_EVP, "md5", NULL },
+ { CSUM_MD4, NNI_BUILTIN|NNI_EVP, "md4", NULL },
+ { 0, 0, NULL, NULL }
+};
+
+struct name_num_obj valid_auth_checksums = {
+ "daemon auth checksum", NULL, 0, 0, valid_auth_checksums_items
};
-int xfersum_type = 0; /* used for the file transfer checksums */
-int checksum_type = 0; /* used for the pre-transfer (--checksum) checksums */
+/* These cannot make use of openssl, so they're marked just as built-in */
+struct name_num_item implied_checksum_md4 =
+ { CSUM_MD4, NNI_BUILTIN, "md4", NULL };
+struct name_num_item implied_checksum_md5 =
+ { CSUM_MD5, NNI_BUILTIN, "md5", NULL };
+struct name_num_item *xfer_sum_nni; /* used for the transfer checksum2 computations */
+const EVP_MD *xfer_sum_evp_md;
+int xfer_sum_len;
+struct name_num_item *file_sum_nni; /* used for the pre-transfer --checksum computations */
+const EVP_MD *file_sum_evp_md;
+int file_sum_len;
+
+#ifdef USE_OPENSSL
+EVP_MD_CTX *ctx_evp = NULL;
+#endif
static int initialized_choices = 0;
-int parse_csum_name(const char *name, int len)
+struct name_num_item *parse_csum_name(const char *name, int len)
{
struct name_num_item *nni;
if (len < 0 && name)
len = strlen(name);
+ init_checksum_choices();
+
if (!name || (len == 4 && strncasecmp(name, "auto", 4) == 0)) {
- if (protocol_version >= 30)
- return CSUM_MD5;
- if (protocol_version >= 27)
- return CSUM_MD4_OLD;
- if (protocol_version >= 21)
- return CSUM_MD4_BUSTED;
- return CSUM_MD4_ARCHAIC;
+ if (protocol_version >= 30) {
+ if (!proper_seed_order)
+ return &implied_checksum_md5;
+ name = "md5";
+ len = 3;
+ } else {
+ if (protocol_version >= 27)
+ implied_checksum_md4.num = CSUM_MD4_OLD;
+ else if (protocol_version >= 21)
+ implied_checksum_md4.num = CSUM_MD4_BUSTED;
+ else
+ implied_checksum_md4.num = CSUM_MD4_ARCHAIC;
+ return &implied_checksum_md4;
+ }
}
- if (!initialized_choices)
- init_checksum_choices();
-
nni = get_nni_by_name(&valid_checksums, name, len);
if (!nni) {
@@ -93,44 +144,72 @@ int parse_csum_name(const char *name, int len)
exit_cleanup(RERR_UNSUPPORTED);
}
- return nni->num;
+ return nni;
}
-static const char *checksum_name(int num)
+static const EVP_MD *csum_evp_md(struct name_num_item *nni)
{
- struct name_num_item *nni = get_nni_by_num(&valid_checksums, num);
-
- return nni ? nni->name : num < CSUM_MD4 ? "md4" : "UNKNOWN";
+#ifdef USE_OPENSSL
+ const EVP_MD *emd;
+ if (!(nni->flags & NNI_EVP))
+ return NULL;
+
+#ifdef USE_MD5_ASM
+ if (nni->num == CSUM_MD5)
+ emd = NULL;
+ else
+#endif
+ emd = EVP_get_digestbyname(nni->name);
+ if (emd && !(nni->flags & NNI_EVP_OK)) { /* Make sure it works before we advertise it */
+ if (!ctx_evp && !(ctx_evp = EVP_MD_CTX_create()))
+ out_of_memory("csum_evp_md");
+ /* Some routines are marked as legacy and are not enabled in the openssl.cnf file.
+ * If we can't init the emd, we'll fall back to our built-in code. */
+ if (EVP_DigestInit_ex(ctx_evp, emd, NULL) == 0)
+ emd = NULL;
+ else
+ nni->flags = (nni->flags & ~NNI_BUILTIN) | NNI_EVP_OK;
+ }
+ if (!emd)
+ nni->flags &= ~NNI_EVP;
+ return emd;
+#else
+ return NULL;
+#endif
}
void parse_checksum_choice(int final_call)
{
- if (valid_checksums.negotiated_name)
- xfersum_type = checksum_type = valid_checksums.negotiated_num;
+ if (valid_checksums.negotiated_nni)
+ xfer_sum_nni = file_sum_nni = valid_checksums.negotiated_nni;
else {
char *cp = checksum_choice ? strchr(checksum_choice, ',') : NULL;
if (cp) {
- xfersum_type = parse_csum_name(checksum_choice, cp - checksum_choice);
- checksum_type = parse_csum_name(cp+1, -1);
+ xfer_sum_nni = parse_csum_name(checksum_choice, cp - checksum_choice);
+ file_sum_nni = parse_csum_name(cp+1, -1);
} else
- xfersum_type = checksum_type = parse_csum_name(checksum_choice, -1);
+ xfer_sum_nni = file_sum_nni = parse_csum_name(checksum_choice, -1);
if (am_server && checksum_choice)
- validate_choice_vs_env(NSTR_CHECKSUM, xfersum_type, checksum_type);
+ validate_choice_vs_env(NSTR_CHECKSUM, xfer_sum_nni->num, file_sum_nni->num);
}
+ xfer_sum_len = csum_len_for_type(xfer_sum_nni->num, 0);
+ file_sum_len = csum_len_for_type(file_sum_nni->num, 0);
+ xfer_sum_evp_md = csum_evp_md(xfer_sum_nni);
+ file_sum_evp_md = csum_evp_md(file_sum_nni);
- if (xfersum_type == CSUM_NONE)
+ if (xfer_sum_nni->num == CSUM_NONE)
whole_file = 1;
/* Snag the checksum name for both write_batch's option output & the following debug output. */
- if (valid_checksums.negotiated_name)
- checksum_choice = valid_checksums.negotiated_name;
+ if (valid_checksums.negotiated_nni)
+ checksum_choice = valid_checksums.negotiated_nni->name;
else if (checksum_choice == NULL)
- checksum_choice = checksum_name(xfersum_type);
+ checksum_choice = xfer_sum_nni->name;
if (final_call && DEBUG_GTE(NSTR, am_server ? 3 : 1)) {
rprintf(FINFO, "%s%s checksum: %s\n",
am_server ? "Server" : "Client",
- valid_checksums.negotiated_name ? " negotiated" : "",
+ valid_checksums.negotiated_nni ? " negotiated" : "",
checksum_choice);
}
}
@@ -150,6 +229,18 @@ int csum_len_for_type(int cst, BOOL flist_csum)
return MD4_DIGEST_LEN;
case CSUM_MD5:
return MD5_DIGEST_LEN;
+#ifdef SHA_DIGEST_LENGTH
+ case CSUM_SHA1:
+ return SHA_DIGEST_LENGTH;
+#endif
+#ifdef SHA256_DIGEST_LENGTH
+ case CSUM_SHA256:
+ return SHA256_DIGEST_LENGTH;
+#endif
+#ifdef SHA512_DIGEST_LENGTH
+ case CSUM_SHA512:
+ return SHA512_DIGEST_LENGTH;
+#endif
case CSUM_XXH64:
case CSUM_XXH3_64:
return 64/8;
@@ -175,6 +266,9 @@ int canonical_checksum(int csum_type)
break;
case CSUM_MD4:
case CSUM_MD5:
+ case CSUM_SHA1:
+ case CSUM_SHA256:
+ case CSUM_SHA512:
return -1;
case CSUM_XXH64:
case CSUM_XXH3_64:
@@ -211,7 +305,22 @@ uint32 get_checksum1(char *buf1, int32 len)
void get_checksum2(char *buf, int32 len, char *sum)
{
- switch (xfersum_type) {
+#ifdef USE_OPENSSL
+ if (xfer_sum_evp_md) {
+ static EVP_MD_CTX *evp = NULL;
+ uchar seedbuf[4];
+ if (!evp && !(evp = EVP_MD_CTX_create()))
+ out_of_memory("get_checksum2");
+ EVP_DigestInit_ex(evp, xfer_sum_evp_md, NULL);
+ if (checksum_seed) {
+ SIVALu(seedbuf, 0, checksum_seed);
+ EVP_DigestUpdate(evp, seedbuf, 4);
+ }
+ EVP_DigestUpdate(evp, (uchar *)buf, len);
+ EVP_DigestFinal_ex(evp, (uchar *)sum, NULL);
+ } else
+#endif
+ switch (xfer_sum_nni->num) {
#ifdef SUPPORT_XXHASH
case CSUM_XXH64:
SIVAL64(sum, 0, XXH64(buf, len, checksum_seed));
@@ -229,7 +338,7 @@ void get_checksum2(char *buf, int32 len, char *sum)
}
#endif
case CSUM_MD5: {
- md5_context m5;
+ md_context m5;
uchar seedbuf[4];
md5_begin(&m5);
if (proper_seed_order) {
@@ -249,20 +358,6 @@ void get_checksum2(char *buf, int32 len, char *sum)
break;
}
case CSUM_MD4:
-#ifdef USE_OPENSSL
- {
- MD4_CTX m4;
- MD4_Init(&m4);
- MD4_Update(&m4, (uchar *)buf, len);
- if (checksum_seed) {
- uchar seedbuf[4];
- SIVALu(seedbuf, 0, checksum_seed);
- MD4_Update(&m4, seedbuf, 4);
- }
- MD4_Final((uchar *)sum, &m4);
- break;
- }
-#endif
case CSUM_MD4_OLD:
case CSUM_MD4_BUSTED:
case CSUM_MD4_ARCHAIC: {
@@ -295,7 +390,7 @@ void get_checksum2(char *buf, int32 len, char *sum)
* are multiples of 64. This is fixed by calling mdfour_update()
* even when there are no more bytes.
*/
- if (len - i > 0 || xfersum_type > CSUM_MD4_BUSTED)
+ if (len - i > 0 || xfer_sum_nni->num > CSUM_MD4_BUSTED)
mdfour_update(&m, (uchar *)(buf1+i), len-i);
mdfour_result(&m, (uchar *)sum);
@@ -313,15 +408,33 @@ void file_checksum(const char *fname, const STRUCT_STAT *st_p, char *sum)
int32 remainder;
int fd;
- memset(sum, 0, MAX_DIGEST_LEN);
-
fd = do_open(fname, O_RDONLY, 0);
- if (fd == -1)
+ if (fd == -1) {
+ memset(sum, 0, file_sum_len);
return;
+ }
buf = map_file(fd, len, MAX_MAP_SIZE, CHUNK_SIZE);
- switch (checksum_type) {
+#ifdef USE_OPENSSL
+ if (file_sum_evp_md) {
+ static EVP_MD_CTX *evp = NULL;
+ if (!evp && !(evp = EVP_MD_CTX_create()))
+ out_of_memory("file_checksum");
+
+ EVP_DigestInit_ex(evp, file_sum_evp_md, NULL);
+
+ for (i = 0; i + CHUNK_SIZE <= len; i += CHUNK_SIZE)
+ EVP_DigestUpdate(evp, (uchar *)map_ptr(buf, i, CHUNK_SIZE), CHUNK_SIZE);
+
+ remainder = (int32)(len - i);
+ if (remainder > 0)
+ EVP_DigestUpdate(evp, (uchar *)map_ptr(buf, i, remainder), remainder);
+
+ EVP_DigestFinal_ex(evp, (uchar *)sum, NULL);
+ } else
+#endif
+ switch (file_sum_nni->num) {
#ifdef SUPPORT_XXHASH
case CSUM_XXH64: {
static XXH64_state_t* state = NULL;
@@ -381,7 +494,7 @@ void file_checksum(const char *fname, const STRUCT_STAT *st_p, char *sum)
}
#endif
case CSUM_MD5: {
- md5_context m5;
+ md_context m5;
md5_begin(&m5);
@@ -396,23 +509,6 @@ void file_checksum(const char *fname, const STRUCT_STAT *st_p, char *sum)
break;
}
case CSUM_MD4:
-#ifdef USE_OPENSSL
- {
- MD4_CTX m4;
-
- MD4_Init(&m4);
-
- for (i = 0; i + CHUNK_SIZE <= len; i += CHUNK_SIZE)
- MD4_Update(&m4, (uchar *)map_ptr(buf, i, CHUNK_SIZE), CHUNK_SIZE);
-
- remainder = (int32)(len - i);
- if (remainder > 0)
- MD4_Update(&m4, (uchar *)map_ptr(buf, i, remainder), remainder);
--
The rsync repository.
More information about the rsync-cvs
mailing list