<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
	<channel>
		<atom:link href="https://dev1galaxy.org/extern.php?action=feed&amp;tid=5003&amp;type=rss" rel="self" type="application/rss+xml" />
		<title><![CDATA[Dev1 Galaxy Forum / [SOLVED] rsync operation takes ages]]></title>
		<link>https://dev1galaxy.org/viewtopic.php?id=5003</link>
		<description><![CDATA[The most recent posts in [SOLVED] rsync operation takes ages.]]></description>
		<lastBuildDate>Wed, 27 Apr 2022 11:53:51 +0000</lastBuildDate>
		<generator>FluxBB</generator>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35772#p35772</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>GlennW wrote:</cite><blockquote><div><p>... an old 4 port modem/routrer as a switch.</p></div></blockquote></div><div class="quotebox"><cite>PedroReina wrote:</cite><blockquote><div><p>... a really good piece of advice.</p></div></blockquote></div><p>Indeed ...<br />I had first looked into that option. (See updated post @2022-04-25 17:34:35)</p><p>Locally, a used (decent quality) Gb router would cost me a bit more than what I paid for a four/five port unmanaged Gb switch.&#160; &#160;<br />So I got one of those. Plug everything up, power it and that&#039;s it. Just works.</p><p>It has a metal case and is unobstusive (100*98*25) so that once I threw it under the desk, I forgot all about it.<br />Slipped my mind and neglected to mention it in the post, maybe the old cache is acting up. </p><p>Sorry about that.</p><p>Thank you both for your input.</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Wed, 27 Apr 2022 11:53:51 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35772#p35772</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35770#p35770</link>
			<description><![CDATA[<div class="quotebox"><cite>GlennW wrote:</cite><blockquote><div><p>I used to use an old 4 port modem/routrer as a switch.</p></div></blockquote></div><p>I think that this is a really good piece of advice. I&#039;m using right now an old router as an WiFi access point. But it the router is too old, maybe the switch could only cope with 100 Mbps. Modern routers can get up to 1Gbps. Anyone interested should check this out.</p>]]></description>
			<author><![CDATA[dummy@example.com (PedroReina)]]></author>
			<pubDate>Wed, 27 Apr 2022 11:19:01 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35770#p35770</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35769#p35769</link>
			<description><![CDATA[<p>Hi, for what it&#039;s worth, I used to use an old 4 port modem/routrer as a switch. </p><p>I found, without the external phone line connected (I can&#039;t remember if it was adsl or dsl) it acted as though it was not a modem.</p><p>giving me 4 extra eth ports (it was some time ago)</p><p>all the best.</p>]]></description>
			<author><![CDATA[dummy@example.com (GlennW)]]></author>
			<pubDate>Wed, 27 Apr 2022 08:45:12 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35769#p35769</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35764#p35764</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>Head_on_a_Stick wrote:</cite><blockquote><div><p>Looks like <a href="https://wiki.archlinux.org/title/Rsync#As_a_daemon" rel="nofollow">https://wiki.archlinux.org/title/Rsync#As_a_daemon</a></p></div></blockquote></div><p>Ahh ...</p><p>Thanks a lot for that. ;^ )<br />I&#039;ll have a look and see what I get from using that method.</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Tue, 26 Apr 2022 19:04:57 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35764#p35764</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35763#p35763</link>
			<description><![CDATA[<div class="quotebox"><cite>Altoid wrote:</cite><blockquote><div><p>It would seem that he achieved speeds using just the <span class="bbc">rsync</span> protocol and bypassing <span class="bbc">ssh</span>. <br />I don&#039;t undertand how he&#039;s set this up and cannot find how to contact the guy.</p></div></blockquote></div><p>Looks like <a href="https://wiki.archlinux.org/title/Rsync#As_a_daemon" rel="nofollow">https://wiki.archlinux.org/title/Rsync#As_a_daemon</a></p>]]></description>
			<author><![CDATA[dummy@example.com (Head_on_a_Stick)]]></author>
			<pubDate>Tue, 26 Apr 2022 17:34:36 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35763#p35763</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35761#p35761</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>Head_on_a_Stick wrote:</cite><blockquote><div><p>... can&#039;t actually disable encryption over ssh ...<br />... use a simple cipher like Blowfish ...</p><div class="codebox"><pre><code># rsync -e &quot;ssh -o compression=no -c blowfish&quot; -a -stats ~/media/bckups root@192.168.1.3:/mnt/sda3</code></pre></div></div></blockquote></div><p>I&#039;m afraid that <span class="bbc">dropbear</span> does not do <span class="bbc">blowfish</span>.</p><div class="codebox"><pre><code>~$ rsync -e &quot;ssh -o compression=no -c blowfish&quot; -a --stats /media/bckups root@192.168.1.3:/mnt/sda3
Unknown cipher type &#039;blowfish&#039;
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
~$ </code></pre></div><p>See: <a href="https://dropbear.nl/mirror/CHANGES" rel="nofollow">https://dropbear.nl/mirror/CHANGES</a></p><div class="quotebox"><blockquote><div><p>CBC ciphers, 3DES, hmac-sha1-96, and x11 forwarding are now disabled by default.<br />They can be set in localoptions.h if required.<br />Blowfish has been removed.</p></div></blockquote></div><div class="quotebox"><cite>Head_on_a_Stick wrote:</cite><blockquote><div><p>... without encryption use rsh instead, as supplied by the rsh-client &amp; rsh-server packages ...</p></div></blockquote></div><p>Unfortunately, I have not found the <span class="bbc">rsh</span> package in the OpenWRT repository so that&#039;s a dead end.</p><p>Looking around for similar situations, I found this:</p><p><a href="https://www.glidk.com/2021/12/22/benchmarking-crappy-router" rel="nofollow">https://www.glidk.com/2021/12/22/benchm … ppy-router</a></p><p>It&#039;s a post by a chap using a small travel router with a lower spec than the WD-MBL:</p><div class="codebox"><pre><code>Brand/Name		WD-MBL		GL-AR750
CPU:			PowePC 44x     Qualcomm Atheros
Model:			APM82181	QCA9531
CPU Cores:		1		1
CPU MHz:		800		650       &lt;----- #
Flash Mb:		512		16
RAM MB:			256		128       &lt;----- #
BogoMIPS:		1600		432.53    &lt;----- # </code></pre></div><p>It would seem that he achieved speeds using just the <span class="bbc">rsync</span> protocol and bypassing <span class="bbc">ssh</span>. <br />I don&#039;t undertand how he&#039;s set this up and cannot find how to contact the guy.</p><p>The WD-MBL, save for the HDD/SD card difference, seems to be much heftier. <br />Could I be able to get at least his 10.0Mb/s?</p><p>Can you make any sense of what he&#039;s done there?</p><p>Thanks in advance.</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Tue, 26 Apr 2022 16:03:18 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35761#p35761</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35759#p35759</link>
			<description><![CDATA[<div class="quotebox"><cite>Altoid wrote:</cite><blockquote><div><p>But I do need rsync.<br />I use <span class="bbc">ssh</span> to login to the NAS, how do I disable encryption?</p></div></blockquote></div><p>Oops, sorry, looks like you can&#039;t actually disable encryption over ssh but you can use a simple cipher like Blowfish, which should be quick, along with no compression:</p><div class="codebox"><pre><code># rsync -e &quot;ssh -o compression=no -c blowfish&quot; -a -stats ~/media/bckups root@192.168.1.3:/mnt/sda3</code></pre></div><p>Reference: <a href="https://galaxysd.github.io/20160302/Fastest-Way-Rsync" rel="nofollow">https://galaxysd.github.io/20160302/Fastest-Way-Rsync</a></p><p>For transfers without encryption use rsh instead, as supplied by the rsh-client &amp; rsh-server packages (<span class="bbc">/usr/bin/netkit-rsh</span> &amp; <span class="bbc">/usr/bin/netkit-rshd</span>).</p>]]></description>
			<author><![CDATA[dummy@example.com (Head_on_a_Stick)]]></author>
			<pubDate>Tue, 26 Apr 2022 08:38:49 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35759#p35759</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35754#p35754</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>thierrybo wrote:</cite><blockquote><div><p>... backup my desktop and nas with rclone (&quot;rsync for the cloud&quot;) ...</p></div></blockquote></div><p>Hmm ...<br />Sorry, I don&#039;t do anything cloudy and have no trust in that type of thing.<br />But that&#039;s just <em>me</em>, YMMV. </p><p>Notwithstanding, thank you very much for your input. ;^)</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Mon, 25 Apr 2022 20:37:23 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35754#p35754</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35753#p35753</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>Head_on_a_Stick wrote:</cite><blockquote><div><p>Disable encryption for rsync operations over ssh ...</p></div></blockquote></div><div class="quotebox"><cite>Altoid wrote:</cite><blockquote><div><p>My box has an on-board 82566DM-2 Gigabit Network adapter.&#160; &#160; &#160;<br />The WD My Book NAS has a Gigabit Ethernet port.</p></div></blockquote></div><p>---</p><div class="quotebox"><cite>Altoid, several days later wrote:</cite><blockquote><div><p>Forgot to mention that at <span class="bbu">this</span> point I purchased/installed a four/five port unmanaged Gb switch (TL-SG105).</p></div></blockquote></div><p>---</p><p>Part of this problem had to do with the CAT5e cables I was using.<br />Found out when I used <span class="bbc">ethtool</span> to check the on-board ethernet adapter on the NAS and saw it was reporting 100M:</p><div class="codebox"><pre><code>root@OpenWrt:~# ethtool eth0
Settings for eth0:
--- snip ---
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 100Mb/s    &lt;-------------------------- #####
    Duplex: Full
--- snip ---
root@OpenWrt:~# </code></pre></div><p>With a set of new CAT5e cables and properly negotiating 1000Mb/s between my box and the NAS, I tested the link with <span class="bbc">iperf</span>:</p><div class="codebox"><pre><code>root@OpenWrt:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.3 port 5001 connected with 192.168.1.2 port 35408
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   771 MBytes   645 Mbits/sec
root@OpenWrt:~# </code></pre></div><div class="codebox"><pre><code>groucho@devuan:~$ iperf -c 192.168.1.3
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.2 port 35408 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   771 MBytes   646 Mbits/sec
pcl@devuan:~$ </code></pre></div><p>Good enough, I guess.</p><p>But I was expecting something like the 94Mb /s I get between the NAS and my netbook which has a 10/100 ethernet adapter.<br />ie: if with a 10/100 connection I can get 94% of 100M, I was expecting at least 85% from the box to NAS connection at 1000M - 850 Mbits /s instead of 645 Mbits /s.</p><p>I&#039;m assuming it is <em>proportional</em>, it probably isn&#039;t.</p><p>I&#039;ve just finished running the same <span class="bbc">rsync</span> job that took over 18 hours (first time/empty destination) the last time I ran it ie: before switching all the patch cables for new CAT5e ones and as a result getting the <span class="bbc">iperf</span> speeds reported above, 646 Mbits/sec</p><p>All through the job, my <span class="bbc">conky</span> panel reported a speed of ~7.45MiB /s</p><div class="codebox"><pre><code>    Incoming: 114KiB /s
    Outgoing: 7.46MiB /s</code></pre></div><p>These were the NAS&#039;s CPU/Memory loads during the job:</p><div class="codebox"><pre><code>PID   Owner  CPU   Mem    Process
 213  root    9%    0%    kswapd0
1904  root   67%    0%    Dropbear
1905  root    0%    7%    rsync  --server
1906  root   22%   13%    rsync  --server</code></pre></div><p>My guess is that NAS&#039;s CPU is totally maxed out.<br />ie: just <span class="bbc">kswap</span> 9% + <span class="bbc">dropbear</span> 67% + <span class="bbc">rsync</span> 22% =&gt;&gt;&gt; 98%</p><p><span class="bbc">dropbear</span> is the SSH2 server/client included in OpenWRT installation image, only 82k.<br />Evidently, with this hardware* a <span class="bbu">67% CPU load</span> is rather expensive to run.</p><p>*Applied Micro APM82181 @800 MHz + 256 MB RAM</p><div class="quotebox"><cite>Head_on_a_Stick wrote:</cite><blockquote><div><p>... use rsh instead of ssh (but only do the latter on a local network ...</p></div></blockquote></div><p>I don&#039;t really neeed encryption as it is all local traffic.<br />ie: going nowhere else but from a back-up drive inside my box to the NAS under my desk.</p><p>But I do need rsync.<br />I use <span class="bbc">ssh</span> to login to the NAS, how do I disable encryption?</p><p>Have never used/heard of <span class="bbc">rsh</span>, have to look it up.</p><p>Thank you very much for your input.</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Mon, 25 Apr 2022 20:34:35 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35753#p35753</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35751#p35751</link>
			<description><![CDATA[<p>For your information, I backup my desktop and nas with rclone (&quot;rsync for the cloud&quot;) to an &quot;OpenStack Swift&quot; provider and I ask rclone to encrypt data.</p><p>When I start a backup, it takes on week to complete on the nas ans 2 days on my computer ....</p>]]></description>
			<author><![CDATA[dummy@example.com (thierrybo)]]></author>
			<pubDate>Mon, 25 Apr 2022 18:16:34 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35751#p35751</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35748#p35748</link>
			<description><![CDATA[<p>Disable encryption for rsync operations over ssh or use rsh instead of ssh (but only do the latter on a local network because it&#039;s not very secure).</p>]]></description>
			<author><![CDATA[dummy@example.com (Head_on_a_Stick)]]></author>
			<pubDate>Mon, 25 Apr 2022 14:04:55 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35748#p35748</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35722#p35722</link>
			<description><![CDATA[<div class="quotebox"><cite>Altoid wrote:</cite><blockquote><div><p>Probably with the same infamous Telefonica ADSL routers. 8^|</p></div></blockquote></div><p>No Telefónica staff at home <img src="https://dev1galaxy.org/img/smilies/smile.png" width="15" height="15" alt="smile" /></p>]]></description>
			<author><![CDATA[dummy@example.com (PedroReina)]]></author>
			<pubDate>Fri, 22 Apr 2022 15:09:20 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35722#p35722</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35721#p35721</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>PedroReina wrote:</cite><blockquote><div><p>... had been very similiar to yours ...</p></div></blockquote></div><p>Probably with the same infamous Telefonica ADSL routers. 8^|</p><div class="quotebox"><cite>PedroReina wrote:</cite><blockquote><div><p>You are correct ...<br />... with 4 up to 16 ports.</p></div></blockquote></div><p>Great.<br />But I have no need for more than 4 ports at the moment.<br />Unless I get a very good price for one with more.</p><p>Thanks a lot for for your input.</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Fri, 22 Apr 2022 11:18:31 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35721#p35721</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35720#p35720</link>
			<description><![CDATA[<div class="quotebox"><cite>Altoid wrote:</cite><blockquote><div><p>That way the link from box to NAS would go through the GB router/switch and not through the telco router with the 100M ports.<br />Am I correct? If so, maybe just a &#039;dumb&#039; desktop switch would do the trick?</p></div></blockquote></div><p>My set up had been very similiar to yours, but nowadays my router is also GB; anyway: for years my router has been only 100M.</p><p>You are correct, any simple <strong>GB switch</strong> would be fine. Depending of your needs and budget, you can choose a model with 4 up to 16 ports.</p><p>Another option I used was installing a second GB network card both in the client and the server and get a direct conexion. Speed transfer was twice than through the switch, but I don&#039;t need so much speed now.</p><p>And the first copy can be very time expensive, as you are experiencing, so I do it using a SATA case attached to the main computer. It can be troublesome, but only when you set it up, later it paids off.</p>]]></description>
			<author><![CDATA[dummy@example.com (PedroReina)]]></author>
			<pubDate>Fri, 22 Apr 2022 09:46:11 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35720#p35720</guid>
		</item>
		<item>
			<title><![CDATA[Re: [SOLVED] rsync operation takes ages]]></title>
			<link>https://dev1galaxy.org/viewtopic.php?pid=35719#p35719</link>
			<description><![CDATA[<p>Hello:</p><div class="quotebox"><cite>fsmithred wrote:</cite><blockquote><div><p>For the initial transfer, maybe you can bypass the router ...</p></div></blockquote></div><p>Yes, that would work.<br />But it&#039;s rather a hassle to do the setup.</p><div class="quotebox"><cite>fsmithred wrote:</cite><blockquote><div><p>... you could add a faster router ...</p></div></blockquote></div><p>I&#039;ve been looking at second hand 4/5 port gigabit routers and desktop switches to plug in all my gear ie: box, netbook and NAS. <br />Nothing into the router save the new router/switch. </p><p>That way the link from box to NAS would go through the GB router/switch and not through the telco router with the 100M ports.<br />Am I correct?</p><p>If so, maybe just a &#039;dumb&#039; desktop switch would do the trick?</p><div class="quotebox"><cite>fsmithred wrote:</cite><blockquote><div><p>Not sure about how the phone works but it might be possible ...</p></div></blockquote></div><p>I am having a contentious issue with the local telco regarding VoIP, which they are intent in shoving down my throat.<br />I don&#039;t want it but they insist, to the point of threatening to cut my land line and leave me without a phone.</p><p>So I&#039;d rather not muck with the ADSL router&#039;s setup.<br />They have a habit of resetting it remotely and screwing up the DNS settings pointing to a VM running inside my box (Pi-Hole+Unbound).</p><p>Think a &#039;dumb&#039; switch may do well enough?</p><p>Thanks for your input.</p><p>Best,</p><p>A.</p>]]></description>
			<author><![CDATA[dummy@example.com (Altoid)]]></author>
			<pubDate>Fri, 22 Apr 2022 07:45:35 +0000</pubDate>
			<guid>https://dev1galaxy.org/viewtopic.php?pid=35719#p35719</guid>
		</item>
	</channel>
</rss>
