<![CDATA[Dev1 Galaxy Forum / wget download articles linked to from a page]]> https://dev1galaxy.org/viewtopic.php?id=5730 Wed, 31 May 2023 08:14:38 +0000 FluxBB <![CDATA[Re: wget download articles linked to from a page]]> https://dev1galaxy.org/viewtopic.php?pid=42429#p42429 Camtaf wrote:

Maybe use sed to remove the <link> </link>, then put the list into a 'for in' loop(?)

kind of a workaround... put the links in a file and then

wget -i file

thank you!

]]> Wed, 31 May 2023 08:14:38 +0000 https://dev1galaxy.org/viewtopic.php?pid=42429#p42429 <![CDATA[Re: wget download articles linked to from a page]]> https://dev1galaxy.org/viewtopic.php?pid=42427#p42427 Hi, I used to download electronics pages with httrack, a webpage/site downloader. The package is in the devuan repos for Daedalus.

With the page on your computer you may move the docs you want and then clean up the left-overs to the rubbish bin.

I hope this helps you.

]]> Tue, 30 May 2023 19:34:55 +0000 https://dev1galaxy.org/viewtopic.php?pid=42427#p42427 <![CDATA[Re: wget download articles linked to from a page]]> https://dev1galaxy.org/viewtopic.php?pid=42425#p42425 Maybe use sed to remove the <link> </link>, then put the list into a 'for in' loop(?)

]]> Tue, 30 May 2023 18:57:39 +0000 https://dev1galaxy.org/viewtopic.php?pid=42425#p42425 <![CDATA[wget download articles linked to from a page]]> https://dev1galaxy.org/viewtopic.php?pid=42421#p42421 Hello everybody!

I want to download all the articles this page

https://shepherdexpress.com/lifestyle/out-of-my-mind

has links to.

I use an alias

type get
get is aliased to `wget -mkEpnp'

but it doesn't work. There is only one downloaded page which is '.rss'

ls ~/shepherdexpress.com/lifestyle/out-of-my-mind/
index.rss

This '.rss' page contains links to the articles 

perl -ne 'print if /link/..!/\\\s*$/' index.rss
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/do-something/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/aviophobia/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/just-how-incarnated-are-you/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/ageism-hurts-more-than-seniors/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/does-the-nose-know/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/meet-your-therapist-nature/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/an-overlooked-key-to-happiness/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/health-anxiety-hurts/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/messages-from-the-embodied-mind/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/when-people-mess-with-your-autonomy/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/how-to-lose-friends-and-alienate-people/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/can-dreaming-heal-emotional-wounds/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/alcohol-is-not-your-friend/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/family-traits-that-hurt-or-heal/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/loves-lessons-learned/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/moneys-impact-on-mental-health/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/there-are-monsters-among-us/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/the-power-of-distraction/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/pathological-liars/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/your-nights-make-your-days/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/when-bad-things-happen/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/looking-back-to-envision-forward/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/christmas-has-an-identity-crisis/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/why-employee-performance-reviews-suck/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/your-brain-on-animals/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/verbal-abuse-is-a-neurotoxin/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/grateful-people-are-happy-people/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/male-dating-dysfunction/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/your-brain-on-television/</link>
<link>https://shepherdexpress.com/lifestyle/out-of-my-mind/got-election-angst/</link>

but for some reason 'wget' doesn't download them.

I also tried 'r' and 'l1' and 'l2' options but it made no difference.

get -l 1 -r --convert-links

Any suggestions how to download those articles are appreciated?
Thank you!

]]> Tue, 30 May 2023 14:59:27 +0000 https://dev1galaxy.org/viewtopic.php?pid=42421#p42421