web archiving

a mini-intro to web arching, with a heavy focus on my personal practices.

the goal

"web archiving" is a deceptively simple term that encompasses many practices, which is why i believe the first step is to determine what you specifically want out of this process. for example, if you want to save a capture of a website (like the wayback machine), then you'd probably be best off with a .warc file or bookmarking program. my goal in my archiving is to save the content to my disk in as plain of files as possible so that it can be experienced in its original state (or as close as i can get to it), while requiring minimal tools and no internet connection. therefore, this guide will be focused more towards "mirroring" content.

tools

basic website mirroring with wget

wget, as mentioned, is a command line tool, meaning we'll be configuring it in pure text as opposed to a settings menu. if you're not familiar with the command line, mozilla has a good jumping off point that covers all three major OSes (i would apologize to any of the BSD forks, but if you're using a BSD OS you don't need me to tell you how to use a terminal). the main thing you need to know is that command line tools are configured through "flags", or options appended to the tool's command. the exact structure of these vary from OS to OS (and sometimes tool to tool), but we're only focused on wget's flag structure right now.

for example, if we had the tool print and wanted to print some text, we could call print "hello world". if we wanted to use print's option to print the text in uppercase, we could call print --upper "hello world". in this structure, print is our tool, --upper is the flag, and "hello world" is the input print is acting upon. not all flags are on/off switches, since not every option in a menu is a checkbox, which means they can be assigned a value. if we wanted our print command from earlier to print the text in a specific color, we could use print --upper --color=green "hello world". i'll explain each flag i use and why i do, so don't worry if this is still confusing, but that's the basic structure you should expect.

the full list of flags can be found in the manual (run man wget if your machine has it or find an online copy) if the site doesn't play nice with the sample configuration. if none of those help, you may be dealing with a site that isn't static. you can test this by disabling javascript and loading the site (ublock origin is the easiest way to do this, just get the extension and disable JS in the menu) to see if it breaks. speaking of...

a note on javascript

the tools and methods above work really well for static sites, but, unfortunately, it's 2026 and everything must be a framework in a framework in a framework that requires a million years to run a billion lines of JS before the actual content can be displayed. fortunately, for me, i've not had to deal with this (until, ironically, making this guide). unfortunately, for you, my knowledge about web archiving becomes spotty at best from here on. i've attempted to find tools that help fill this gap, but, with the exception of maybe archivebox (i haven't used it to its fullest), archiving sites that rely on JS will, to my knowledge, require some scripting, knowledge of web automation, and general programming intuition.

an example

to show you an archiving workflow in action, i'll walk you through how i'd archive a website, specifically the one you're on right now.

my first guess was to run the wget command we formulated earlier, but something's not right. wget only got the index page and its assets, which happens because i use JS to load the sidebar dynamically so that i don't have to edit every .html file when i add a new item. since wget doesn't render the site, it doesn't see the sidebar and can't find the other pages (if you're reading this far enough into the future, maybe i've finally made the site work without JS). there are probably more advanced tools to help with this issue that i could research, but, since this site is mostly static and i don't scrape dynamic websites often, it's probably not worth setting them up or making an entire scraping program. instead, it seems that if we were to load the page and grab the navigation links for wget it'd have no issue finishing it's job. it is hacky, but let's try rendering the site and extracting the sidebar's <a> tags programmatically. for this, we need a web automation library (i told you we'd come back to it) that can render the website as it would be shown in a browser and act upon it. i'll be using python and selenium since i'm the most used to those, but if you really wanna "modernize" this you could playwright.

first, i'll set the URL for the website at the top of the python file, since i don't plan on reusing this code, for future use and set up selenium. i'll also go ahead and set selenium to be "headless" so that i don't have to deal with visible browser windows.

# import selenium and its libraries
from selenium import webdriver
from selenium.webdriver.firefox.options import Options

# if you wanted to make this more robust you could take the URL as input
URL = "https://snailboy.neocities.org/"

# we don't need to see the browser when doing it
options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

i'll have the program print the URLs i want wget to crawl, which includes the base URL, so that i can pipe them into wget through bash.

# load URL and print it to the list
driver.get(URL)
print(URL)

if i only wanted a tags that were in the sidebar i could specify only elements inside <nav>, but since there are no other <a> tags on the homepage that i'm concerned about i'll just grab all of them. there are external links, so i need to only print URLs that contain the base URL (unless i also want to mirror the entirety of tumblr, which i feel will get me banned, so best not).

# extract link tags
links = driver.find_elements(By.TAG_NAME, "a")
for link in links:
    href = link.get_attribute("href")
    if URL in href: # check that it's not a different site
        print(href)

hmmm... after testing the list of links with wget it's nearly there, but it doesn't get the sidebar's external link icons... which means we should also tell it to get the image source URLs. just to be safe, i'll also check that those URLs are contained in the base URL. finally, i need to close the driver.

# extract image tags
images = driver.find_elements(By.TAG_NAME, "img")
for img in images:
    src = img.get_attribute("src")
    if URL in src:
        print(src)

# close browser when done
driver.quit()

# import selenium and its libraries
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

# if you wanted to make this more robust you could take the URL as input
URL = "https://snailboy.neocities.org/"

# we don't need to see the browser when doing it
options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

# initialize browser
driver = webdriver.Firefox(options=options)

# load URL and print it to the list
driver.get(URL)
print(URL)

# extract link tags
links = driver.find_elements(By.TAG_NAME, "a")
for link in links:
    href = link.get_attribute("href")
    if URL in href: # check that it's not a different site
        print(href)

# extract image tags
images = driver.find_elements(By.TAG_NAME, "img")
for img in images:
    src = img.get_attribute("src")
    if URL in src:
        print(src)

# close browser when done
driver.quit()

now i just pass the URLs into wget with py mirror.py | xargs wget --mirror --execute robots=off --page-requisites --adjust-extension --convert-links --no-verbose --wait 1 --random-wait and... it kinda works? the issue with downloading the URLs separately is that wget doesn't see them as a cohesive website and doesn't quite fix all the links. i could go back and try to have selenium calculate the page routing, but i think i can live with launching a HTTP server by running npx http-server . -o -p 8889 -c-1 instead of just running firefox index.html.

further reading

this guide was obviously very biased and probably not actually a "proper" introduction, but if it drew your curiosity toward web archiving at all then i think it did it's job. i found a more in-depth article while researching that may offer more help if this guide wasn't quite what you needed. in addition, archivebox has an index of the web archiving community that you can use to find resources for your specific archival needs (or just browse for fun).

bonus

youtube videos

pretty much all of my needs for archiving youtube content have been taken care of by using yt-dlp. usually yt-dlp -t $FORMAT "$LINK" is good enough, but double check that the presets still work if it breaks. if you need something more specific, you can check out the documentation for all the information.