Website-downloader
ApplicationAhmadIbrahiim/Website-downloader
Download complete website source code including all assets using Node.js.
Overview
This application allows users to download the complete source code of any website, including JavaScript, CSS, images, and other assets. It uses wget and archiver to mirror and compress the site, then sends it back via a socket channel. The tool supports one-click deployment on various cloud platforms.
README Preview
## Complete Website Downloader 💾\nDownload the complete source code of any website (including all assets) 🔨.\n\n👉 Live Demo: https://website-downloader.onrender.com\n\n\n\n\n \n\n\n\n## Description 📒\n Website downloader works with `wget` and `archiver` to download all websites assets and compress then sends it back to the user through socket channel\n \n **wget params the being used**\n \n `wget --mirror --convert-links --adjust-extension --page-requisites \n--no-parent http://example.org`\n\n **Explanation of the various flags:**\n\n - --mirror – Makes (among other things) the download recursive.\n- --convert-links – convert all the links (also to stuff like CSS stylesheets) to relative, so it will be suitable for offline viewing.\n- --adjust-extension – Adds suitable extensions to filenames (html or css) depending on their content-type.\n- --page-requisites – Download things like CSS style-sheets and images required to properly display the page offline.\n- --no-parent – When recursing do not ascend to the parent directory. It useful for restricting the download to only a portion of the site\n### Deploy on cloud providers\n[](https://replit.com/github/AhmadIbrahiim/Website-downloader)\n[](https://glitch.com/edit/#!/import/github/AhmadIbrahiim/Website-downloader)\n[](https://railway.app/new/template?template=https://github.com/AhmadIbrahiim/Website-downloader)\n[](https://app.cyclic.sh/api/app/deploy/AhmadIbrahiim/Website-downloader)\n[](https://app.koyeb.com/deploy?type=git&repository=github.com/AhmadIbrahiim/Website-downloader&branch=main&name=Website-downloader)\n[](https://render.com/deploy?repo=https://github.com/AhmadIbrahiim/Website-downloader)\n\n\n## How to run it 🤔\n\n- `git clone https://github.com/AhmadIbrahiim/Website-downloader.git`\n- `cd Website-downloader`\n- `$ npm install`\n- `$ npm start`\n- `http://localhost:3000/`\n\n\n\n# How To Contribute:\n - Open Issue(s) with any bugs you notice.\n - Please create Pull Requests if you think it would
FAQ (2)
TroubleshootingHow to fix "'wget' is not recognized as an internal or external command" when running the website download feature locally?
The download feature requires GNU Wget, but it is missing or not available on your system PATH. On Windows, install Wget with either winget install JernejSimoncic.Wget or choco install wget. Restart your terminal or IDE so the updated PATH is loaded, then verify the installation by running wget.exe --version. The local download feature should work afterward.
TroubleshootingHow do I fix Website-downloader creating huge deleted .zip files that do not free disk space when a user closes the page?
The disk space is not released because the Node.js process still holds an open file descriptor to the deleted .zip file. Identify the stuck process:
sudo lsof +L1 | grep '\.zip (deleted)'This shows output such as:
node 270102 root 20w REG ... /www/wwwroot/Website-downloader/public/sites/.zip (deleted)Stop the stuck process after pausing downloads or the website service:
sudo kill -9 270102To terminate all Node processes holding deleted ZIP files:
sudo lsof +L1 | awk '/node/ && /\.zip \(deleted\)/ {print $2}' | xargs -r sudo kill -9Verify that the space is released:
df -hFor the permanent fix, update Website-downloader to the release containing the pull request that fixes issue #74. The patch must abort or destroy the download stream when the client disconnects, remove the temporary ZIP file, and prevent the URL-fetch retry loop from continuing. Restart the Node service after updating, for example:
pm2 restart all
# or
sudo systemctl restart <your-website-downloader-service>