项目简介
该应用允许用户下载任何网站的完整源码,包括JavaScript、CSS、图片等所有资源。它利用wget和archiver镜像并压缩网站,通过socket通道传回。支持一键部署到多种云平台。
README 预览
## Complete Website Downloader 💾\nDownload the complete source code of any website (including all assets) 🔨.\n\n👉 Live Demo: https://website-downloader.onrender.com\n\n\n\n\n \n\n\n\n## Description 📒\n Website downloader works with `wget` and `archiver` to download all websites assets and compress then sends it back to the user through socket channel\n \n **wget params the being used**\n \n `wget --mirror --convert-links --adjust-extension --page-requisites \n--no-parent http://example.org`\n\n **Explanation of the various flags:**\n\n - --mirror – Makes (among other things) the download recursive.\n- --convert-links – convert all the links (also to stuff like CSS stylesheets) to relative, so it will be suitable for offline viewing.\n- --adjust-extension – Adds suitable extensions to filenames (html or css) depending on their content-type.\n- --page-requisites – Download things like CSS style-sheets and images required to properly display the page offline.\n- --no-parent – When recursing do not ascend to the parent directory. It useful for restricting the download to only a portion of the site\n### Deploy on cloud providers\n[](https://replit.com/github/AhmadIbrahiim/Website-downloader)\n[](https://glitch.com/edit/#!/import/github/AhmadIbrahiim/Website-downloader)\n[](https://railway.app/new/template?template=https://github.com/AhmadIbrahiim/Website-downloader)\n[](https://app.cyclic.sh/api/app/deploy/AhmadIbrahiim/Website-downloader)\n[](https://app.koyeb.com/deploy?type=git&repository=github.com/AhmadIbrahiim/Website-downloader&branch=main&name=Website-downloader)\n[](https://render.com/deploy?repo=https://github.com/AhmadIbrahiim/Website-downloader)\n\n\n## How to run it 🤔\n\n- `git clone https://github.com/AhmadIbrahiim/Website-downloader.git`\n- `cd Website-downloader`\n- `$ npm install`\n- `$ npm start`\n- `http://localhost:3000/`\n\n\n\n# How To Contribute:\n - Open Issue(s) with any bugs you notice.\n - Please create Pull Requests if you think it would
常见问题 (2)
故障排除在本地运行网站下载功能时,如何解决"'wget' is not recognized as an internal or external command"错误?
故障排除
在本地运行网站下载功能时,如何解决"'wget' is not recognized as an internal or external command"错误?下载功能需要 GNU Wget,但它在您的系统 PATH 中缺失或不可用。在 Windows 上,可以使用 winget install JernejSimoncic.Wget 或 choco install wget 安装 Wget。安装后请重启终端或 IDE,以便加载更新后的 PATH,然后运行 wget.exe --version 验证安装。之后本地下载功能应可正常工作。
故障排除如何修复 Website-downloader 生成的巨大已删除 .zip 文件在用户关闭页面时不释放磁盘空间的问题?
故障排除
如何修复 Website-downloader 生成的巨大已删除 .zip 文件在用户关闭页面时不释放磁盘空间的问题?磁盘空间未被释放,因为 Node.js 进程仍然持有指向已删除 .zip 文件的打开文件描述符。找出卡住的进程:
sudo lsof +L1 | grep '\.zip (deleted)'
输出示例如下:
node 270102 root 20w REG ... /www/wwwroot/Website-downloader/public/sites/.zip (deleted)
在暂停下载或网站服务之后,停止该卡住的进程:
sudo kill -9 270102
要终止所有持有已删除 ZIP 文件的 Node 进程:
sudo lsof +L1 | awk '/node/ && /\.zip \(deleted\)/ {print $2}' | xargs -r sudo kill -9
验证空间已释放:
df -h
要彻底修复该问题,请将 Website-downloader 更新到包含修复 issue #74 的 pull request 的版本。该补丁必须在客户端断开连接时中止或销毁下载流、删除临时 ZIP 文件,并阻止 URL 抓取重试循环继续执行。更新后重启 Node 服务,例如:
pm2 restart all
# 或者
sudo systemctl restart <your-website-downloader-service>