Conduitry/fetch-booru
1
Fork 0
Code Releases Activity

move fetch options to own directory; only send special user-agent to e621 API

Browse Source
This commit is contained in:
Conduitry 2024-03-24 17:05:30 -04:00
parent b8f5929b07
commit ed3c8bcc00
5 changed files with 13 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -1,4 +1 @@
/node_modules/
/*.json
!/package.json
!/package-lock.json

18
README.md
View File

@ -1,23 +1,5 @@
# fetch-booru
## `[hostname].json` fetch options
Each request will look for a corresponding `[hostname].json` and, if present, will use it as the second argument in all `fetch` calls.
For example, if you have a `cf_clearance` cookie for the Cloudflare protection in front of a given site, you can create
```json
// example.com.json
{
"headers": {
"cookie": "cf_clearance=whatever",
"user-agent": "whatever"
}
}
```
and this cookie and user-agent will be used for API calls and for downloading the images.
## License
[MIT](LICENSE)

11
fetch_options/README.md Normal file
View File

@ -0,0 +1,11 @@
# `fetch_options/[hostname].json`
Each request will look for a corresponding `fetch_options/[hostname].json` and, if present, will use it as the second argument in all `fetch` calls.
For example, if you have a `cf_clearance` cookie for the Cloudflare protection in front of a given site, you can create `fetch_options/example.com.json`
```json
{ "headers": { "cookie": "cf_clearance=whatever", "user-agent": "whatever" } }
```
and this cookie and user-agent will be used for network calls to `example.com`. Note that these will not automatically apply to subdomains.

1
fetch_options/e621.net.json Normal file
View File

@ -0,0 +1 @@
{ "headers": { "user-agent": "fetch-booru (+https://git.chor.date/Conduitry/fetch-booru)" } }

2
lib/fetch.js
View File

@ -4,7 +4,7 @@ import { log } from './log.js';
const fetch_options_lookup = {};
export async function fetch_retry(url) {
const { hostname } = new URL(url);
const fetch_options = await (fetch_options_lookup[hostname] ??= fs.promises.readFile(new URL(`../${hostname}.json`, import.meta.url), 'utf8').then(JSON.parse).catch(() => ({ headers: { 'user-agent': 'fetch-booru (+https://git.chor.date/Conduitry/fetch-booru)' } })));
const fetch_options = await (fetch_options_lookup[hostname] ??= fs.promises.readFile(new URL(`../fetch_options/${hostname}.json`, import.meta.url), 'utf8').then(JSON.parse).catch(() => {}));
for (;;) {
let resp;
try {