fetch-booru

Conduitry/fetch-booru

Fork 0

Commit graph

0fd2fe6b40 rating:[sqe] queries batch-multi-tag Conduitry 2024-05-22 15:51:55 -0400
62c08ab34f WIP Conduitry 2024-04-25 19:22:05 -0400
02ea5c7ee3 fix rounding errors in elapsed time in logs master Conduitry 2024-04-29 17:25:21 -0400
b6c82eb4f5 batch-e621: just use tags array in JSON Conduitry 2024-04-25 18:08:20 -0400
4f7f2ca7fe add time since startup to logs Conduitry 2024-04-04 20:00:32 -0400
af0ed03467 batch-*: make tag aliasing code a little clearer Conduitry 2024-03-29 09:18:13 -0400
42c1e57eb7 improve path behavior on Windows Conduitry 2024-03-27 18:29:29 -0400
6c76a14448 limit to 32 concurrent file downloads Conduitry 2024-03-27 17:47:50 -0400
21f56991ac batch-*: do not count downloads made by a concurrent instance Conduitry 2024-03-25 08:05:48 -0400
ad42d2a820 batch-e621: allow searching for unknown/invalid tags Conduitry 2024-03-24 21:05:40 -0400
ed3c8bcc00 move fetch options to own directory; only send special user-agent to e621 API Conduitry 2024-03-24 17:05:30 -0400
b8f5929b07 dl-rule34: use api.rule34.xxx Conduitry 2024-03-24 16:30:58 -0400
55ad4391a7 move binaries into bin directory Conduitry 2024-03-24 16:14:30 -0400
bff80e3fd3 retry downloads on error while streaming to file Conduitry 2024-03-24 14:07:28 -0400
c3eb4512cf batch-allthefallen: improve multiple tag handling Conduitry 2024-03-24 09:31:20 -0400
e0f8446d1f batch-allthefallen/dl-allthefallen: add initial version Conduitry 2024-03-24 02:29:13 -0400
42530b7b20 dl-*: ignore files with no URL Conduitry 2024-03-24 02:27:45 -0400
6a1464574d batch-*: default to decoding URL-escaped characters in directory names Conduitry 2024-03-24 02:27:23 -0400
06791a1baf rename project to fetch-booru Conduitry 2024-03-23 19:17:41 -0400
c6e1f31cec batch-e621/dl-e621: factor out handling API responses Conduitry 2024-03-23 19:04:46 -0400
462d118367 batch-e621/aug-e621: adjust db dump processing Conduitry 2024-03-23 17:21:42 -0400
d554498a15 return true from done() if there were no errors Conduitry 2024-03-23 15:21:46 -0400
24d4f7cac7 dl-*: avoid trying to create directory for every file Conduitry 2024-03-23 14:29:38 -0400
c4faeb6a8a retry fetch on network errors Conduitry 2024-03-23 11:25:46 -0400
eee96cb7ca batch-e621/aug-e621: determine current dump by datetime rather than querying server Conduitry 2024-03-23 05:43:35 -0400
847daf7315 extract mkdir into dl.js Conduitry 2024-03-21 22:20:29 -0400
f460f62a3a dl-inkbunny: add initial version Conduitry 2024-03-21 22:10:13 -0400
c3e223c596 aug-e621: rework Conduitry 2024-03-21 22:08:34 -0400
1f59badc73 remove reference to USE_WGET environment variable Conduitry 2024-03-21 22:08:04 -0400
20931c715a batch-inkbunny: rework etc Conduitry 2024-03-21 22:07:35 -0400
4d987f95c8 dl-rule34/dl-tbib: rework Conduitry 2024-03-21 22:06:32 -0400
8140cd32bc dl-e621: rework and extract additional shared e621 code Conduitry 2024-03-21 22:05:23 -0400
62a4d0500c batch-inkbunny: add initial version Conduitry 2024-03-17 10:51:49 -0400
6125f52a3e batch-e621: port over custom fetch options feature Conduitry 2024-03-16 23:19:04 -0400
e9b7741781 batch-e621: refactor Conduitry 2024-03-16 21:58:32 -0400
81c067025e batch-e621: improve performance of handling CSV column names Conduitry 2024-03-16 18:20:28 -0400
99934c3a8c batch-e621: only download db_export index page once Conduitry 2024-03-16 11:29:37 -0400
64a149930a batch-e621: warn about tags the database doesn't know about Conduitry 2024-03-16 10:54:44 -0400
747813a85b avoid converting web streams to Node streams Conduitry 2024-03-15 18:50:09 -0400
9f2ed5577d batch-e621: avoid downloading same file multiple times in parallel Conduitry 2024-03-15 12:30:47 -0400
bd9cf8f283 batch-e621: avoid converting Node stream to web stream Conduitry 2024-03-15 08:48:07 -0400
dd22118430 batch-e621: prevent copy-to-self error when post is in both API response and CSV Conduitry 2024-03-14 20:01:05 -0400
833ac2adee batch-e621: download/copy files in parallel Conduitry 2024-03-14 07:07:34 -0400
9f161235d8 batch-e621: add initial version Conduitry 2024-03-13 18:06:07 -0400
af92d46fdd remove dl-furrybooru Conduitry 2024-03-13 13:26:09 -0400
b39947face allow directory names to be URI encoded tag names Conduitry 2023-12-27 06:59:04 -0500
393edeba2c retry on 429 responses Conduitry 2023-09-28 22:18:50 -0400
8657712a20 add missing package-lock.json exception to .gitignore Conduitry 2023-09-06 11:15:54 -0400
81327664a2 do not readdir when there are no posts Conduitry 2023-09-05 18:11:02 -0400
f225084c94 update to package-lock.json version 3 Conduitry 2023-09-05 15:39:47 -0400
4273af2ba6 readdir target directory rather than checking whether each file exists Conduitry 2023-09-04 07:20:05 -0400
7a3c394d46 add USE_WGET workaround Conduitry 2023-09-04 06:52:26 -0400
3ae54e22fe adjust naming of temporary files Conduitry 2023-02-17 08:48:41 -0500
ad373fa4e6 add dl-tbib Conduitry 2023-02-09 08:57:31 -0500
0f1ff32170 support md5 and id queries in boorus Conduitry 2023-02-08 23:53:27 -0500
1fca797207 retry requests returning 5xx after 5 seconds Conduitry 2023-01-24 17:44:34 -0500
92fb64675a avoid 3/10,000,000 event Conduitry 2022-12-11 07:08:47 -0500
6de122ed6b ignore malformed [hostname].json file Conduitry 2022-12-10 23:53:48 -0500
506e2f0e2f move fetch options to [hostname].json Conduitry 2022-12-10 15:10:03 -0500
b3aafd4cc4 remove ExperimentalWarning suppression Conduitry 2022-11-16 22:12:48 -0500
5e5b2fca96 simplify output for non-TTY Conduitry 2022-06-26 20:37:02 -0400
988b49b9ae add dl-rule34 Conduitry 2022-06-19 22:39:34 -0400
a1cfa60e51 factor out shared booru code Conduitry 2022-06-19 22:39:10 -0400
95983cbffe rework directory/query handling in aug-* Conduitry 2022-06-16 20:14:02 -0400
dd05ab91b3 dl-e621: support downloading posts by ID or MD5 Conduitry 2022-06-16 20:11:38 -0400
d472b8e360 rework directory/query handling in dl-* Conduitry 2022-06-16 20:11:15 -0400
37b0663e5f stream responses to files rather than buffering them Conduitry 2022-06-15 09:01:40 -0400
49e14fd008 add initial version of aug-e621 Conduitry 2022-06-15 03:55:13 -0400
92566da2d3 rename _dl.js to _shared.js Conduitry 2022-06-15 03:51:22 -0400
eca076a091 export get function rather than passing it Conduitry 2022-06-14 21:07:45 -0400
51b4a3aa45 factor out naming directory after query Conduitry 2022-06-14 20:58:32 -0400
b2977c2ecf refactor fetch options loading Conduitry 2022-06-14 13:19:44 -0400
16d48852f1 refactor: look up fetch options per request Conduitry 2022-06-13 19:08:48 -0400
6a278ec527 mention dl-*.fetch-options.json in readme Conduitry 2022-06-13 08:12:07 -0400
635ad1b3b6 tidy for loops Conduitry 2022-06-11 08:23:08 -0400
de2febf6d9 use MD5 hash as filename in dl-furrybooru Conduitry 2022-06-11 07:26:49 -0400
67973ed59a tidy Conduitry 2022-06-08 08:34:42 -0400
eafe5a35f7 rework loadiing of fetch options Conduitry 2022-06-07 19:45:27 -0400
fa407c99aa tidy progress output Conduitry 2022-06-05 23:49:17 -0400
865255894c use regex capturing groups in dl-furrybooru Conduitry 2022-06-05 23:11:48 -0400
75bfadaf74 refactor Conduitry 2022-06-05 20:27:46 -0400
40bc42c170 support headers file in dl-e621 as well Conduitry 2022-06-05 16:17:44 -0400
79ada02977 add initial version of dl-furrybooru Conduitry 2022-06-05 16:05:58 -0400
cbd47412f0 ignore dl-*.headers.js files Conduitry 2022-06-05 15:36:54 -0400
a07573b421 support passing options into _shared.js to use for all fetches Conduitry 2022-06-05 15:36:25 -0400
20120154df improve console output in dl-e621 Conduitry 2022-06-05 06:04:38 -0400
0c110b5498 DRY fetch calls in _shared.js Conduitry 2022-06-04 10:08:28 -0400
c012e496ff suppress ExperimentalWarning about fetch Conduitry 2022-06-04 09:58:53 -0400
e4253463a4 move more filesystem handling into _shared.js Conduitry 2022-06-04 06:48:20 -0400
88b945ad54 add initial version of dl-e621 Conduitry 2022-06-03 23:03:17 -0400
df77951791 add ESLint config Conduitry 2022-06-03 22:34:54 -0400
67a1a6b2a8 initial commit Conduitry 2022-06-03 22:33:21 -0400