Commit graph

13 commits

Author SHA1 Message Date
stefanodvx
0d986d4573 fix: resolve memory leaks in download utility
1. writing chunks directly to disk instead of buffering in memory
2. using fixed-size buffers (32KB) for all I/O operations
3. optimizing buffer allocation strategy in downloadInMemory
4. implementing proper file synchronization with mutex locks
5. calculating chunk boundaries on-the-fly instead of pre-allocating

the memory profiling showed excessive allocations in bytes.growSlice
which has been addressed by minimizing intermediate buffers and
eliminating unnecessary memory copies

these changes should fix the observed OOM issues when downloading
large files while maintaining the same functionality
2025-04-28 14:35:22 +02:00
stefanodvx
37c2fbf215 twitter: update endpoint
this endpoint is not protected at the moment, but would probably in a few days
2025-04-27 20:59:35 +02:00
stefanodvx
a959807524 twitter: temporary fix
we're aiming to automatically generate x-client-transaction-id header. due to a recent change on twitter backend, this header is now required. sadly, its not static, so would propably break again anytime soon
2025-04-26 19:50:30 +02:00
stefanodvx
5336968e05 code cleanup 2025-04-24 12:20:35 +02:00
stefanodvx
e1f424b948 cleanup code 2025-04-20 13:03:53 +02:00
stefanodvx
12ad427baf fixes config 2025-04-20 12:54:20 +02:00
stefanodvx
0a63df9ce6 set configuration for each extractor 2025-04-20 12:19:14 +02:00
dogghi no
e35bd2fe57
implement sonic for json + remove some TODOs from readme + fix go version requirement (#5) 2025-04-17 14:16:30 +02:00
stefanodvx
12c12e53f7 fixes some lint errors 2025-04-16 17:03:16 +02:00
stefanodvx
3e351e7e43 use host matching instead of regex (desc)
why? using regex on every single message the bot receives, even simple patterns, can be very harmful for your cpu lol
2025-04-16 14:11:55 +02:00
stefanodvx
58bd5827b3 main: changes, fixes
- refactors http session usage across modules
- adjusts video info extraction
- improves temporary directory, concurrency limits, buffering, and error handling in download routines
2025-04-14 23:45:54 +02:00
stefanodvx
10c113f400 env: supports for http and socks5 proxies 2025-04-14 21:50:49 +02:00
stefanodvx
3faede7b1c Init 2025-04-14 13:05:43 +02:00