profile
viewpoint

Ask questions[Feature request] Save/Cache API responses

As I am sure the developers have also noticed, APIs have quotas - and developing 99.99% requires you to retry/resend the same API requests (getting the same responses) which unnecessarily uses up your quota.

In a production sense, it is also feasible that poor quality or high quality disparate input datasets will produce reoccurring domains and making an API request for the same data in short intervals is wasteful.

With these 3 uses, the data caching is actually a very critical missing feature.

There are many viable options to chose from;

  1. Manage a cache of raw responses and wrap the HTTP library/adaptor to source these raw responses from the cache location depending on the file timestamp and a configuration TTL for the cached file.
  2. Rely on HTTP headers like Last-Modified, E-Tag, 304 response code, from a HEAD method request (quota is usually only consumed on other http verbs).
  3. Checksum files, store the hash in the db file, use this similarly to option 1.

Hope this helps.

OWASP/Amass

Answer questions chrisdlangton

@caffix this is wonderful news - thank you! I reviewed the diff and am not 100% sure but i don't think there was a way to specify where responses are save to (file system)?

It seems cache is pure graph database and responses are private to the app and users still cannot access their data from third party integrations..

useful!

Related questions

redeclared in this block hot 2
Something broke amass for me hot 1
Failed to open the data operations output file: open amass_output/amass_data.json: too many open files hot 1
Run time SIGSEGV running multiple domains hot 1
The enumeration was unable to build the pool of resolvers - Amass hot 1
issue with alterations.txt hot 1
The enumeration was unable to build the pool of resolvers hot 1
The enumeration was unable to build the pool of resolvers hot 1
source:https://uonfu.com/
Github User Rank List