Crawling engines indexing web api results

Question

All,

A crawling engine will extract links from websites and index them.

Given the following:

• A web api that receives an identifier and returns fil