How can I call many URLs from a list asynchronously

Question

I have a few hundred thousand URLs that I need to call. These are calls to an application server which will process them and write a status code to a table. I do not need to

Answer 1

I agree with the top post to use Runspaces. However the provided code doesn't show how to get data back from the request. Here's a PowerShell module recently published to my GitHub page:

https://github.com/phbits/AsyncHttps.

It will submit async HTTP requests to a single domain over SSL/TLS (TCP port 443). Here's an Example from the README.md

Import-Module AsyncHttps
Invoke-AsyncHttps -DnsName www.contoso.com -UriPaths $('dir1','dir2','dir3')

It returns a System.Object[] containing the results of each request. The result properties are as follows:

Uri       - Request Uri
Status    - Http Status Code or Exception Message
BeginTime - Job Start Time
EndTime   - Job End Time

After looking at your example, you'll probably need to make the following modifications:

1. Allow usage of an alternative port (webserver:8080). The easiest would be to update the URI in the scriptblock. Alternatively add another parameter to the module and scriptblock just for the port.
2. Test that Query Parameters are properly formatted and not mangled by percent encoding when used in the HTTP request. Consider skipping the use of UriBuilder in the scriptblock as long as your list of Uri Paths are known to be OK.