429 Too Many Requests - Angular 7 - on multiple file upload

问题

I have this problem when I try to upload more than a few hundred of files at the same time.

The API interface is for one file only so I have to call the service sending each file. Right now I have this:

onFilePaymentSelect(event): void {
    if (event.target.files.length > 0) {
      this.paymentFiles = event.target.files[0];
    }
    let i = 0;
    let save = 0;
    const numFiles = event.target.files.length;
    let procesed = 0;
    if (event.target.files.length > 0) {
      while (event.target.files[i]) {
      const formData = new FormData();
      formData.append('file', event.target.files[i]);
      this.payrollsService.sendFilesPaymentName(formData).subscribe(
      (response) => {
        let added = null;
        procesed++;
        if (response.status_message === 'File saved') {
          added = true;
          save++;
        } else {
          added = false;
        }
        this.payList.push({ filename, message, added });
      });
    i++;
  }
}

So really I have a while for sending each file to the API but I get the message "429 too many request" on a high number of files. Any way I can improve this?

回答1:

Working with observables will make that task easier to reason about (rather than using imperative programming).

A browser usually allows you to make 6 request in parallel and will queue the others. But we don't want the browser to manage that queue for us (or if we're running in a node environment we wouldn't have that for ex).

What do we want: We want to upload a lot of files. They should be queued and uploaded as efficiently as possible by running 5 requests in parallel at all time. (so we keep 1 free for other requests in our app).

In order to demo that, let's build some mocks first:

function randomInteger(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const mockPayrollsService = {
  sendFilesPaymentName: (file: File) => {
    return of(file).pipe(
      // simulate a 500ms to 1.5s network latency from the server
      delay(randomInteger(500, 1500))
    );
  }
};

// array containing 50 files which are mocked
const files: File[] = Array.from({ length: 50 })
  .fill(null)
  .map(() => new File([], ""));

I think the code above is self explanatory. We are generating mocks so we can see how the core of the code will actually run without having access to your application for real.

Now, the main part:

const NUMBER_OF_PARALLEL_CALLS = 5;

const onFilePaymentSelect = (files: File[]) => {
  const uploadQueue$ = from(files).pipe(
    map(file => mockPayrollsService.sendFilesPaymentName(file)),
    mergeAll(NUMBER_OF_PARALLEL_CALLS)
  );

  uploadQueue$
    .pipe(
      scan(nbUploadedFiles => nbUploadedFiles + 1, 0),
      tap(nbUploadedFiles =>
        console.log(`${nbUploadedFiles}/${files.length} file(s) uploaded`)
      ),
      tap({ complete: () => console.log("All files have been uploaded") })
    )
    .subscribe();
};

onFilePaymentSelect(files);

• We use from to send the files one by one into an observable
• using map, we prepare our request for 1 file (but as we don't subscribe to it and the observable is cold, the request is just prepared, not triggered!)
• we now use mergeMap to run a pool of calls. Thanks to the fact that mergeMap takes the concurrency as an argument, we can say "please run a maximum of 5 calls at the same time"
• we then use scan for display purpose only (to count the number of files that have been uploaded successfully)

Here's a live demo: https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

Open up the console to see that we're not uploading all them at once

来源：https://stackoverflow.com/questions/60097110/429-too-many-requests-angular-7-on-multiple-file-upload