unexpected behavior using zip-stream NPM on Google k8s

问题

I am working on creating a zip of multiple files on the server and stream it to the client while creating. Initially, I was using ArchiverJs It was working fine if I was appending buffer to it but it fails when I need to add streams into it. Then after having some discussion on Github, I switched to Node zip-stream which started working fine thanks to jntesteves. But as I deploy the code on GKE k8s I Started getting Network Failed errors for huge files.

Here is my sample code :

const ZipStream = require("zip-stream");

/**
 * @summary Adding readable stream provided by https module into zipStreamer using entry method
 */
const handleEntryCB = ({ readableStream, zipStreamer, fileName, resolve }) => {
  readableStream.on("error", () => {
    console.error("Error while listening readableStream : ", error);
    resolve("done");
  });
  zipStreamer.entry(readableStream, { name: fileName }, error => {
    if (!error) {
      resolve("done");
    } else {
      console.error("Error while listening zipStream readableStream : ", error);
      resolve("done");
    }
  });
};

/**
 * @summary Handling downloading of files using native https, http and request modules
 */
const handleUrl = ({ elem, zipStreamer }) => {
  return new Promise((resolve, reject) => {
    let fileName = elem.fileName;
    const url = elem.url;
    //Used in most of the cases
    if (url.startsWith("https")) {
      https.get(url, readableStream => {
        handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
      });
    } else if (url.startsWith("http")) {
      http.get(url, readableStream => {
        handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
      });
    } else {
      const readableStream = request(url);
      handleEntryCB({ readableStream, zipStreamer, url, fileName, resolve, reject });
    }
  });
};

const downloadZipFile = async (data, resp) => {
  let { urls = [] } = data || {};
  if (!urls.length) {
    throw new Error("URLs are mandatory.");
  }
  //Output zip name
  const outputFileName = `Test items.zip`;
  console.log("Downloading using streams.");
  //Initialize zip-stream instance
  const zipStreamer = new ZipStream();
  //Set headers to response
  resp.writeHead(200, {
    "Content-Type": "application/zip",
    "Content-Disposition": `attachment; filename="${outputFileName}"`,
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Methods": "GET, POST, OPTIONS"
  });
  //piping zipStreamer to the resp so that client starts getting response
  //as soon as first chunk is added to the zipStreamer
  zipStreamer.pipe(resp);
  for (const elem of urls) {
    await handleUrl({ elem, zipStreamer });
  }
  zipStreamer.finish();
};

app.post(restPrefix + "/downloadFIle", (req, resp) => {
  try {
    const { data } = req.body || {};
    downloadZipFile(data, resp);
  } catch (error) {
    console.error("[FileBundler] unknown error : ", error);
    if (resp.headersSent) {
      resp.end("Unknown error while archiving.");
    } else {
      resp.status(500).end("Unknown error while archiving.");
    }
  }
});

I tested for 7-8 files of ~4.5 GB each on local, it works fine and when I tried the same on google k8s, I got network failed error. After some more research, I Increased server timeout on k8s t0 3000 seconds, than it starts working fine, but I guess the increasing timeout is not good. Is there anything I am missing on code level or can you suggest some good GKE deployment configuration for a server that can download large files with many concurrent users? I am stuck on this for the past 1.5+ months. please help!

Edit 1: I edited the timeout in the ingress i.e Network services-> Load Balancing ->edit the timeout in the service

来源：https://stackoverflow.com/questions/60464596/unexpected-behavior-using-zip-stream-npm-on-google-k8s