Proxying file upload with node/express.js and node request results in corrupt files with almost dobble the file size

问题

I'm trying to proxy our main server via a node.js/express.js server. I've got everything working as it should except file uploads. Files are uploaded and reach the server, however somewhere between the data being passed over to node.js request and the data reaching the server the file is corrupted.

A file with content-length 1833 ends up having a content-length 3274 once it reaches the server. I expect the file that reaches the server to stay at 1833. Running without the proxy works as expected.

This is the proxy POST endpoint

router.post("*", wrapAsync(async (req: any, res: any) => {
    const path = ROOT_PATH + req.originalUrl;
    logger.info("Proxy request to: POST " + path);
    const headers = JSON.parse(JSON.stringify(req.headers)); // create a copy of headers
    delete headers["content-length"]; // Without this node.js requests will set the content-length to null
    if (req.session.auth) {
        headers.Authorization = req.session.auth.Token;
    }
    const options = {
        agentOptions,
        body: req.rawBody,
        encoding: null as any,
        headers,
        json: false,
        resolveWithFullResponse: true,
        url: path
    };
    const result = await request.post(options);
    handleJSONResponse(result, res);
}));

The value for req.rawBody are created in the following little middleware:

app.use("/rest/api", (req: any, res, next) => {
    // This is a special piece of proxy middleware that provides the raw buffer
    req.rawBody = "";
    req.setEncoding("utf8");
    req.on("data", (chunk: any) => {
        req.rawBody += chunk;
    });
    req.on("end", () => {
        next();
    });
}, proxyRouter);

It is either something in the middleware or something in node.js request (either a bug or something in how I've configured it) that seems to be causing the issue. It is possible for me to handle the files in a separate endpoint so long as it gets POST:ed to the same endpoint in the server (ie, it does not have to be directly proxied, I have control what URLS the frontend uses)

回答1:

The issue was in the middleware

(req: any, res, next) => {
    // This is a special piece of proxy middleware that provides the raw buffer
    req.rawBody = "";
    req.setEncoding("utf8");
    req.on("data", (chunk: any) => {
        req.rawBody += chunk;
    });
    req.on("end", () => {
        next();
    });
}

Solved it by using bodyParser.raw() instead, ie:

app.use("/rest/api", bodyParsers.raw(), proxyRouter);

来源：https://stackoverflow.com/questions/60006391/proxying-file-upload-with-node-express-js-and-node-request-results-in-corrupt-fi

标签

node.js

express

proxy

request