Different predictions if running in Node instead of Browser (using the same model_web - python converted model)

问题

pretty new to ML and tensorflow!

I made an object detection model with http://cloud.annotations.ai that permits to train and convert a model in different formats, tfjs (model_web) too. That website provides also boilerplates for running the model within a browser (react app)... just like you do - probably it is the same code, didn't spend enough time.

So I have this model running inside a browser, giving prediction about objects in a photo with pretty good results considering the amount of example I gave and the prediction score (0.89). the given bounding box is good too.

But, unfortunately, I didn't have "just one video" to analyze frame by frame inside a browser, I've got plenty of them. So I decided to switch to node.js, porting the code as is. Guess what? TF.js relies on DOM and browser components, and almost none examples that works with Node exists. So not a big deal, just spent a morning figuring out all the missing parts. Finally I'm able to run my model over videos that are splitted in frames, at a decent speed - although having the "Hello there, use tfjs-node to gain speed" banner when I'm already using tfjs-node - but results seems odd. Comparing the same picture with the same model_web folder gave the same prediction but with lower score (0.80 instead of 0.89) and a different bounding box, with object not centered at all.

(TL;DR)

Does tfjs have different implementation of the libraries (tfjs and tfjs-node) that makes different use of the same model? I don't think it can be a problem of input because - after a long search and fight - i figure out two ways to give the image to tf.browser.getPixel in Node (and I'm still wondering why I have to use a "browser" method inside tfjs-node). Anyone made comparisons?

So... that's the code I used, for your reference:

model_web is being loaded with tf.loadGraphModel("file://path/to/model_web/model.json");

two different ways to convert a JPG and make it works with tf.browser.getPixel()

const inkjet = require('inkjet');
const {createCanvas, loadImage} = require('canvas');

const decodeJPGInkjet = (file) => {
    return new Promise((rs, rj) => {
        fs.readFile(file).then((buffer) => {
            inkjet.decode(buffer, (err, decoded) => {
                if (err) {
                    rj(err);
                } else {
                    rs(decoded);
                }
            });
        });
    });
};

const decodeJPGCanvas = (file) => {
    return loadImage(file).then((image) => {
        const canvas = createCanvas(image.width, image.height);
        const ctx = canvas.getContext('2d');
        ctx.drawImage(image, 0, 0, image.width, image.height);
        const data = ctx.getImageData(0, 0, image.width, image.height);
        return {data: new Uint8Array(data.data), width: data.width, height: data.height};
    });
};

and that's the code that use the loaded model to give predictions - same code for node and browser, found at https://github.com/cloud-annotations/javascript-sdk/blob/master/src/index.js - doesn't works on node as it is, I changed require("@tensorflow/tfjs"); with require("@tensorflow/tfjs-node"); and replaced fetch with fs.read

const runObjectDetectionPrediction = async (graph, labels, input) => {
    const batched = tf.tidy(() => {
        const img = tf.browser.fromPixels(input);
        // Reshape to a single-element batch so we can pass it to executeAsync.
        return img.expandDims(0);
    });

    const height = batched.shape[1];
    const width = batched.shape[2];

    const result = await graph.executeAsync(batched);

    const scores = result[0].dataSync();
    const boxes = result[1].dataSync();

    // clean the webgl tensors
    batched.dispose();
    tf.dispose(result);

    const [maxScores, classes] = calculateMaxScores(
        scores,
        result[0].shape[1],
        result[0].shape[2]
    );

    const prevBackend = tf.getBackend();
    // run post process in cpu
    tf.setBackend("cpu");
    const indexTensor = tf.tidy(() => {
        const boxes2 = tf.tensor2d(boxes, [result[1].shape[1], result[1].shape[3]]);
        return tf.image.nonMaxSuppression(
            boxes2,
            maxScores,
            20, // maxNumBoxes
            0.5, // iou_threshold
            0.5 // score_threshold
        );
    });
    const indexes = indexTensor.dataSync();
    indexTensor.dispose();
    // restore previous backend
    tf.setBackend(prevBackend);

    return buildDetectedObjects(
        width,
        height,
        boxes,
        maxScores,
        indexes,
        classes,
        labels
    );
};

回答1:

Do different implementation of the libraries (tfjs and tfjs-node) that makes different use of the same model

If the same model is deployed both in the browser and in nodejs, the prediction will be the same thing.

If the predicted value are different, it might be related to the tensor used for the prediction. The processing from the image to the tensor might be different resulting in different tensors being used for the prediction thus causing the output to be different.

i figure out two ways to give the image to tf.browser.getPixel in Node (and I'm still wondering why I have to use a "browser" method inside tfjs-node)

The canvas package use the system graphic to create the browser like canvas environment that can be used by nodejs. This makes it possible to use tf.browser namespace especially when dealing with image conversion. However it is still possible to use directly nodejs buffer to create a tensor.

来源：https://stackoverflow.com/questions/60221268/different-predictions-if-running-in-node-instead-of-browser-using-the-same-mode

标签

javascript

node.js

tensorflow

tensorflow.js

tfjs-node