Node can't find certain modules after synchronous install

Question

I\'ve got a script that synchronously installs non-built-in modules at startup that looks like this

const cp = require(\'child_process\')

function requireOr         


        

Answer 1

I think your best option is either:

• (ugly) to install package globally, instead of locally
• (best solution ?) to define YOUR new 'package repository installation', when installing, AND when requiring

First, you may consider using the npm-programmatic package.

Then, you may define your repository path with something like:

const PATH='/tmp/myNodeModuleRepository';

Then, replace your installation instruction with something like:

const npm = require('npm-programmatic');
npm.install(`${module}`, {
        cwd: PATH,
        save:true
}

Eventually, replace your failback require instruction, with something like:

return require(module, { paths: [ PATH ] });

If it is still not working, you may update the require.cache variable, for instance to invalide a module, you can do something like:

delete require.cache[process.cwd() + 'node_modules/bluebird/js/release/bluebird.js'];

You may need to update it manually, to add information about your new module, before loading it.

Answer 2

It seems that the require operation after an npm install needs a certain delay. Also the problem is worse in windows, it will always fail if the module needs to be npm installed. It's like at a specific event snapshot is already known what modules can be required and what cannot. Probably that's why require.cache was mentioned in the comments. Nevertheless I suggest you to check the 2 following solutions.

1) Use a delay

const cp = require("child_process");

const requireOrInstall = async module => {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    // Use one of the two awaits below
    // The first one waits 1000 milliseconds
    // The other waits until the next event cycle
    // Both work
    await new Promise(resolve => setTimeout(() => resolve(), 1000));
    await new Promise(resolve => setImmediate(() => resolve()));
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
  }
}

const main = async() => {
  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = await requireOrInstall("find-free-port");
  const express = await requireOrInstall("express");
  const socket = await requireOrInstall("socket.io");
}

main();

await always needs a promise to work with, but it's not needed to explicitly create one as await will wrap whatever it is waiting for in a promise if it isn't handed one.

2) Use a cluster

const cp = require("child_process");

function requireOrInstall(module) {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
    process.exit(1007);
  }
}

const cluster = require("cluster");

if (cluster.isMaster) {
  cluster.fork();
  cluster.on("exit", (worker, code, signal) => {
    if (code === 1007) {
      cluster.fork();
    }
  });
} else if (cluster.isWorker) {
  // The real work here for the worker

  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = requireOrInstall("find-free-port");
  const express = requireOrInstall("express");
  const socket = requireOrInstall("socket.io");

  process.exit(0);
}

The idea here is to re-run the process in case of a missing module. This way we fully reproduce a manual npm install so as you guess it works! Also it seems more synchronous rather the first option, but a bit more complex.

Answer 3

cp.execSync is an async call so try check if the module is installed in it's call back function. I have tried it, installation is clean now:

const cp = require('child_process')

function requireOrInstall (module) {
    try {
        require.resolve(module)
    } catch (e) {
        console.log(`Could not resolve "${module}"\nInstalling`)
        cp.execSync(`npm install ${module}`, () => {
            console.log(`"${module}" has been installed`)
            try {
                return require(module)
            } catch (e) {
                console.log(require.cache)
                console.log(e)
            }
        })

    }
    console.log(`Requiring "${module}"`)

}

const http    = require('http')
const path    = require('path')
const fs      = require('fs')
const ffp     = requireOrInstall('find-free-port')
const express = requireOrInstall('express')
const socket  = requireOrInstall('socket.io')

When node_modules not available yet :

When node_modules available already: