nodeJS inserting Data into PostgreSQL error

Question

I have a weird error using NodeJS with a PostgreSQL and I hope you can maybe help me out.

I have a huge amount of data sets, about 2 Million entries that I want to inser

Answer 1

I'm not sure, but it looks like you got wrong data structure at the last element(309249) and PostgreSQL cannot parse some property

Answer 2

I'm the author of pg-promise. Your whole approach should be changed to the one below.

Proper way to do massive inserts via pg-promise:

const pgp = require('pg-promise')({
    capSQL: true
});

const db = pgp(/*connection details*/);

var cs = new pgp.helpers.ColumnSet([
    'id',
    {name: 'points', cast: 'double precision[]'},
    {name: 'mid', cast: 'double precision[]'},
    {name: 'occurences', cast: 'json[]'}
], {table: 'currentcitynet'});

function getNextInsertBatch(index) {
    // retrieves the next data batch, according to the index, and returns it
    // as an array of objects. A normal batch size: 1000 - 10,000 objects,
    // depending on the size of the objects.
    //
    // returns null when there is no more data left.
}

db.tx('massive-insert', t => {
    return t.sequence(index => {
        const data = getNextInsertBatch(index);
        if (data) {
            const inserts = pgp.helpers.insert(data, cs);
            return t.none(inserts);
        }
    });
})
    .then(data => {
        console.log('Total batches:', data.total, ', Duration:', data.duration);
    })
    .catch(error => {
        console.log(error);
    });

UPDATE

And if getNextInsertBatch can only get the data asynchronously, then return a promise from it, and update the sequence->source callback accordingly:

return t.sequence(index => {
    return getNextInsertBatch(index)
        .then(data => {
            if (data) {
                const inserts = pgp.helpers.insert(data, cs);
                return t.none(inserts);
            }
        });
});

Related Links:

• tx
• sequence / spex.sequence
• ColumnSet
• Multi-row insert with pg-promise