Tracking DB querying time - Bookshelf/knex

Question

I would like to monitor the time taken by a query on my API's db. I so created the following function, using bookshelf-signals, a Bookshelf plugin. :

bookshelf.on('fetching', () => {
  server.app.fetching = new Date().valueOf();
});

bookshelf.on('counting', () => {
  server.app.fetching = new Date().valueOf(); 
});

bookshelf.on('fetched', () => {
  server.statsd.gauge('db_query', new Date().valueOf() - server.app.fetching);
});

... so that I can retrieve the time just before and just after a fetch/count; I did the same with deleting-deleted and saving-saved.

What I think I fail to understand is when fetching and fetched are supposed to be triggered... When I tried to to see when fetching and fetched were triggered, basically it ended up with this :

'fetching event A'
'fetching event B'
'fetching event C'
'fetched event C'
'fetched event B'
'fetched event A'

Resulting in the timers returning wrong values obliviously, do you have any lead/clue ?

I also saw that one could trigger 'query' events on Knex, and thought of using this as an alternative solution. However, it seems that it only works if I specify the table where I query, ie :

 knex('whatever_table').on('query', () => {///});

Making it impracticable in the case where I want to apply an event handler on every model... I think I should stick with Bookshelf, but how can I do with the way the events are handled?

Thank you in advance!

Answer 1

I just wrote some small test code how to trace transaction duration with knex.

https://runkit.com/embed/679qu91ylu4w

/**
 * Calculate transaction durations in knex
 * 
 */
require('sqlite3');
var knex = require("knex")({
  client: 'sqlite', 
  connection: ':memory:', 
  pool: { min: 1, max: 10 }
});

function isTransactionStart(querySpec) {
  return querySpec.sql === 'BEGIN;';
}

function isTransactionEnd(querySpec) {
  return querySpec.sql === 'COMMIT;' || querySpec.sql === 'ROLLBACK;';
}

const transactionDurations = {};

knex.on('query', querySpec => {
  console.log('On query', querySpec);

  if (isTransactionStart(querySpec)) {
    if (transactionDurations[querySpec.__knexUid]) {
      console.error('New transaction started, before earlier was ended');
      return;
    }
    transactionDurations[querySpec.__knexUid] = new Date().getTime();
  }

  if (isTransactionEnd(querySpec)) {
    const startTime = transactionDurations[querySpec.__knexUid];
    if (!startTime) {
      console.error('Transaction end detected, but start time not found');
    }
    const endTime = new Date().getTime();
    transactionDurations[querySpec.__knexUid] = null;
    console.log('TRANSACTION DURATION', endTime - startTime);
  }
}); 

// just as an example of other available events to show when they are called
knex.on('query-response', (res, querySpec) => {
  // console.log('On query response', res, querySpec);
}); 

knex.on('query-error', (err, querySpec) => {
  // console.log('On query error', err, querySpec);
}); 

try {
    a = await Promise.all([
      knex.transaction(trx => {
        return trx.raw('select 1');
      }),
      knex.transaction(trx => {
        return trx.raw('select 2');
      }),
      knex.transaction(trx => {
        return trx.raw('error me');
      })
    ]);
} catch (e) {
  console.log('Got ERROR:', e);
}

The same king of approach should work also for query timing. To prevent timer bookkeeping from leaking memory you should add some cleanup code though.

Query duration timer should be started in query event and stopped in query-response or query-error depending which one triggers first.

To be able to match query - query-response pair querySpec.__knexQueryUid attribute can be used.

Answer 2

Based on Mikael Lepistö snippet I came up with this :

const dbEvents = (server, sdc) => {
  knex.on('query', data => {
    server.app[data.__knexQueryUid + ''] = new Date().valueOf();
  });

  knex.on('query-response', (data, obj, builder) => {
    sdc.counter('db_queries_time', new Date().valueOf() - server.app[obj.__knexQueryUid + '']);
    sdc.increment('nr_db_queries');
  });
};

And I then call the function when I start the server - I am working with Hapijs.

EDIT: sdc is a statsd client, I use it to send the DB time :)