Which points to triangulate for bundle adjustment after Visual Odometry

Question

I am gradually building upto bundle adjustment from a monocular visual odometry pipeline. Using 2D-2D feature correspondences (eg. ORB features), a simple VO pipeline has been w