This depends on whether you intend to manage the lifetime of objects polymorphically, using pointers to the interface class.
If you do, then the destructor must be virtual, in order to correctly delete the objects. Deleting a base-class pointer that doesn't have a virtual destructor is invalid, and gives undefined behaviour.
If you don't, then you should enforce this by making the destructor non-virtual and protected, so only derived classes can be deleted.