Design a polymorphic class hierarchy without virtual destructor

Question

I find myself in a situation that calls for rapid instantiation and deletion of many objects, which ends up taking a lot of CPU and user time, so I need to shave off as much as possible from that process. I do realize a non-virtual destructor will not be a significant gain, but it will still save a few percent of the time, and together with a few other optimizations will stack into a tangible improvement from a user's perspective.

I know that the rule of thumb is that in such polymorphic hierarchies it is recommended practice to begin with making the destructor virtual, so that when objects are deleted the appropriate code is ran to do all the necessary clean-up.

But what if it didn't matter which destructor will be called? What if all the subclass specific dynamic data that needs collecting is packed and instantiated in the specific constructor to a pointer dedicated for that purpose in the "very-base-class"? In this case, every object can instantiate and access it specific plus inherited dynamic data through the pointer which is "collected" by the base class destructor which can be inlined, so regardless of its actually content and size that dynamic memory will be de-allocated by the OS, which only needs the address and does not care about size, so no memory leaks, no overhead from virtual calls.

So, does this sound like a good idea? Any potential pitfalls?

EDIT: A simple test:

class A {
  public:
  ~A() {cout << "A" << endl; }
};

class B : public A {
  public:
  ~B() {cout << "B" << endl; }
};

int main()
{
   A * a = new A;
   A * b = new B;
   B * br = new B;

   delete a;
   delete b;
   delete br;

   return 0;
}

output:

A
A
B
A

So regardless of what pointer is deleted, the base class destructor which collects all memory is called no matter what. Is there possibility for anything else to happen? If so when and why?

Answer 1

A very big pitfall, yes - undefined behavior. The following code exhibits it:

struct A
{};
struct B : A
{};

//....
A* a = new B;
delete a;

Deleting a base class pointer that points to a derived class leads to UB if the base class doesn't have a virtual destructor. If you plan on doing this, you need the virtual destructor.