Switching noise is naturally the first place to look, when an IRQ is firing rapidly and unexpectedly. But have you verified that your IRQ handler is completely handling each interrupt event? And that another interrupt event while handling the prior one will not lead to unusual behavior?
It could very well be a rare, spurious interior firing due to noise, but then exacerbated by an IRA handler that doesn't clear properly, leading to high speed sprioois events.
What are the approximate sizes for the internal and external pull-up resistors you've attached? And what is the impedance for the actual interrupt source, when it actually fires?