I've had this problem before. The big problem has to do with the fact that the wood may be "humidified" and the adhesive on the Velcro may be weak.
Go out, but some more Velcro, re-apply but leave the lid open. Oh, and use rubber bands to apply enough pressure to both the Velcro and the hygrometer so that the adhesive gets itself into the grain of the wood.
You might want to let this sit for a couple of days, or even a week while the glue sets up.
And, once applied, I HIGHLY recommend you never remove you hygrometer unless you absolutely must.