permethrin or pyrethrin are your best allies here. If you're all about convenience, grab an aerosol like Ortho Home Defense ...

These treatments typically contain pyrethrin or pyrethroid ... If there is an active wasp nest, a simple solution of two tablespoons of dish soap added to a spray bottle of water can be used ...