Have to agree with opinions of odah and Blackhawker. The risks may be slight, but they are still very real risks. I definitely wouldn't touch off any type of birdshot round inside of the buildings you describe as the small shot will bounce back off of almost anything - wood, nails, old tools, etc, etc., posing a great danger to your eyes especially.
Remember, one of the cardinal rules of gun safety is "Be sure of your target and what's beyond". Obviously inside of a building you have no idea what's beyond the four walls you can see. Even though it may be "out in the middle of nowhere" you just never know when a hiker, berry picker, hunter, etc may just happen to wander by.
It sounds like fun, but I wouldn't do it myself.