Would you be mad if your neighbor put up a privacy fence?

You should build the fence. All of your reasons are valid, even the ones that are halfway unspoken (the neighbor's grandchildren). You may even start a trend. I guarantee your other neighbors have thought about building a fence but haven't because no one else has 'broken the ice.' One way to alleviate any paranoia about what the neighbors might think is to have a little party. Once the fence is built, have a little backyard hamburger and weenie roast and invite the neighbors. Let them have a good look at it. Tell them all about the fence. I'm willing to bet you'll see a few more fences pop up in short order.