I enjoy driving, but I see too many people oblivious to the requirements of driving a pedal assisted ballistic weapon, so while I think it would be good for many people, it is not for me. Besides I ride my motorcycle to work most of the time (and I want to know how they will deal with lane splitting). I also prefer the rigid feel every bump raw drive that is my FJ40 over any vehicle I have ever driven.
I do have concerns about the technology as well. Some around the actual implementation and some around what "others" feel is the right way to drive. It takes a mix of defensive an offensive (in both meanings of the word) driving to prevent accidents and injury. I also would like to see an autonomous vehicle navigate turn outs on a windy mountain road (like HWY 9 or Bear Creek), where most people are not able to cope with those roads and cause backups because they have no clue what a turn out is used for. The autonomous car would try to do the speed limit, while the passengers who get car sick foul the interior.
I see some benefits of self driving cars too. When a light turns green the snake effect _could_ be eliminated since every car would be able to accelerate at the same time.
I also feel that there would be some "gaming" of the cars. If I knew that an autonomous car would move to the right if I came up fast behind it, and flashed my lights, I would do that to every one of them just for fun. It could be a great source of amusement to see what you could force one of those cars to do. I already see people messing with the Tesla autopilot modes.
I am also reminded of a science fiction story I read as a kid (cannot remember the name right now), where a group of travelers come across a futuristic multi lane freeway where all the cars are flying by at 100 mph with barely a space between them. After some experimentation of throwing rocks (or other debris), the travelers step into the flow and the cars move around them, allowing them to cross the freeway with no stoppage in traffic and no one getting hit. (if anyone knows the name of that book I would be in your debt)
I do see that the insurance companies and government would try to cost you out of your old style auto's and motorcycles. Not something I would be happy with.