I have never had any problems downtown or at the Jacksonville Landing. I lived in Orange Park for 17 years and always felt safe going downtown.
You can go to St. Augustine there is alot to do there and is only about 40 minutes. There is the old town, fort, gator farm, beach, ripleys beleive it or not, boat ride in the matanzas bay. There is alot to do in the area of Jacksonville. The Jacksonville Zoo. If you have children they usually have stuff going on throughout the year (Rivercity kids day). Metropolitan Park on the water usually has good free concerts.I miss living there.