Autonomous vehicle driving in urban environments is a challenging task that requires localization accuracy exceeding that available from GPS-based inertial guidance systems. For map-based driving, a 3D laser scanner can be utilized to localize the vehicle within a previously recorded 3D map. Such scanners are however not feasible for mass production due to cost considerations. In this paper we present a localization algorithm that creates an off-line predefined map and then localizes with respect to this map. First, the map is constructed by a service vehicle equipped with a calibrated stereo camera rig and a high precision navigation system. Then, the global localization ego-pose can be obtained in any vehicle equipped with a standard GPS and a single forward looking camera for extracting and matching features to relevant map candidates. We use a recently proposed estimation method called SOREPP (Soft Optimization method for Robust Estimation based on Pose Priors) that utilizes relevant priors for achieving high performance, fast and reliable estimation, even with a small fraction of inliers. During the estimation it uses all the matched correspondences without need for random sampling to find the inliers. This method eventually obtains an outlier-free set of landmarks, used to estimate the ego-pose with high accuracy. We evaluate our algorithm on real world data comprised of a challenging 4.5km drive. Our algorithm achieves accurate localization results: a mean lateral absolute error of 14.35cm and a mean longitudinal absolute error of 18.63cm.