AngularJS remove # and have error when refresh page

Reloading the page or manually type URL like "localhost/mysite.com/contact" gives wrong "Not Found"(404 error) on the server. This is because MVC does not know about angular routing and by MVC pattern it will look for an MVC page for 'contact' which does not exist in MVC routing path. I think there is a solution by .haccess file by redirect all links to the index page, but it, not a good idea.

I already add a baseURL constant to the application in head. and the js part is below:

 app.config(function($routeProvider,$locationProvider) {
  $routeProvider.
  when("/contact", {
    templateUrl : "partial/contact.html"
   }).
   when("/aboutus", {
    templateUrl : "aboutus.html"
   }).
    when("/service", {
    templateUrl : "service.html"
   }).
   otherwise({
    redrictTo: "index.html"
    })
    $locationProvider.html5Mode(true);
});

How can I fix the issue angular to get the correct page (localhost/mysite.com/contact) when I go to the contact page by typing URL manually.