Thursday, November 26, 2015

OSM(Openstreetmap) tile server installation on windows


                      Building Own Openstreetmap Tile server on windows.




History-

In last month i got a chance to work on the project which involve a real time assets tracking.
initially i decided to go with the google map API , but after some time i came to know that google is
charging for the API .and cost is about 8K USD/Year which is very big price in India.
so i search for some opensource map providers and here i found solution to my problem.

after going through the couple of wiki documents i decided to create a my own OSM tile server to use with leaflet. but here is the problem again for me most of the documentation i found for creating a own tile server is targeting the linux/ubuntu platform.

finally i found some good article on OSM server installation on windows.

1) https://github.com/mapserver/mapserver/wiki/RenderingOsmDataWindows
2)http://softwaresimian.com/2012/12/02/openstreetmap-osm-install-on-windows-part-i-the-database/
3)http://softwaresimian.com/2013/02/20/openstreetmap-osm-install-on-windows-part-ii-generate-tiles/
4)http://www.zachatrocity.com/blog/creating-a-mapnik-tileserver-on-windows/


Refering all this article together finally i got a success to build my own tile server for Open street map.

while working on this i came to know that creating the tile server on windows platform is very hectic work and u will not find much documentation targeting the windows platform.
 so i decided to  write this article for my developer community :)

Creating OSM tile server involve following steps.

1)-Data base creation
2)-Exporting data from Opnestreetmap (U can also use the other providers )
3)Importing Data into database
4)Generating tiles from database




Required software
1)    7-zip
  • used to extract the compressed .osm files (although it is not necessary to extract them)
  • supports .bz2 compression
  • you should enable all formats through Tools/Options
  • Go the Start Menu > 7-Zip > 7-Zip File Manager
  • Within the 7-Zip File Manager choose Tools > Options
  • Under the system tab choose 'Select all'
  • Press OK: now any bz2 files should automatically open in 7-Zip

2) Cygwin-
  • cygwin is used to Import the osm data into postgres database.
  • Note-if u find the downloadable of the Osm2PgSql then no need to use Cygwin,use Osm2Pgsql instead. as for this date no binaries of Osm2Pgsql  are available for windows only because of this reason i am using the Cygwin(cygwin package also contain osm2pgsql exe in it) to import the data. refer this for the issue
         
  • download and save the installer (hint: don't delete this file after, as it can be used to modify your install)
  • run the installer
  • keep defaults
  • for the "Choose a Download Site" select any mirror
  • make sure you add the following packages:
  • wget (in the "Web" category)
  • unzip (in the "Archive" category)
  • gcc (in the "Devel" category, named 'gcc-g++: C++ compiler)
  • make (in the "Devel" category)
  • python (in the "Python" category)
  • click "Next"
  • click "Finish"

3) NotePad++ - An  editor.
4) PostGres SQL

Download PostgreSQL


Install PostgreSQL


  • Click "Next" through the first few Setup wizard steps. *On the "Password" screen, type "postgres". Note: PostgreSQL has its own users and superusers, which are separate and distinct from Windows users and administrators. As a default, PostgreSQL installation sets up both a PostgreSQL superuser named "postgres" and a Windows system user account named "postgres" that will use the same password.
  • Click "Next" through the rest of the Setup wizard steps, leaving all values as their defaults (such as the Port).
  • The last step of the Setup wizard should ask if you want to launch something called "Stack Builder."
  • Make sure that the checkbox for "Stack Builder" is checked.
  • Click "Finish", which both finishes PostgreSQL installation and launches the Stack Builder.
  • On the first page of the "Stack Builder" window, in the drop down, select "PostgreSQL on port 5432"
  • Click "Next"
  • On the next page, click on the "+" sign next to "Spatial Extensions"
  • Check the box next to "PostGIS 2.2 for PostgreSQL 9.3 v2.0.0
  • Click "Next" until presented with the "PostGIS 2.2.0" setup window.

Install PostGIS

  • Start the installer through the Stack Builder or the standalone installer.
  • Click "I Agree"
  • Make sure "Create Database" is not selected
  • Click "Next" until you get to a "Database Connection" page
  • Enter "postgres" for the password and click "Next"
  • Click "Yes" to the "Would you like us to register the GDAL_DATA environment variable (see Appendix C), click "Yes"
  • Click "Close" on the "Installation Complete" window.
  • Click "Finish" in the Stack Builder window

Verify PostgreSQL and PostGIS Installation Success

  • In your Start Menu, goto PostgreSQL 9.3/pgAdmin III/
  • This should bring up an administrative window, with an "Object Browser" on the left side of the window.
  • In the Object browser, double click on the icon with the red "X" labeled "PostgreSQL (localhost:5432)"
  • Enter your password, "postgres"
  • The red "X" should disappear and a set of items should be listed below the "PostgresSQL" label.
  • if all is OK then congrats you successfully install Postgres
for PostGIS you need to enable the extension on the database we are going to this in next step.
Note -in Postgres 9.3 u will not get the template PostGis DB so don't confused with the steps mention here (This is is for PostGres 9.1)



Creating the OSM Database

  • Make sure the 'C:/Program Files/PostgreSQL/xx/bin/' directory is added to your PATH environment variable

  • to test this, open a new command prompt window, and type: psql
  • if not found, then you must add the above directory to your PATH environment variable (the following steps are for Windows7, but might be slightly different on other versions)
  • Start > Control Panel > System
  • Click Advanced System Settings
  • On the "Advanced" Tab, at the bottom of the tab, click the "Environment Variables" buttons.
  • In the "System variables" section - NOT the "User variables" section - select the "Path" variable
  • Click Edit
  • Put your cursor in the "Variable Value" input area
  • Move the cursor to the far right and add a ; if the line does not already end with one
  • Then add the custom path to a directory needed like "C:\Program Files\PostgreSQL\9.3\bin"
  • Click "OK"
  • Click "OK" again
  • Now Open the PGAdmin3 utility from your start button and create a database name "osm"
  • Now open a new command window and navigate to postgis path and try the psql command
  • These instructions assume that you already have a PostgreSQL 'postgres' user
  • Open a command prompt and enter the following commands ( the password will always be: postgres )
  •  psql -U postgres -d osm -f "C:\Program Files\PostgreSQL\9.2\share\contrib\postgis-2.2\postgis.sql"
  • psql -U postgres -d osm -f "C:\Program Files\PostgreSQL\9.3\share\contrib\postgis-2.2\spatial_ref_sys.sql"
  • psql -U postgres -d osm -f "C:\Program Files\PostgreSQL\9.3\share\contrib\postgis-2.2\legacy.sql"
  • We must also apply a patch. Download the patch file to your local machine: legacy-postgis-gist.sql (you might have to "File/Save Page As" in your browser). Then execute in the directory that you downloaded the file to:
  •          psql -U postgres -d osm -f "legacy-postgis-gist.sql"
  • try connecting to the 'osm' database:
  •          psql -U postgres -d osm
  • then type: \d
  • a list of tables should be displayed, such as:
  •                    List of relations
  •      Schema |       Name        | Type  |  Owner
  •     --------+-------------------+-------+----------
  •      public | geography_columns | view  | postgres
  •      public | geometry_columns  | view  | postgres
  •      public | spatial_ref_sys   | table | postgres
  • note: to exit that database connection, type: \q
Thats it your database is ready now to import the OSM data

Downloading the Data

Download the data from Opnestreetmap by clicking on export.
Note-for testing purpose try to download the small chunk of data because importing the data into database is a lengthy work based on the size of the data :)


Importing Data into Postgres using Osm2pgsql utility(here i am using this from cygwin for windows)

1)Download Osm2pgSql from here or u can also use the cygwin to  import the data (prefer to use Osm2pgSql)


  • Unpack the file to somewhere convenient (In my case it is C:\Osm2PgSql)
  • Add the directory where you unpacked the file to your PATH
  • Run the command osm2pgsql -h to verify the install [Should get a help screen]
  • Replace the file named default.style with a more up to date version from here
  • We now have the capability to process map data and a database to store it in. 
  • Now we must import the OSM data into our 'osm' table, by using the osm2pgsql commandline utility.
  • Downlaod the proj\nad from here the ms4w package containd the proj\nad folder in it.

  • First we must make sure to use the correct EPSG file, so open a command window and type:
  •          SET PROJ_LIB=C:\proj\nad
  • Execute the following in that same command window, and execute the following command ( use the password: postgres:
  • osm2pgsql -U postgres -W -m -d osm -p osm -E 3857 -S "C:\osm2pgsql\default.style(This is the path to your default.style)"  C:\data\india.pbf (This is your data file )
  • If you got the error allocating ways(nodes) please using parameter -s (slim mode)
  • Verify that the tables were created, by either using pgAdmin III (StartMenu/Programs/PostgreSQL 9.3/pgAdmin III/) or connect to the database through the commandline and display the table names with the command:
  •          psql -U postgres -d osm -c \d
  • which returns a list of the new tables, such as:
  •  Schema |             Name              | Type  |
  • --------+-------------------------------+-------+
  •  public | geography_columns             | view  |
  •  public | geometry_columns              | view  |
  •  public | osm_line                      | table |
  •  public | osm_point                     | table |
  •  public | osm_polygon                   | table |
  •  public | spatial_ref_sys               | table |
  • Note-Data importing will take a time may be a day based on the size of the data you are importing
         If all is OK then Congrats you completed one important step in building the tile server :)


Generating the tiles

       Setting up Enviorment
  • Next you will need to install MAPNIK, which is a cross platform toolkit for desktop and server based map rendering.  The Windows version is a bit behind but is still perfectly serviceable for our needs.

  • Download the software from here. Unpack it and add a path to its \lib directory. [The version was  mapnik-2.2.0 when I downloaded]

  • set PATH=%PATH%;c:\mapnik-2.2.0\lib(path of your mapnik folder)

  • Now test to see if it works:

  • cd c:\mapnik-2.2.0\demo\c++

  • rundemo ..\..\lib\mapnik

  • Once you are happy with this, you need to install the Python bindings from here. Once installed, add to your path and generate a required environment variable.

  • Add Python to your path [ C:\Python27 ]

  • Make a new environment variable PYTHONPATH pointing to c:\mapnik-2.0.0\python\2.7\site-packages;

  • Check python / mapnik are OK with:

  • cd c:\mapnik-2.0.0\demo\python

  • python rundemo.py
     Tile rendering
  • Now we can get started on tile rendering.  You will need subversion for this.  Get the latest mapnik rendering tools using the following command:

  • svn export http://svn.openstreetmap.org/applications/rendering/mapnik

  • To render tiles from the database, you will also need some other information files for coastlines etc.  Download these and unpack into c:\mapnik\world_boundaries

  • http://tile.openstreetmap.org/world_boundaries-spherical.tgz

  • http://tile.openstreetmap.org/processed_p.tar.bz2

  • http://tile.openstreetmap.org/shoreline_300.tar.bz2

  • http://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-populated-places/

  • http://www.naturalearthdata.com/downloads/110m-cultural-vectors/110m-admin-0-boundary-lines/

  • Now perform a quick check to make sure everything is installed correctly.  Go to the C:\mapnik directory and execute the following command.

  • generate_xml.py -h

  • Now need to generate the osm.xml file which controls tile generation:  Type:

  • generate_xml.py –host localhost –user postgres –dbname osm –symbols symbols/ –world_boundaries world_boundaries/ –port 5432 –password postgres
  • You are now in a position to generate tiles. Run the  following to test image generation:

  • set MAPNIK_MAP_FILE=osm.xml

  • set MAPNIK_TILE_DIR=C:\tiles

  • set HOME=C:\Users

  • generate_image.py

  • If ok, generate tiles from the loaded database [can take a very very long time … we are talking days] using the following command. Important Note that u need to modify the bound entry in generate_tiles.py  as per your requirement [e.g. bbox = (…)] to control which areas you render tiles for and to what zoom level. render_tiles(…)

  • generate_tiles.py
  • Congrats are having your own tile server now :)
  • you can easily use this with the leaflet.js
  • you just need to change the tile server url while using leaflet.js
  • example-
  • L.tileLayer('http://your domain name here/{z}/{x}/{y}.png', {
  •     attribution: '&copy; <a href="http://osm.org/copyright">OpenStreetMap</a> contributors'
  • }).addTo(map);
  • feel free to add your comments and suggestions :)

32 comments:

  1. Thanks for your tutorial. Although you haven't talk much about the tile server (mod_tile, renderd or tirex) setup. You have managed to create tile via the python script but how to server them via apache web server?
    Thanks

    ReplyDelete
    Replies
    1. http://www.zachatrocity.com/blog/creating-a-mapnik-tileserver-on-windows/

      Tile Server
      Creating a tile server from where we are now is a cake walk. It just depends on what you want to use for your server. I would recommend using Node.js with something like node-tileserver. If you are forced to use something like IIS its really just as simple as routing all requests to C:\RhinoTile\osm\tiles\zoom\x\y.png or where ever you put your tiles.

      Delete

    2. https://youtu.be/wVnimcQsuwk

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Thanks a lot. Really helpful. I have gone through the whole process what you have explained but can you please explain the last step which is use leaflet.js to localhost with detaile explaination. I have the same requirement.I want to create and host same map as OSM. Also, i have started from small example so if i want to add new osm file from what step i should start?

    ReplyDelete
    Replies
    1. http://www.zachatrocity.com/blog/creating-a-mapnik-tileserver-on-windows/

      Tile Server
      Creating a tile server from where we are now is a cake walk. It just depends on what you want to use for your server. I would recommend using Node.js with something like node-tileserver. If you are forced to use something like IIS its really just as simple as routing all requests to C:\RhinoTile\osm\tiles\zoom\x\y.png or where ever you put your tiles.

      https://youtu.be/wVnimcQsuwk

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Getting there...

    At osm2pgsql -U postgres -W -m -d osm -p osm -E 3857 -S "C:\osm2pgsql\default.style C:\data\india.pbf

    I get osm2pgsql: illegal option --W

    Do you know how to get around this?

    The osm2pgsql says :-

    release notes: 'Windows version built by Dominik Perpeet (http://customdebug.com/osm2pgsql/index.html)

    But the link does not work!

    Any ideas?

    Thanks

    Andy.

    ReplyDelete
  6. I am getting :-

    Reading in file: C:\Users\Andy\Downloads\osm.pbf
    Processing: Node(85490k 131.5k/s) Way(0k 0.00k/s) Relation(0 0.00/s)COPY_END for
    COPY osm_nodes FROM STDIN;
    failed: ERROR: duplicate key value violates unique constraint "osm_nodes_pkey"

    DETAIL: Key (id)=(196350) already exists.
    CONTEXT: COPY osm_nodes, line 83241512

    Any idea why?

    Thanks

    Andy.

    ReplyDelete
    Replies
    1. Hi! I am having the same error. Could anyone help with this please ?

      Delete
  7. What an awesome and very nice post. I just stumbled upon your weblog and wanted to say that I’ve really enjoyed browsing your blog posts.

    In any case I will be subscribing to your rss feed and I hope you write again very soon!

    pave tile store

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Hi. I am having the same error as aw912345 2 comment above. Can anyone help with this please ?

    ReplyDelete
  11. Why did you remove all the comments???

    ReplyDelete
  12. Piotr - did you get anything to work?

    ReplyDelete
  13. Hello Sir Hope u r doing well i follwed your blloged i am facig this issue if help me it would be great for me i m sending the error.... command getting all thing bt at last showing error please go through and let me know what is solution

    C:\mapnik-v2.2.0\demo\c++>rundemo.exe C:\mapnik-v2.2.0\lib\mapnik
    running demo ...
    looking for 'shape.input' plugin in... C:\mapnik-v2.2.0\lib\mapnik/lib/mapnik/input/
    looking for DejaVuSans font in... C:\mapnik-v2.2.0\lib\mapnik/lib/mapnik/fonts/DejaVuSans.ttf
    ### std::exception: Could not create datasource for type: 'shape'

    ReplyDelete
  14. I have given up with this as it does not work.

    The easiest way is to do a Linux VM and follow the instructions from here :-

    https://github.com/AcuGIS/opentileserver

    It does work very well and they even have an easy Nominatim install as well :-

    https://github.com/AcuGIS/OpenNameSearch

    Andy.


    ReplyDelete
  15. I found a few issues that were quite frustrating.

    the linked osm2pgsql kept failing on me, with the error pointed out above. I found another version, which eventually worked, but the parameters had to be different. My server has only 4GB of memory.

    I downloaded the osm2pgsql from

    https://ci.appveyor.com/api/projects/openstreetmap/osm2pgsql/artifacts/osm2pgsql_Release.zip?tag=0.90.1

    and ran it with this...
    osm2pgsql -U postgres -W -m -d osm -p osm --cache 900 -s -c -E 3857 -S "D:\osm-data\x64\default.style" ..\british-isles-latest.osm.pbf

    (Change your paths and databases as necessary).

    I then ran into a problem with mapnik generate_xml.py. The instructions were not that clear. It kept complaining that the reference planet_osm_polygon was not found. To fix this, the mapnik required an extra parameter of prefix. (Oh, I also found that you needed double hyphen... -- for all of the options).

    My generate_xml is as follows...
    generate_xml.py osm.xml output_osm.xml --host localhost
    --user postgres --dbname osm --symbols symbols/ --world_boundaries world_boundar
    ies/ --port 5432 --prefix osm --password put_your_password_here

    Took me a while to get this far, but I hope this helps someone.

    David.

    ReplyDelete
  16. I liked the content on this site. Would like to visit again.

    Tile Installations Ontario

    ReplyDelete
  17. Totally incredible posting! Heaps of valuable data and motivation, both of which we all need!Relay value your work. the low cpc ads

    ReplyDelete
  18. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts.เครื่อง เซิฟเวอร์

    ReplyDelete
  19. SO Beautiful post, from beginning to end of post
    TileMode is your one-stop-shop for Kitchen, bathroom floor and wall tiles - with a huge range of colors, sizes, styles and finishes, View our large selection of glass, stone, marble, metal and other discount priced tile

    ReplyDelete
  20. Sometime few blogs become very helpful while getting relevant and new information related to your targeted area. As I found this blog and appreciate the information delivered to my database.

    beds
    bed
    mattress
    mattresses

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete
  22. eToro is the most recommended forex trading platform for rookie and professional traders.

    ReplyDelete
  23. Hi, in my case demos didn't worked, but I figured out how to fix it.

    to run c++ demo i had to type rundemo ../..
    to run python demo i had to reinstall python to 32 bit version, which is needed by Mapnik

    now everything works correctly, hope it helps for someone :)

    ReplyDelete
  24. Even though this page is a little dated and required a lot of effort to get operational, I can attribute my success to this page. Thank you!

    ReplyDelete