I.T. Progress

Date: 03/04/2010 – recorded on paper records

I.T. have moved WMS server into DMZ and released Apps Server. Go live date for beta testing of end delivery app has been delayed slightly due to late release of these items.

Meeting: I.T. Apps Server Delivery

Date: 22/03/2010 – recorded on paper records

Today we met with I.T. to discuss the delivery of our new apps server and the disk storage for the WMS server.

The main points which came up are the need for the WMS to have speedy disk access and good network access. The Apps Server is also ready to be released. This will affect my API as the apps server will host it!

The minutes for the meeting are:

GIS Framework
Project Management Office
Held on Monday 22 March 2010

Present:
Brian Hole (BH)
Steve Stone (SS)
Tracey Brant (TB)
John Collinson (JC)
Linda Huxley (LH) Minutes

Apologies:
Andy Elliott

A meeting took place to discuss the proposal and questions raised from e-mail entitled GIS Framework Proposal 17/03/2010.
Q1.  Storage of Data?
6TB for 2010/ 2011 and increasing to 10TB for 2011.
6TB on Data Base Server, some live data, some cache.
(TB) will forward a copy of the latest network diagram and plan to (SS) who will then be able to identify the requirements.
Q2.  How much Data?
2TB for WMS – figures will be provided by (TB/JC).
(JC) 30 map views per second, 15K rpm discs is recommended, but not enough space (IO level required).
(AE) Needs to record details of all system changes that are made on a regular basis, this will enable a set of standard changes AE will be able to authorise.  (TB) will forward details around changes.
Q3.   Back-up of Data?
Oracle product will be used for the back-up of data, one copy no historic data.  (SS) There is currently a 30 day policy
(TB) Data is required indefinitely.  80% is static, and 20% of data is live.
Recovery time and Business Continuity are areas for consideration.  (BH)  It is necessary to find a balance between risk and cost.
(SS)  What percentage is legislative and what isn’t?
WMS how many Cache files can be held?
(TB) There is some very historic data (maps from the 1800’s).  (SS) advised that if 100% important to be accessible archiving maybe the solution.
Q4.  How often does the historical data need to be backed up?
Q5.  What is the change rate of live data?
Q6.  How often does the live data need to be backed-up?
Q7.  How often if the data accessed?
Questions 4 to 7 have been discussed above.
(SS) requires further information (TB) and (JC) to provide in more detail :-
Which data, which is legislative/desirable/?
How long is data required?
How often is the data to be backed-up?
Are multiple back-ups required?
To keep back-up requirements to a minimum.
Q8.  Approximately how many concurrent users will be accessing the data at one time?
This is difficult to answer as the public also use.  Eg (DFU)
Q9.  Performance requirements of storage disks?
(JC) Load testing is planned.  3 Second average for public (DFU) is thought desirable/reasonable.
There is a lot of work to do first around data synchronisation.
(JC) raised Apps Service, (BH)/(SS) believe both are ready.  (BH) will find out.
(JC) If possible WMS DMZ live for 1st April.

Open Layers Code Working!

Date: 17/03/2010 – recorded on paper records

We have now got the Open Layers code working correctly. All the data coming out of my API is translated into the correct projectiong (Google Web Marcator) for the Open Layers framework to interpret and put on the map. Hurrah!

I.T. Release Database Server

Date: 10/03/2010 – recorded on paper records

Today I.T. have released a new server to house our Oracle Database. This server will provide faster calculations for spatial queries (ie addresses within a parish) and will give speedier access to the datafiles. This will in turn make my API even faster!

Open Layers Issue…

Date: 10/03/2010 – recorded on paper records

The application built on the server side API will also plug into a client side API which involves using Open Layers, another (yes another!) API which deals with the map window and locations.

There have been serious issues implementing this and getting our data talking to the Open Layers framework. There have also been big issues using projections. Most of our data is stored in a British projection, aligned to the OS grid however to support worldwide mapping we must use a global projection which is the Google Web Mercator.

Routing Speed Testing

Date: 10/03/2010 – recorded on paper records

Tested the APIs connections to the routing API RWNetServer. These tests have proved really successful with a successful UK long route created as an XML dataset in just a few seconds. My API is taking an origin and destination, passing that to the routing API and translating the response into a GIFramework dataset which is then output as XML.

Database Speed Testing

Date: 04/03/2010 – recorded on paper records

Today I have been testing the server side API’s database connections. This has included testing the speed and ensuring all types of data can be accessed, the geo referencing is correct and the relevant metadata is recorded on a data request.

Metadata includes datatypes, database column types, column sizes, column precision, key columns and constraints.

Speed testing proved successful, being tried out on our largest datasets such as UK wide address gazeteers and Ordnance Surveys vector based Mastermap.

WMS and The Tile Server

Date: 03/03/2010 – recorded on paper records

The issue of the tile server has been niggling at us for a while and we have now found a resolution. We will use an open source WMS application called GeoServer, which has an inbuilt caching system called GeoWebCache.

Geoserver will plug into all our data sources and create map images on request from my API, returning a link to the image created. GeoWebCache will cache all requests to the WMS software, thus create a tile server type system. This situation is a perfect solution as only the areas needed on a regular basis will be cached thus resolving the space issues we had in the tile server scenario.

The time issue is also solved as the cache is built up over time.

We are concerned about the speed of first request but hope to cache all widely used datasets before we go live.

I.T. Release FME Server

Finally our I.T. department have released a server to use to install FME Server. This will help my development as i can now create FME functionality and access it through the server from my code.

FME Training Course Attended

Date: 20/01/2010 – recorded on paper records

I have been attending a training course in FME workbench which will directly feed into FME server. FME will plug into the server side API and allow me to read virtually any spatial file format.

The course has been going well and I have learnt a lot already. There have been some interesting discoveries such as how to create FME workflows which will transform data, which i can then access via HTTP GET and POST calls.