Make GpsGate Server run faster!

As your GpsGate Server installation grows and the server load increases, you really want to know what is going on there inside GpsGate. What is taking the resources, and how can it be optimized. This blog gives you an introduction to profiling and tuning your GpsGate Server installation.


To follow the guide lines in this blog you need to run GpsGate Server v2.3.1 build 2076 or later. And it must have been downloaded and installed after June 10. 2010, or you installed the latest core patch June 10. 2010 or later. The patch enables "Live Profiling" which this blog depends on, and it also increases the server performance by 2 to 5 times. If you have a busy server it is worth installing the patch, and to read this. If you have a hosted server we will keep track of your server and you do not really need to bother, but if you indeed want to see profiling data, contact us, and we will enable it for you.


There are basically two main things that consumes resources in GpsGate Server, first a GPS tracker that sends a message containing position and status signals, and the processing of this message. We call the processing of one message a "Message Roundtrip". We will show how you count and measure all the Message Roundtrips on your server.


The other things that consumes resources are requests to the web part. Ajax requests from the web interface, and SOAP requests that may come in from third party integrations. We call those "Web Roundtrips". Profiling of those are not covered here.


After going though how to profile your "Message Roundtrips" we will discuss how to prevent the server from being congested when there are small periods of peak load. Imagine a hi-way where all cars are running smoothly, and all of a sudden there is a smaller disturbance and the traffic comes to a stand still. This can also happen on a server with high load. We will discuss methods to avoid this.

Enable profiling of NMEA Service


The profiler is very fast, and it can be run on a production server. In fact it is meant to be run on a production server, to allow you to understand and tune your production server. It is recommended that you always have profiling on if you have a server with lots of traffic.


To enable profiling in NMEA Service create a file named "profiler.config" and place it in the "Franson NMEA Service" folder where you installed GpsGate Server. This folder is typically located under "C:\InetPub\wwwroot\GpsGateServer\Franson NMEA Service".


This is what "profiler.config" should contain:

<?xml version="1.0" encoding="utf-8" ?>
<Profiler>
    <Channels>
        <Channel>
            <Name>Message</Name>
            <Period kind="Interval" interval="300" />
        </Channel>
    </Channels>
    <Targets>
        <Target channels="Message" assembly="Franson" type="Franson.Profiling.NLogTarget">
            <Template sort="COUNT">
                <Header>[CHANNEL] Time: [CHANNEL_TIME] Count: [CHANNEL_COUNT] CPM: [CHANNEL_COUNT_PER_MINUTE] Usage: [CHANNEL_USAGE]</Header>
                <Label>Time: [TIME] Count:[COUNT] Avg: [AVG_TIME] Max: [MAX_TIME] [LABEL]</Label>
                <Footer></Footer>
            </Template>
        </Target>
    </Targets>
</Profiler>

"interval" is set to 300, which means that profiling data will be generated every 300 seconds, every 5 minutes that is. When testing 300 is a good value, in every day usage 3600 (every hour) is a better value.


In this sample NLog is used to print out the result. To get the profiling data into a separate logfile add this to "NLog.config" in the "Franson NMEA Service" folder.


Inside the targets tag add the following tag:


  <target name="profiling" xsi:type="File" fileName="C:\ProfilingLog\NMEA_${shortdate}.log" layout="${longdate} ${message}"/>


And inside the rules tag add:


  <logger name="profiling.*" minlevel="Info" writeTo="profiling"/>


Now, when this is done, restart NMEA Service. When NMEA Service starts you should get the message "Profiler config loaded for Message" in your profiler log file. Every 5 minutes new profiling data will be written to the logfile. This is what it can look like:

Optimizing NMEA Service


Here is some typical output the profiler has written to the log file.


2010-06-09 14:37:41.5010 Message Time: 142296.875 Count: 5136 CPM: 85.60 Usage: 3.95 %
2010-06-09 14:37:41.5010 Time: 99828.125 Count:3890 Avg: 25.6627 Max: 906.25 RouteMessage Teltonika+SOS+Alarm+Inside Geofence+Outside Geofence
2010-06-09 14:37:41.5010 Time: 30453.125 Count:711 Avg: 42.8313 Max: 1515.625 RouteMessage Teltonika+SOS+Alarm+Inside Geofence+Outside Geofence+temperatuur+Temp higher than 20C


"Message Time" is total number of milliseconds consumed for Message Round Trips. "Count" is the number of roundtrips. "CPM" is the number of round trips per minut. "Usage" is how many per cent of the servers CPU was used processing round trips. Note that every CPU core you have on your server is 100%. On a Quad Core CPU you have 400% available.


The second and third row shows different kinds of round trips, how much time they consume, and how many times they happen. All time data is in milliseconds. "RouteMessage Teltonika" means that processing of message has started and the Teltonika protocol is used. And then follows all Event Rules that are assigned.


If you extend the Label tag in Profiler.config to include USAGE and COUNT_PER_MINUTE, you can see the CPU usage for individual types of round trips. Like this:


<Label>Time: [TIME] Count:[COUNT] Avg: [AVG_TIME] Max: [MAX_TIME] CPM: [COUNT_PER_MINUTE] Usage: [USAGE] [LABEL]</Label>


Ok, good. Now we have the data. We know how much CPU is required by different sets of rules and trackers. To free up server resources you can remove unnecessary and unused Event Rules. You can typically remove the sample rules that are present after install by default if you are not using them. And then consider slow down the update rate from the trackers that consumes most resources. To have smart updates rules in your trackers and consider speed, change, distance change, turns etc is often a very good way to reduce server load. And you can easily monitor the effect this will have using the profiler.


There are also other profiling channels available to measure other things inside GpsGate Server, like database access. This can be useful for a super advanced user, but is not covered here.

Example Profiling and NLog config files.

Here are some some useful config files you can download, install and use. They will produce profiling files, and separate error log files. Using those config files is a recommended setup.


1. Download and UNZIP:
http://franson.com/gpsgateserver/samples/ProfilerSetup.zip

2. Copy the files to the folders “Franson NMEA Service” and “IIS” where you installed GpsGate Server.

3. Restart NMEA Service.


The following folders are created:
C:\GpsGateServer\ProfilingLog (Profiling statistics is written here)
C:\GpsGateServer\ErrorLog (Error log is written here)
C:\GpsGateServer\InfoLog (Info log is written here)

Control peak loads


As mentioned in the beginning of this blog a server can come to a stand still when it is temporarily overloaded, and it can have difficulties to get out of it and continue normal operations. Here is why; if there is a temporary peak of traffic which the server cannot handle, transactions are queued up. And a lot of parallel queued up transactions are very slow to process, slower per transaction than under normal load, and this leads to a bad circle where all incoming transactions are queued up, and making the average time to complete a roundtrip longer and longer. Until we finally have server that hardly responds.


We want to avoid that. And here is how to avoid it.

You configure this in "IIS\Web.config" and "Franson NMEA Service\GpsGate.Service.exe.config". Place the following tag inside the "appSettings tag, and then restart NMEA Service and IIS. The value 4 is the number of parallel round trips you allow.

Set this number to the number of CPU cores you have on your server. But never to a lower number than 4.


  <add key ="MaxDatabaseConnections" value="4"/>


If you have too few connections available you will get the error message “Waiting for a free connection timed out.” in your log file. A very useful tips is to log errors in a separate file, in this way they are easier to detect. This is described in this forum post.


To get “Waiting for a free connection timed out.” under a few seconds is normal. If you get it under longer periods of time, minutes or hours, you should consider optimize your Event Rules, or update rates of your trackers. Or move to a better performing hardware.


Hope this was useful and a good start, we will release more tools and information about how to get control over your GpsGate Server performance over time.

    More profiling channels


    In the core patch released June 30. 2010 three new useful profiling channels were added.

    MessageError - Counts the number reports from trackers that are not properly processed by the server. And makes a top lists of the reasons.

    MessageCount - Counts number of reports received from each tracker. And makes a top list. If you have performance problems you can see which trackers are sending most information to the server.

    EventRule - Checks how much CPU resources each Event Rule consumes. If you have performance problems you see in per cent how much CPU time each individual Event Rule consumes.

    MySQL optimizations


    Use MySQL ODBC 5.1.5 or later. Earlier versions have too many bugs and are not recommended for production use.

    Decrease the timeout before MySQL closes an inactive connection. This is done by inserting the line below into the my.ini config file, and then restarting MySQL and IIS.

    # Changes default timeout from 28800 seconds to 60 seconds
    wait_timeout = 60

     


      Discuss this blog post on the forum please



    Download free GpsGate Server

    Install it on your own server. The installation is free for 5 users.

    Download Now