Data Server Manual

The Data Server is designed to facilitate efficient data exchange between two or more Texmate Tiger programmable meter controllers over a TCP/IP connection as well as provide protected web access to the same data.

By default the Data Server listens on port 9995 for incoming connections from Tiger meters. The HTTP server runs by default on port 8080, and this is where you set up meter conection details, the graphing paramters, and view or download raw data.

Contents:

  1. Overview
  2. Installation Guide
  3. TTP Protocol
  4. Sample Apps
  5. Server Configuration
  6. Graphs and statistics, and data downloads
  7. Multiple-Meter display
  8. Data Storage
  9. The Administration Control Panel
    1. The initial Administrator user
    2. Users
    3. Units
  10. The manage utility

  1. Overview

    The Data Server is designed to fulfill two purposes:

    1. Facilitate exchange of data between Tiger Meters at disparate locations over TCP/IP.
    2. Log all data exchanged for trend analysis via a web browser.

    To accomplish these two tasks the Data Server is made up of two core parts, The HTTP server running by default on port 8080, and the TTP server which communicates over TCP/IP via Texmate's TTP (Tiger Transfer Protocol) on port 9995 by default.

    When two or more Tiger Meters need to communicate with each other from dispatate locations there is an inherent problem in one not knowing the exact location of the other; the Data Server solves this conundrum. The Data Server runs on a computer with a known IP address. Both the meters are set up with this address so that they can communicate without requiring the expense of having their own fixed IP adresses, or problematic lookups to discover the other units address. One unit authenticates itself with the Data Server and leaves a packet of data on the server for the other to in turn authenticate itself and retrieve that data. All of this data is logged for future analysis and export.

    The Data Server is an enterprise platform: You can expand it's power with Apps for both the HTTP server and the TTP server. Apps for the Data Server (as opposed to Apps for the Tiger Controller products or embedded devices) are written in Python, a friendly, easy to use high level language used at places like Google and Industrial Light and Magic.

    This guide will help you set up communication between meters and show you how to access the Graphs and Reports module, an aid in analysing the stored data.

    ↑ Top


  2. Installation Guide

    See the Instalation Guide on texmate.com/software/docs/installation-guide-windows/ for details.

  3. TTP Protocol

    See the TTP Protocol Specification on texmate.com/software/docs/ttp-protocol/ for details.

  4. Sample Apps

    Sample Apps are provided for you to see how TTP (Tiger Transfer Protocol) works for communicating with the TexmateDSE server. A simple example of the protocol is given below:

    Upload some data from a Tiger controller:
    TTP /up/unitcode/<2009-1-16 15:30:45;11;22;33;44> unitkey
    response:
    OK

    Notes on the timestamp format:

    1) Leading digits on the year are optional, and will be interpreted as filling out to 2000 if missing. Thus, 7-1-16 is equivalent to 2009-1-16, and 97-09-12 is equivalent to 2097-09-12.

    2) Microseconds are optional, and can be sent, so that the upload could look like this:
    TTP /up/unitcode/<2009-1-16 15:30:45.10;11;22;33;44> unitkey
    Trailing zeros can be truncated from the microseconds. If you choose to truncate the microseconds portion, an implicit specification of 0 is assumed for each missing digit. Thus, 2009-1-16 15:30:45.10 is equivalent to 2009-1-16 15:30:45.10.100000, and 2009-1-16 15:30:45 is obviously equivalent to 2009-1-16 15:30:45.000000.

    Multiple samples can be uploaded at once (as many as you wish):
    TTP /up/unitcode/<2009-1-16 15:30:45;11;22;33;44><2009-1-16 15:30:46;12;23;34;45> unitkey
    response:
    OK

    Download the last posted data sample from a Tiger controller:
    TTP /dl/unitcode/last unitkey
    response:
    OK <123 2009-01-16 15:30:45 L1=11 L2=22 L3=33 L4=44>

    Each transmission, both request and response, terminates with a Carriage Return character.

    Possible error responses:
    400 Bad request
    402 Unauthorized
    404 Data Not found
    405 Unit Not Found
    406 Duplicate sample
    500 Internal error
    501 Not implemented

    The 406 Duplicate sample error is a special case, in that when multiple samples are uploaded in a single transaction, and a duplicated timestamp is detected, processing of the remaining samples in the transaction continues and the duplicated timestamp(s) are returned appended to the error message, along with their offset. eg:
    TTP /up/unitcode/<2009-1-16 15:30:45;11;22;33;44><2009-1-16 15:30:45;11.50;23;34.5;46><2009-1-16 15:30:46;12;23;34;45> unitkey
    response:
    406 Duplicate sample 2<2009-1-16 15:30:45>
    But sample 1 and 3 completed successfully. Multiple duplicates are returned if more than one sample datetime is duplicated. Timestamps are returned as sent, thus 7-1-16 15:30:45 is returned unchanged if duplicated.

    Below are some sample Apps for both the Tiger 380 and 320 Series of Controllers on Texmate.com:

    Texmate also has a comprehensive set of documentation for our Texmate App language available at http://www.texmate.com/macros/.

    ↑ Top


  5. Server Configuration

    The configuration file txm.conf located in the INSTALLDIR (On Microsoft Windows platforms this defaults to C:\Program Files\Texmate\Data Server 3\) has a great deal of configuration parameters for you to change.

    The search locations for txm.conf used by the Data Server are as follows:

    1. User home directory (not used when run as service)
    2. WINDIR
    3. C:\
    4. INSTALLDIR

     

    On unix-like systems (linux, OS X etc)

     

    1. ~/
    2. /etc/
    3. INSTALLDIR

    A minimal configuration file is shown below:

    	[main]
        	DS_PORT: 9995
        	HTTP_PORT: 8080
        

    DS_PORT is the port that the Texmate Tiger series programmable meter controllers communicate with.

    HTTP_PORT is the port that the http server will run on for graphing and setting up meters.

    Other changeable parameters in the main section:

    MAINTENANCE_INTERVAL can be set to the number of hits the TTP Server will process before checking to see if there are any runaway, or otherwise dead, channels that can be closed. The default is 500. You should set this higher for a highly loaded server.

    CHANNEL_TIMEOUT is the maximum time in seconds that a request can process for before it is considered dead, and will be closed the next time that MAINTENANCE_INTERVAL is reached.

    SOCKET_OPEN determines whether the TTP Server leaves the socket open after each TTP request so that further requests can be sent by the same meter without reopening the socket. As of version v2.0.1.1, the sockets are left open for further transmission, so setting this to 'no' will close the sockets after each TTP request.

    Email section:

    Email settings can be configured so that the application can send out email messages via your SMTP server based on requests sent in via Units. You may need to ask your system administrator about these settings.

    The email settings are the same as found in the Django project. See the following page for a thorough explanation of each setting: http://docs.djangoproject.com/en/1.1/ref/settings/#email-host.

    Default email settings are shown below:

    	[email]
        	EMAIL_HOST: localhost
        	EMAIL_HOST_PASSWORD: 
        	EMAIL_HOST_USER: 
        	EMAIL_PORT: 25
        	EMAIL_USE_TLS: no
        	DEFAULT_FROM_EMAIL: [email protected]
        	EMAIL_ADMINISTRATOR_ADDRESS: [email protected]
        

    HTTP Apps section:

    Apps can be added to both the TTP Interface and the HTTP Interface to extend the power of the application. See the Data Server App Guide for more information.

    HTTP Apps are Django apps. The HTTP Interface itself is a Django project with apps for units, data, and so forth. See the Django Project documentation for more information: http://docs.djangoproject.com/en/1.1/ for more information.

    Defining your own apps is easy. Develop them, and then place them in the HTTPApps/ folder in the Installation directory, and list them according to the example format.

    For example, if your app is an iPhone web interface in the folder ``HTTPApps/iphone``, then you could define your app and url location as in the example below. iphone.url will enable your url patterns in iphone/urls.py at the iphone/ url on the server.

    	[httpapps]
        	iphone: on
        	iphone.url: iphone/
        

    The .url line of the http app definition is not necessary - the App will still be available, it just won't be mapped to urls for client access in a web browser. If there is an admin.py file (an admin interface definition) then your app will be manageable via the built-in Administration Control Panel.

    TTP Apps section:

    TTP Apps are simple producers as documented in the source code of the example below found in <INSTALLDIR>/TTPApps/ok.py. See the Data Server App Guide for full information about TTP Apps and extending the TTP protocol.

    	[ttpapps]
        	ok: on
        

    To enable your TTP App, place it in the TTPApps folder, and add the filename (without the .py extension) to the ttpapps section as in the above example. The on directive tells the TTP Server to enable this App. You can list your App, but disable it like so: ok: off.

    Database section:

    	[database]
        	DATABASE_ENGINE: sqlite3
        	DATABASE_NAME: db\data_server3.db
        	DATABASE_HOST: 
        	DATABASE_PORT: 
        	DATABASE_USER: 
        	DATABASE_PASSWORD: 
        	COMMIT_INTERVAL: 1
        

    The Data Server can use connect to any database that has a Django adaptor available. You may need to install a Python database connector for some databases as only SQLite and MySQL adaptors are included with the Data Server distribution. Full details of each DATABASE_* setting are available here: http://docs.djangoproject.com/en/1.1/ref/settings/#database-engine. See the Data Storage section of this manual for more detail on databases and the Data Server.

    COMMIT_INTERVAL is the data upload transaction commit interval. This defaults to commiting the transaction to disk for every data upload (a value of 1) but to increase performance on a very highly loaded server you can specify much higher number such as 100 or even 500. This will flush the new database additions to disk every 100 or 500 upload transactions and has been benchmarked to increase performance up to four-fold which is a significant increase. Doing so will make your data more exposed to loss from power failure or a system crash as uncomitted transactions will be lost. But for a highly loaded server averaging say 200 complete upload and download connections per second, this represents only the last two or three seconds of data which is very small, and in most cases quite acceptable. Completed transactions not comitted to disk are automatically flushed to disk on server shutdown.

    Graph section:

    You can also add a graph section to your configuration file to configure some graphing options like this:

    		[graph]
        		MAX_PLOT_POINTS: 0
        		

    MAX_PLOT_POINTS is the maximum number of points allowable to plot per line on the graphs. This is to help prevent large datasets from loading up the server too much while generating the graph. For all but very slow servers you can safely leave this number to the default of 0 (unlimited plot points.)

    ↑ Top


  6. Data Storage

    By default, the data received by the server is stored in a database file named data_server3.db and it is found in the db/ sub directory of the installation folder. We recommend that you make a backup of this file periodically in case of power failure, server crash or some other event that may corrupt the information stored in this file rendering the server inoperable.

    The format of this file is an SQLite 3 database, and you can find more information and free tools to extract data and view the contents, create custom reports, and any thing else you may wish to do with your data, at the SQLite website.

    You are not limited to the bundled SQLite database. You can deploy the Data Server core database on a number of Django supported (and unsupported) enterprise databases such as Oracle, MySQL, and PostgreSQL. Adaptors are available for IBM DB2, Microsoft SQL Server, and others. By default SQLite is installed and initialized with the Windows installer. SQLite is an excellent open format database for smaller installations of the Data Server.

    You may need to install a Python database connector for some databases as only SQLite and MySQL adaptors are included with the Data Server distribution. Data Server is built with Python 2.5, so you will need to install Python 2.5.x, install your database adaptor (and optionally a Django database adaptor), ensure that it works in your local installation of Python 2.5, and then restart the Data Server.

    If you want to use a different database to the standard included SQLite database, see http://docs.djangoproject.com/en/1.1/ref/databases/ for notes on built-in database connectors for Django, and available third party database connectors. MySQL, Oracle, and SQLite are built-in while others are available, see the above url for details.

    You will also need to look at the database settings available here.

    If you change the database engine from the default you will want to run the create the appropriate database tables for the application in your database. You can do this from a terminal (cmd.exe on Windows) by typing: manage syncdb from the installation directory. NOTE: On Windows Vista and Windows 7 you will need to run "Command Prompt" as an Administrator if you are using the default SQLite database. This is because, by default, a normal User does not have permission to modify the database file. Then you can run the manage syncdb command successfully. Read the The manage utility documentation for more detail.

    ↑ Top


  7. Graphs and statistics, and data downloads.

    The Graphs and statistics, and data downloads page for each Unit is a user-configurable graphing and statistics module to view line plots and statistics of the data stored by the server in its data store. All the data transferred to and fro with the Tiger units is stored and can be viewed in graph form or exported from the data store in a csv format suitable for use in spreadsheets and for creating custom reports.

    The home page for the http server is a list of Units and Views. It presents a list of Tiger units that you can view graphs and extract data from, and a list of configured Multiple-Meter displays which display the last available data for selected units on a single screen. Simply click on a units graphs or reports option, then choose a date range that you are interested in. Graph properties such as number of data fields displayed, axis labels, and the number of axes can be changed from within the Administration Control Panel.

    Report properties such as which data fields are exported and even which units can have reports can be changed from within the Administration Control Panel.

    You can set the maximum number of allowable plot points for a graph line in the Server Configuration File.

    ↑ Top


  8. Multiple-Meter displays

    Multiple-Meter displays are designed to allow you the flexibility to see the latest data for many meters all on a single self-updating screen.

    The Views are added and configured in the Administration Control Panel by clicking the "Multiple-Meter display" link. All that is required is to give the View a name, select a refresh rate, some units and which channels to show for each unit, and you are ready to view the data from these units. Multiple-Meter displays show up in the home page of the Data Server Web interface.

    ↑ Top


  9. The Administration Control Panel

    The Data Server comes with an extensive Administration Control Panel. It is located at the url /admin/ on a running server, so by default this would be http://localhost:8080/admin/ where 8080 is the HTTP_PORT specified in the Configuration Settings file, and localhost is the domain or ip address of the server.

    The default administration control panel username is Administrator (without the quotes) while the password is admin. You should change the password at the first available opportunity for security reasons. You can also change the username of the user Administrator to something else.

    Users marked with is_staff have access to the Administration Control Panel. Permissions can be set on Users to control who has access to what in the Administration Control Panel. The Users section has more information.

    1. The initial Administrator user
    2. Users
    3. Units
    4. Data
    5. Email and Alarms

    1. The Administrator user

      The Administrator user the initial user created who is the overall server administrator and has full control over the server from the Administration Control Panel. By default, when a clean installation is performed, the Administrator password is set to admin. You should change it as soon as possible.

      To change a password, go to the Administration Control Panel, click the "Users" link to bring up a list of all users. Next locate and click the user you wish to change, then click the "change password form" link and enter the new password in the two boxes before pressing the Change button. Passwords can contain any combination of characters, but have a minimum of 1 character.

      Warning: If you forget the Administrator password (or superuser login/password) you will not be able to log back into the Administration Control Panel remotely. There is a utility provided to reset the Administrator password which can only be run from the server computer itself.

      Note: You can change the all the initial Administrator user details in the Administration Control Panel. For example, you can:

      • Change the username of Administrator
      • Remove from Administrator superuser status.
      • Add more superusers.

      Resetting the Administrator password

      By default, when a clean installation is performed, the password for the User Administrator is set to admin. Should you forget what you changed this password to, you can change it back to admin only from the computer that the data server is running on. Please note that this program will only run if the database is an SQLite database (the default) and you will need to run the program as an Administrator.

      To reset the password, locate the reset_admin_password program in the installation directory of the server (On Microsoft Windows platforms this is "C:\Program Files\Texmate\Data Server 3\" by default) and run it as an Administrator.

      You should now log into the server and change the password as detailed above.

      Note: reset_admin_password changes the password for the User with an id (primary key) of 1. This corresponds to the default User named Administrator. Therefore you can change the username, and this program will still reset the password as intended.

      If you have changed the database type, from SQLite, you will have to use your database administrative control program to reset your password for the Data Server. The appropriate database table is named auth_user, and the SQL code that reset_admin_password runs is as follows:
      UPDATE auth_user SET password = 'sha1$$d033e22ae348aeb5660fc2140aec35850c4da997' WHERE id = 1;
      You can modify this as according to your requirements.

      ↑ Top


    2. Users

      Users are managed through the Administration Control Panel. You can create Users, edit User details, change User passwords, assign User permissions, and delete Users. You will find it self explanatory once you visit the Administration Control Panel.

      User Permissions need a bit more explaining. By default you need to be a logged in User to view the main part of the HTTP Interface. You can change this setting in the Configuration Settings. The Administration Control Panel provides for assigning permissions directly to Users, or to Groups (and assigning Users to Groups.) These permissions are all to do with the Administration Control Panel access only, and will only matter if you have marked the User as staff by checking the is_staff checkbox. The is_superuser checkbox will confer superuser (Administrator) status to that particular User. Just be careful not to delete yourself.

      ↑ Top


    3. Units

      Units are managed through the Administration Control Panel. You can create Units, edit Unit details, change Unit codes and keys, and delete Units. You will find it self explanatory once you visit the Administration Control Panel.

      Details for the Tiger App

      The TTP Protocol connection fields are the Unit Code and Unit Key. Both of these are supplied with each request the Unit makes to the server so that the Unit's identity and authorization can be established. The Unit Code should contain alphanumeric characters only.

      The number of data fields are the number of pieces of data that the unit will be transferring to and from the server with each connection request. Each upload request must include the number of data fields specified each time.

      Graphing Options

      Each Unit is able to have a graph detailing send data. Graphing options are simple: you get to choose the graph dimensions to display on your screen (in pixels), the number of axes on each plot, the label for each axis, as well as the fields displayed on each axis.

      ↑ Top


    4. Data

      Data Samples and Alarms are managed through the Administration Control Panel.

      Samples and Alarms can be viewed and deleted (either singly, or in bulk), but not changed, through the Administration Control Panel. The lack of ability to change a Sample or Alarm once added is intentional.

      Note: every action through the Administration Control Panel is logged to the database. The "Delete selected Samples" action logs every deleted Sample as a separate entry in the log. Thus using that action will not result in a large decrease in the database size - instead use the "Delete ALL Samples", or "Delete old Samples..." buttons at the top of the Sample list pages. These functions will only log a single entry representing the latest deleted Sample and can save disk space. "Delete All Alarms" and "Delete Old Alarms..." are also available for Alarms

      Tip: If you are using Oracle, or some other enterprise database, you will may want to disable sample and alarm deletion as well as modification - add the following directives to your Server Configuration:
      DISABLE_ADMIN_SAMPLE_DELETION: yes
      DISABLE_ADMIN_ALARM_DELETION: yes
      and restart the HTTP Interface.


    5. Emails and Alarms

      Mailing lists and Alarm messages are managed through the Administration Control Panel.

      Each Unit can have a single Mailing List which is a list of Users (with email addresses) to whom email and alarms (via email) are sent. The configuration of the Unit Alarms is very flexible so that you need to do the minimum of work in the App loaded into the Unit, and can configure the messages easily in the Administration Control Panel. The Subject and Message of each Alarm uses the Django Template Language for formatting the email to be sent.

      The following are a list of available objects that can be used in the Subject and Message of an Alarm:

      1. {{ level }} is an integer representing the Alarm Level.
      2. {{ unit }} is the Unit. Properties such as {{ unit.name }} are also available: see the Model documentation for Unit for all available properties.
      3. {{ data }} the list of data sent with the TTP alarm request. The data list is also available as individual list items, eg: {{ data.0 }} for the first data item (usually the timestamp), {{ data.1 }} for the next piece of data sent, and so forth.

      You can do logical tests to build out your custom alarm message, eg: {% if data.0 %}The localtime for this Alarm was {{ data.0 }}{% endif %}. See the Django Template Language documentation for more detail.

  10. The manage utility

    manage.exe is a command-line utility for administrative tasks. You can access it by running Command Prompt as an Administrator (right-click, select "Run as administrator"), and changing your directory to the installation folder (cd C:\Program Files\Texmate\Data Server 3). Type manage help for a list of available commands. For help on a specific command, type manage help <command>.

    Further documentation of the manage utility is provided here: http://docs.djangoproject.com/en/1.1/ref/django-admin/ NOTE: substitute manage for manage.py and django-admin.py in the Django documentation.

Texmate Data Server Documentation