Site Configuration

To configure the site you either have to have Site Configuration or Admin privileges. Configuration is used for managing many (but not all) of the portal characteristics. Sites, instruments, and variables are configured on their own specific pages.

Logging In

  • Click “Sign In” located at the upper right corner of your portal
  • Log into your portal using
    User: admin@chordsrt.com
    Password: realtimedata
    
  • Change your password by clicking “Users”
  • Click on “admin@chordsrt.com”
  • Click “Edit User”
  • Click “Change Password”
  • Enter your new Password and use realtimedata as the old password to save the changes
  • Click “Update”

All Done!

Creating Users

Users can create their own accounts but admins will have to give them permissions to view and work on other parts of the portal. To create/become a new user:

  1. Click “Sign In” in the upper right corner of your Portal
  2. Click “Sign Up
  3. Enter your Email and enter your new Password, minimum of 8 characters long
  4. Click “Sign up

You will automatically be set to “guest” so an Admin will need to change your permissions before you can do more on the portal.

Tip for Admins If you want to check user permissions while using two windows in your browser either make one browser incognito mode or open a new window in another web browser. Otherwise you will mix up the sites cookies and have to clear your browser cache.

Portal Configuration

Admins can edit the portal configuration as well as users with Site Configuration permission.

  • Click on “Configure” selection in the upper right part of the screen. It will take you to a page called Configuration You will find buttons that will allow you to
  • Export/Import previous Configuration
  • Export/Import previous InfluxDB Data
  • Configure JSON-LD Options
  • Configure InfluxDB Tags
  • Fill out the following under Standard Options:

Project name

DOI (optional, but please use the CHORDS DOI of 10.5065/d6v1236q or another DOI that is linked to the CHORDS DOI in the metadata)

Affiliation

Page title (e.g. the name of the project you’re working such as “Orcas”, “3D-PAWS”, “STORM”, “NCAR weather stations”, etc.)

Domain name The domain name should either be the fully qualified domain name of the portal (if you have one) or the IP address of the server on which the portal is hosted, e.g. from AWS. Make sure to change the default value as the archiving function may fail if it is not set!

Data Archive URL This is the link to the archive of data being ingested by CHORDS. Users will generally archive the data in the portal on regular intervals.

Time Zone

Project Description A short description of your project. This may use [HTML Markup]. If you haven’t used HTML markup or aren’t sure what it is, you may see this link.

Logo File You may upload a customized logo for your portal here. It will appear in the portal header page.

Max Download Points Set how many maximum points you want in your data downloading.

Note: Do NOT make this blank

Select Security options Default security options enable others to view data. If you don’t want to let guests or others view your data check “Restrict view of data”. Do NOT modify the Measurement Security Key

  • Select Metadata Ontology Vocabularies.
  • Enter CUAHSI Data Services, if desired.
  • Fill out contact information.
  • Press Save.

Sites

Create a new site by clicking on Sites in the middle left part of the menu.

Click the New Site button

Fill out the empty fields

Click Create Site

Instruments

Click on the Instruments button in the middle left part of the screen

Click New Instrument

Fill out the blank spaces and click Create Instrument

Definitions

Name: Name you gave the instrument

Sensor_Id: ID for a specific sensor. This takes priority over Instrument_ID and creates a unique key. This allows the sensor to be moved from one instrument to another if need be and still retain the same URL format for uploading data.

Topic Category: Pull down menu of categories and instruments

Description: (Add additional information you feel necessary here. Like location, website, or purpose.)

Site: Place instrument is located at.

Display Points: How many points are shown on the graph.

Plot offset: Time the plot for display.

Sample Rate: How often you are sampling data, in seconds (fastest is 1 second). It is also used for visual purposes as this determines when the little circle next to the instrument is green or red.

Variables

Click on your Instrument

Scroll down and click on Add New Variable

Change the short name, name, units, the minimum and maximum for plotting, the measured property, and general measurement category to match what your Instrument will be measuring.

Metadata

There are two key places you will want to make sure to include a little extra metadata (description of data).

First, your measured properties and units are part of the metadata, but don’t just call them T or I. One letter doesn’t count for metadata as it’s needed to describe what the data is showing. So give it a more descriptive name (temp, WaterPr). Your data users will appreciate this as metadata is downloaded with your data.

Second, under your configuration page there is a section for entering JSON-LD metadata. Metadata from your entries create a JSON-LD record that is encapsulated within your Portal instance. These records allow Google (through its Data Set Search tool) and other harvesters to discover the existence of your server. This increases the visibility of your data on the web. For more information visit Project 418

Email

  • To change an email address simply click on “Users” and then click “Edit”
  • Type email into the Email Box and click “Update User”

Creating and Sending Data

Admins and Site Configs do not have permission to create test data or send data to CHORDS for security purposes. Any Registered user with Measurement Creator checked can create test data and send data to CHORDS. However Admins will have to give that user an API key. To create an API key for a user:

  • Click Users
  • Click the Measurement User
  • Click Edit
  • Click Renew API Key

User Management

Editing User Permissions

  1. To edit users go to the Users tab
  2. Click on the email/username of the user you want to edit
  3. Click “Edit User”
  4. Select the permissions you want for your user
  5. Make sure to unselect “guest”
  6. Click “Update User”
Note:

if you’re going to update a user make SURE to uncheck the “guest” box before saving your changes. If you don’t “guest” will uncheck all your previous updates and the user will have NO permissions.

Note:

IF you select Guest in user configuration it will unselect ALL other options. Please make sure to unselect Guest before you save User Permissions.

Guest: The Default for any new user except Admins. It has the minimum permissions and users should request that their Admin upgrade them to a Registered user at the least.

A Guest User Can

  • Read About page
  • View self information

Registered User: This option is better than Guest but still limited in use. Combining it with Data Downloader or Measurement Creator will enable extra functionality in CHORDS giving your Registered User’s more POWER. Just make sure to use your powers for good and not evil, ok?

A Registered User Can

  • Read About page and view data
  • View Site
  • View Instruments
  • Can View and edit self information

Data Downloader: You guessed it. This option lets Registered Users view and download data from instruments.

A Data Downloader Can

  • Download data
  • View Instruments, (only if combined with Registered user)

Measurement Creator: Behold! The power of creation! This option let’s Registered Users view and create instruments and variables.

Measurement Creator Can

  • Create test data
  • Create test measurements

Site Config: This user is similar to an Admin. They can do everything except editing users and creating test data.

A Site Config Can

  • Edit Site Configuration
  • Read About page and view data
  • View Site
  • View Instruments
  • Can View and edit self information
  • Download data
  • View Instruments
  • Create Instruments

Storing Measurements

Sending a URL from UNIX

Data In

It is easy to submit new data to a Portal, simply using standard HTTP URLs. The URL can be submitted directly from the address bar of your browser (but of course this would get tedious). We will first describe the URL syntax, and follow this with examples that demonstrate how easy it is to feed your data to a CHORDS Portal, using Python, C, a browser or the command line. These are only a few of the languages that work, and you should be able to figure out a similar method for your own particular language. Almost all programming languages have functions for submitting HTTP requests.

URL Syntax

Sample URLs for submitting measurements to the Portal:

http://myportal.org/measurements/url_create?instrument_id=[INST_ID]&wdir=038&wspd=3.2&at=2015-08-20T19:50:28
http://myportal.org/measurements/url_create?instrument_id=[INST_ID]&p=981.2&email=[USER_EMAIL]&api_key=[API_KEY]
http://myportal.org/measurements/url_create?instrument_id=[INST_ID]&p=981.2&email=[USER_EMAIL]&api_key=[API_KEY]&at=2015-08-20T19:50:28&test

myportal.org is the hostname of your Portal. The fields after “?” are qualifiers, each separated by “&”. Measurements for variables are specified by shortname=value pairs. You do not need to include measurements for all variables defined for the instrument, if they are not available.

Qualifier Optional Meaning
instrument_id=id No The Portal assigned instrument identifier.
at=time Yes Specify a timestamp to be applied to the measurements. If at is not specified, the measurement will be stamped with the time that it was received by the Portal (often quite adequate). The time format is ISO8061.
email=[USER_EMAIL] Yes If the Portal has been configured to require a security key for incoming measurements, the user email, email qualifier is needed.
api_key=[API_KEY] Yes If the Portal has been configured to require a security key for incoming measurements, it is specified with the api_key qualifier. Keys are case sensitive and specific for a given user with the measurements permission enabled.
test Yes Add the test qualifier to signify that the measurements are to be marked as test values. Test measurements may be easily deleted using the Portal interface.

Programming Examples

Data can be submitted to a portal just by typing the URL into the address bar of a browser. It's unlikely that you would use this method for any serious data collection!
  wget http://chords.dyndns.org/measurements/url_create?instrument_id=25&wdir=121&wspd=21.4&wmax=25.3&tdry=14.3&rh=55&pres=985.3&raintot=0&batv=12.4&at=2015-08-20T19:50:28&email=[USER_EMAIL]&api_key=[API_KEY]

  curl http://chords.dyndns.org/measurements/url_create?instrument_id=25&wdir=121&wspd=21.4&wmax=25.3&tdry=14.3&rh=55&pres=985.3&raintot=0&batv=12.4&at=2015-08-20T19:50:28&email=[USER_EMAIL]&api_key=[API_KEY]
  
The wget and curl commands, available in Linux and OSX, can accomplish the same thing from a console.
  #!/usr/bin/python

  #Put a collection of measurements into the portal
  import requests
  url = 'http://my-chords-portal.com/measurements/url_create?instrument_id=3&t=27.1&rh=55&p=983.1&ws=4.1&wd=213.5&email=[USER_EMAIL]&api_key=[API_KEY]'
  response = requests.get(url=url)
  print response
  ...
  <Response [200]>
  
This example uses the libCurl library in a C program to send a measurement URL to a portal.
  #include <stdio.h>
  #include <curl/curl.h>

  int main(void)
  {
    CURL *curl;
    CURLcode res;

    curl = curl_easy_init();
    if(curl) {
      // if using instrument_id:
      char* url = "http://chords.dyndns.org/measurements/url_create?instrument_id=25&wdir=121&wspd=21.4&wmax=25.3&tdry=14.3&rh=55&pres=985.3&raintot=0&batv=12.4&at=2015-08-20T19:50:28&email=[USER_EMAIL]&api_key=[API_KEY]";

      // if using sensor_id:
      char* url = "http://chords.dyndns.org/measurements/url_create?sensor_id=your_sensor_id&wdir=121&wspd=21.4&wmax=25.3&tdry=14.3&rh=55&pres=985.3&raintot=0&batv=12.4&at=2015-08-20T19:50:28&email=[USER_EMAIL]&api_key=[API_KEY]"; 

      curl_easy_setopt(curl, CURLOPT_URL, url);
      /* example.com is redirected, so we tell libcurl to follow redirection */
      curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);

      /* Perform the request, res will get the return code */
      res = curl_easy_perform(curl);
      /* Check for errors */
      if(res != CURLE_OK)
        fprintf(stderr, "curl_easy_perform() failed: %s\n",
                curl_easy_strerror(res));

      /* always cleanup */
      curl_easy_cleanup(curl);
    }
    return 0;
  }
  

Data Out

It’s just as easy to get data out of a Portal as it is to put data in. This can be done directly from the Portal web page. Or you can use HTTP URL’s to fetch data. The URL can be submitted directly from the address bar of your browser, which will deliver the data in standard formats such as CSV files, JSON files, or plain JSON.

You can also retrieve data using your favorite programming language to construct a program to send URLs and receive data, letting you build analysis and visulaization apps that can process your real-time observations. Using JavaScipt, you can even build widgets and pages that display your data on your own web site.

We will first describe the URL syntax for retrieving data, and follow this with examples that demonstrate how easy it is to integrate your analysis activities with a CHORDS Portal using Python, HTML, IDL, Matlab, R, sh, etc. You get the idea.

URL Syntax

Sample URLs for fetching data from the Portal:

http://myportal.org/instruments/1.csv
http://myportal.org/instruments/1.csv?start=2015-08-01T00:30&end=2015-08-20T12:30
http://myportal.org/instruments/4.geojson?email=[USER_EMAIL]&api_key=[API_KEY]
http://myportal.org/instruments/3.xml
http://myportal.org/instruments/3.json?last

myportal.org is the hostname of your Portal. The fields after “?” are quallifiers, each separated by “&”.

The number following “instruments/” is the instrument identifier.

Following the instrument identifier is the format that the data will be returned in (csv, geojson, json or xml).

Some formats result in a data file being returned to you browser, which can be saved in a directory. The other formats directly return text, which can be easily ingested into programs.

Format File or Text Data Product
.csv File Data in a comma-separated-value GeoCSV (CSV) file. CSV files can be opened automatically by spreadsheet programs such as MS Excel.
.geojson File Data in a GeoJSON structured file, following RFC 7946. Most scripting programs can easily read JSON into a structured variable.
.xml File Data in an eXtensible-Markup-Language (XML) structured file.
.json Text Data in straight JSON format. This format is used to bring data directly into a processing program.

Fields after “?” are quallifier pairs, with each separated by “&”. The qualifiers are optional, and are used to refine the data request.

If time qualifiers are not specified, data for the curent day are returned.

Qualifier Meaning
start=time Start time of the data span, in ISO8061 format.
end=time Start time of the data span, in ISO8061 format.
email=[USER_EMAIL] If the Portal has been configured to require a security key for downloading data, the user email must also be specified with the email qualifier.
api_key=[API_KEY] If the Portal has been configured to require a security key for downloading data, it is specified with the api_key qualifier. Keys are case sensitive and must be paired with the user email associated with the api key.

Data Backup

There are many ways to backup your data and configuration on your CHORDS portal. First you should note that when you run python chords_control --config a backup copy of your configuration files is created. You can also manually back up your portal by saving your configuration and database once everything is set up.

To save Configuration:

  • Go to Configuration tab
  • Click on Export Configuration

To save Influxdb data:

  • Go to Configuration tab
  • Click on either Export Influxdb data

You can also upload previous information from an old portal to a new one by using the Import buttons located at the top of the configuration page.