At the core of the OPQ system lies a centralized MongoDB database. The majority of this data is managed by the OPQMauka and OPQMakai systems, though some of it (Locations, Regions, Users) is managed by OPQ View, and the Health collection is managed by the Health process.
The set of MongoDB collections and their relationships constitutes the OPQ Data Model. Here's an overview of the collections:
|box_events||High-fidelity data associated with a single box.|
|BoxOwners||A bi-directional mapping from OPQ Boxes to the Users that have an "ownership" role on them.|
|events||High fidelity data (potentially across multiple boxes) when measurement values trigger further data collection.|
|fs.files, fs.chunks||Implements GridFS for storage of binary waveform data.|
|health||Timestamped data indicating if OPQ Boxes and services appear to be running.|
|locations||Definitions for locations in terms of latitude and longitude.|
|measurements||Short term, low fidelity OPQBox data.|
|opq_boxes||Information about individual OPQBoxes, such as its current (and prior) locations.|
|regions||Implements aggregations of locations and regions.|
|system_stats||Timestamped documents with near-real time information about the state of the system.|
|trends||Long term, aggregated OPQBox trend data.|
|UserProfiles||Information about users: username, first and last name, and role (user or admin).|
The OPQ system is comprised of a multitude of different tools, libraries, and frameworks. In order to minimize confusion, we mostly follow a basic set of naming conventions for our collections and documents that we feel will keep things as simple as possible:
- All collection names and documents fields are in lower-case
- Collection names should always be plural
- Use underscores over camel-case to separate words
The box_events collection provides the event meta-data for a given OPQBox.
As an event can be associated with multiple OPQBoxes, it is therefore important to understand that there can be (and often are) multiple box_event documents with the same event_id.
Query on the event_id and box_id fields to find data for a given OPQBox for a specific event.
|event_id||Integer||The event_id generated by an events document.|
|box_id||String||The OPQBox from which this data was produced.|
|event_start_timestamp_ms||Integer||Unix timestamps indicating the requested start time for the high fidelity data.|
|event_end_timestamp_ms||Integer||Unix timestamps indicating the requested end time for the high fidelity data.|
|window_timestamps_ms||[Integer]||An array of Unix timestamps that correlate with every 2000 samples (10 grid cycles) of recorded box data. This can be useful for debugging purposes, as we can determine the continuity of box data.|
|location||String||Location slug see Location for details.|
|data_fs_filename||String||Indicates the GridFS filename that holds the box_event's actual raw waveform data.|
Supplemental indexing: event_start_time_ms and box_id form a unique composite index.
The BoxOwners collection provides a bi-directional mapping between OPQ Boxes and the users who have ownership over them.
|username||String||The box owner's username (email address).|
|boxId||String||The Box owned by this user. A string such as "1", "2" etc. Not the documentID!|
Supplemental indexing: none.
The events collection provides access to high fidelity waveform data that was retrieved from OPQ Boxes in response to non-nominal measurements.
|event_id||Integer||A unique integer value generated for each event.|
|description||String||Indicates additional information about the event|
|boxes_triggered||[String]||A list of all OPQBoxes associated with the given event - however it is important to note that this does not always correspond to all of the OPQBoxes for which we have received actual data from for the event.|
|boxes_received||[String]||List of all OPQBoxes from which high fidelity data was received for the event|
|latencies_ms||[Integer]||an array of timestamps (milliseconds since epoch) indicating the time when data from each OPQBox was received. Maintains a 1 to 1 correlation with boxes_received.|
|target_event_start_timestamp_ms||Integer||Unix timestamps indicating the requested start time for the high fidelity data.|
|target_event_end_timestamp_ms||Integer||Unix timestamps indicating the requested end time for the high fidelity data|
Supplemental indexing: event_id is a unique index, target_event_start_time_ms is an index.
GridFS is a MongoDB specification for storing large documents. As an OPQBox can collect a very large amount of data for each given event (often exceeding the 16 MB MongoDB document size limit), we've opted to utilize GridFS to store our high-fidelity data.
At its core, GridFS is a very simple system consisting of two collections, fs.files and fs.chunks.
The fs.files collection stores file metadata:
|filename||String||Generated by GridFS, corresponds to the box_event's data_fs_filename field,|
|length||Integer||Generated by GridFS|
|chunkSize||Integer||Generated by GridFS|
|uploadDate||Date||Generated by GridFS|
|md5||String||Generated by GridFS|
|metadata.event_id||Integer||Added by Makai, used to find the corresponding box_event document.|
|metadata.box_id||String||Added by Makai, used to find the corresponding box_event document.|
|metadata.incident_id||String||Added by Mauka, used to find the corresponding incident document.|
Supplemental indexing: none
Note: The GridFS specification requires the metadata field be used to store any external information for the given file document. See GridFS files.metadata for more information.
The fs.chunks collection has the following structure:
Supplemental indexing: none
The OPQ Health service creates documents representing its findings on the current health of the system with the following structure:
|timestamp||Date||Each entry has a timestamp, which is a UTC string indicating the time at which the entry was generated.|
|service||String||Indicates the OPQ Service whose status is being described in this entry. Service should be one of the following: "box", "mauka", "makai", "view", "mongodb" and "health". Yes, OPQHealth reports on its own health!|
|serviceID||String||For some services, such as "box", additional identifying information is required. The serviceID field provides that information. In the case of OPQBoxes, the serviceID field provides the boxID.|
|status||String||Status is either "up" or "down".|
|info||String||Info is an optional field that can be used by OPQHealth to provide additional information about an entry.|
Supplemental indexing: timestamp.
The Incident entity is under construction and not yet available.
The incidents collection contains documents that classify one or more events. An incident represents a deviation from nominal values for either frequency, voltage, or THD that has been classified.
|incident_id||Integer||A unique integer representing the incident|
|start_timestamp_ms||Integer||Start of the incident (ms since epoch)|
|end_timestamp_ms||Integer||End of the incident (ms since epoch)|
|measurement_type||String||One of [VOLTAGE, FREQUENCY, THD, or TRANSIENT]|
|deviation_from_nominal||Float||Absolute value of measurement deviation from nominal|
|measurements||[Measurement]||Copied from event|
|gridfs_filename||String||Filename of trimmed waveform copied from event|
|classifications||[Classification]||List of classifications that can be applied to incident (see table below)|
|ieee_duration||String||A string indicating one of the standard IEEE durations associated with this incident (see table below)|
|annotations||[Annotation]||List of annotations associated with this incident|
|metadata||Object||Key-Value pairs providing meta-data for this incident|
Supplemental indexing: box_id, location, start_timestamp_ms, measurement_type, classifications
Various organization such as IEEE, ITIC, CBEMA, and SEMI have proposed standardized terminology for classifying power quality deviations. The following two tables cover the classifications based on these standards. Note that the standards overlap, so an incident could have multiple, simultaneous classifications (such as both ITIC_PROHIBITED and VOLTAGE_SWELL, or both SEMI_F47_VIOLATION and VOLTAGE_SAG). For clarity, we indicate the type of incident and the duration of the incident separately, except in the case of SEMI_F47_VIOLATION, where the durations do not conform to the IEEE categories and so are included in the classication.
|EXCESSIVE_THD||Exceeds IEEE 1159 recommendations for THD (5% over 200 ms windows).|
|ITIC_PROHIBITED||Voltage observed in the ITIC prohibited region.|
|ITIC_NO_DAMAGE||Voltage observed in the ITIC no damage region.|
|VOLTAGE_SWELL||Voltage greater than 1.1 pu|
|VOLTAGE_SAG||Voltage between 0.1 - 0.9 pu|
|VOLTAGE_INTERRUPTION||Voltage less than 0.1 pu|
|FREQUENCY_SWELL||Frequency greater than 60.1 Hz|
|FREQUENCY_SAG||Frequency between 58 Hz and 59.9 Hz|
|FREQUENCY_INTERRUPTION||Frequency less than 58 Hz|
|SEMI_F47_VIOLATION||Voltage observed at 0.5 pu for more than 200ms, 0.7 pu for more than 0.5 seconds, or 0.8 pu for more than 1 second.|
Note: pu stands for "per unit" and 1pu = nominal. In the U.S. 1pu = 120V.
The following table classifies the duration according to standard IEEE terminology for durations. Note that the precise duration of the incident can be determined by subtracting start_timestamp_ms from end_timestamp_ms.
|INSTANTANEOUS||A duration between 0.5 and 30 cycles|
|MOMENTARY||A duration between 30 cycles and 3 seconds|
|TEMPORARY||A duration between 3 seconds and 1 minute|
|SUSTAINED||A duration greater than 1 minute|
The locations collection provides entities that define locations that can be associated with OPQ Boxes, Trends, Events, and other entities in the system.
|slug||String||A unique, human-friendly string identifier.|
|coordinates||Array||Contains longitude and latitude coordinates in that order.|
|description||String||A description of the location.|
Supplemental indexing: slug is a unique index, description is a unique index.
Note that the coordinates array must list longitude first, then latitude. See this StackOverflow Question for more details. Mongo has great support for GeoSpatial queries, so this will be fun to have.
Location slugs should be considered permanent once defined. Since these slugs have the potential to appear in other documents throughout the database, you will have to guarantee that the location does not appear anywhere else in the database in order to delete it.
Likewise, you should not change the coordinate values willy-nilly. Only change them if they incorrectly specify the intended location.
The measurements collection provides low-fidelity OPQBox snapshot data for a specific moment in time. Documents in this collection are produced at a very rapid rate; OPQ Makai requests data from each OPQ Box at a rate of six times per second. As such, each measurement document can essentially be thought of as an OPQBox "heartbeat", providing a timestamp and some additional low-fidelity data. Documents are persisted in the collection for a period of 24 hours before expiring.
|box_id||String||The Box ID.|
|timestamp_ms||Integer||The Unix UTC timestamp in milliseconds.|
|thd||Float||Total Harmonic Distortion (over this measurement window)|
|expireAt||Date||Currently set to 24 hours.|
Supplemental indexing: box_id and timestamp_ms are a unique composite index.
The opq_boxes collection provides information about each individual OPQBox in the system.
|box_id||String||A unique string identifier for the OPQBox. This value is always referenced throughout the data model when we need to store a box_id value within a document. Currently they are strings like "1", "2", "3".|
|name||String||A unique user-friendly string identifier for the OPQBox. Unlike the box_id value, which is often used internally throughout the data model, the name value should be thought of as the external representation of the OPQBox.|
|description||String||Optional: can be used to further describe an OPQBox.|
|calibration_constant||Float||A box-specific value that is used to adjust the values returned by the analog-to-digital converter chip so that we get accurate voltage and frequency values.|
|location||String||A location slug that identifies the current location of this box.|
|location_start_time_ms||Integer||A UTC millisecond time stamp indicating the time that data from the current location began being transmitted.|
|location_archive||Array||Contains objects with fields location and location_start_time_ms. This provides a historical record of the locations associated with this box.|
|public_key||String||A 32 byte key used for CurveZMQ authentication.|
Supplemental indexing: box_id is a unique index, name is a unique index.
“Regions” represent aggregations of Locations. Conceptually, a region consists of a region name along with a list of locations that are included in that region. This is implemented via a bi-directional table called "regions":
|regionSlug||String||The unique identifier for a region.|
|locationSlug||String||The unique identifer for the location associated with this region.|
Supplemental indexing: none.
Note that regions do not have a "description". Their slug should be self-descriptive, such as a city name (i.e. "Kailua, HI") or zip code (i.e. "96734"). Note that the relationship is many-to-many: a region can be associated with multiple locations, and a location can be associated with multiple regions.
You can create a hierarchy of regions. For example, you can specify 5 locations as being in "96734", and those same 5 locations (as well as many others) can be included in the region named "Hawaii".
Region and location slugs together constitute a single namespace (i.e. you can’t have two locations or two regions both called “96734”, nor can you have a location called “96734" and a region also called “96734”).
The SystemStats collection consists of a single timestamped document which provides "near-real time" information about the state of the system. The goal of the SystemStats collection is to facilitate system scalability: a server-side cron job can update the SystemStats document once every 10 seconds, and then all connected clients will be updated with the revised document that summarizes the most recent state of the system. This is preferable to clients retrieving individual Measurements documents, for example.
|timestamp||Date||The time when this data was collected.|
|events_count||Integer||The total number of events in the database.|
|events_count_today||Integer||The total number of events created so far today.|
|box_events_count||Integer||The total number of box_events in the database.|
|box_events_count_today||Integer||The total number of box_events created so far today.|
|measurements_count||Integer||The total number of measurements in the database.|
|measurements_count_today||Integer||The total number of measurements created so far today.|
|trends_count||Integer||The total number of trends in the database.|
|trends_count_today||Integer||The total number of trends created so far today.|
|opq_boxes_count||Integer||The total number of opq_boxes in the database.|
|users_count||Integer||The total number of users.|
|box_trend_stats||Array||Each element is an object with fields boxId (the box ID), firstTrend (the timestamp of the earliest trend collected for this box), lastTrend (the timestamp of the most recent trend associated with this box), and totalTrends (the total number of trends associated with this box)|
Supplemental indexing: none.
The trends collection provides OPQBox measurements of voltage, frequency, and THD that are persisted indefinitely. Each trend document represents data aggregated over a one minute data collection window for an individual OPQBox. Trend data is essentially a "roll-up" of the measurement data associated with a box.
|box_id||String||The box ID.|
|timestamp_ms||Integer||The Unix UTC timestamp.|
|location||String||The location slug indicating the place this box was located at the time of data collection.|
|voltage||Object||An object with fields min, max, and average. Each are floats, representing the voltage values calculated for the minute preceding this timestamp.|
|frequency||Object||An object with fields min, max, and average. Each are floats, representing the frequency values calculated for the minute preceding this timestamp.|
|thd||Object||An object with fields min, max, and average. Each are floats, representing the THD values calculated for the minute preceding this timestamp.|
Supplemental indexing: box_id and timestamp_ms are a composite index.
Users are represented in OPQ by three collections: the Users collection (maintained by Meteor, which provides password and basic account information, which is not shown below), UserProfiles (additional profile information, shown below), and BoxOwners (which provides a two-way mapping between OPQ Boxes and their owner(s), discussed elsewhere on this page).
Here's the UserProfile collection structure:
|username||String||The user's username, which is their email address.|
|firstName||String||Their first name.|
|lastName||String||The user's last name.|
|role||String||Currently, the defined roles are "user" and "admin".|
|phone||String||Their phone number + provider|
|unseen_notifications||Bool||Turns true when user has new notifications they haven't seen yet.|
- text and email are booleans and represents how user wants to receive notifications.
- max_per_day is a string and represents how often user wants to be sent notifications. It can be set to 'once a day', 'once an hour', or 'never'.
- notification_types is an array of notification types the user wants to receive. All notification types users can subscribe to are stored in an array in the Notification collection as notificationTypes.
Supplemental indexing: username is a unique index.
A notification document is created whenever something of interest happens and maps to users that are interested in that particular notification type. Notification documents are removed from db after a week.
Here's the UserProfile collection structure:
|username||String||The username of the user that is associated with this notification doc|
|type||String||The type of notification. i.e. 'system service down'. UserProfiles collection stores all the notification types a user is subscribed to in an array of strings.|
|timestamp||Date||Time when the notification doc was created.|
- summary is a string that provides additional info about the event that triggered it
More fields to be added as more notification types are added.
|delivered||Boolean||Turns true once the notification is sent to the user it is associated with.|
In OPQ View, only admins are able to define entities such as users, locations, regions, and OPQ Boxes. Users basically have only "read access" to the data in the system. (This will change in future when we provide the ability for users to annotate events and incidents.)