Mining External Data

DIAdem 2018 Help

Edition Date: June 2018
Part Number: 370858P-01
View Product Info

DOWNLOAD (Windows Only)

DIAdem 2017 Help
DIAdem 2018 Help
DIAdem 2019 Help
DIAdem 2020 Help
DIAdem 2020 SP1 Help

In DIAdem NAVIGATOR you use the DataFinder to browse data files on your computer and on the network. In the search results, DIAdem lists the files, channel groups, or channels it finds. You can drag and drop this data into the Data Portal or navigate in the files with the file browser.

To find data from a series of tests or data relating to a specific subject, enter an appropriate search term in the search input area. If you search for meteorological data, for example, enter Precipitation in the search input area if the data set you want contains the term Precipitation as a channel name or in the comments. DIAdem browses through all the data files for the text Precipitation and lists the data files it finds on the Search Results tab.

Refer to Configuring DataFinders for information on specifying the folders that the DataFinder searches.

The properties display below the search results displays the properties of a selected file, selected channel group, or selected channel. Click one of the files to view the file properties. In the properties display, DIAdem displays file properties such as the filename, path, creation date, name, and author. Drag up the top edge of the properties display to view the entire properties list.

To browse in the files, right-click the file in the search results and select Display in File Browser. DIAdem opens the file browser and selects the file. The file browser offers a tree view of the folders which you searched. You can navigate down to channel level in the folders that have the appropriate file formats.

If you navigate in a file down to channel level, DIAdem displays the channel data as a curve in the channel preview. For example, select a channel in a TDM file to display the channel values as a curve in the channel preview on the right edge of the properties display. Drag the right edge of the properties display to the left to enlarge the channel preview.

Extending the Search

To define a search in more detail, click Advanced Search. You can enter several search conditions in the search input area. You can search for files, channel groups, and channels, because the Advanced Search is based on the TDM data model. For example, if you search for meteorological data that includes a channel called Precipitation and that is more than one month old, specify two search conditions in the search input area. As soon as you specify a search condition in one line, DIAdem automatically creates a second line. To view all the search conditions, drag the bottom edge of the search input area down.

Click in the first line of the entry field under Level and select Channel. Click <Enter a property> in the Property column and select Name. Click the third column and select the = operator. Click the button with the three dots at the end of the first line. DIAdem lists the names of all the indexed channels. Enter the letter P under Search in list and select Precipitation. DIAdem enters the channel name Precipitation in the Value column.

In the second line, select File in the Level column and select Creation date in the Property column. In the third column, select Date before <= as the operator. In the Value column, first click the displayed date and then click the button with the arrowhead to open the calendar. Select a date that dates back at least one month. If you click the button with the three dots at the end of the second line, DIAdem displays the time range of the creation date for all the indexed files.

Click Search to find files that match both search conditions. DIAdem lists the files it finds on the Search Results tab. You can also search for channel groups and channels. If you want to find channel groups that match the specified search conditions, click the little triangle on the Search button and select Search Groups. On the Search Results tab, DIAdem lists channel groups containing a channel called Precipitation and files with the specified creation date. If you want to view the corresponding channels in the results, select Search Channels.

The search results display the names of the located files, channel groups, or channels, and also the path and the properties used in the search, for example, the creation date. To display more information, drag another property, for example, the channel minimum, from the properties display into the search results of the found channels. DIAdem adds a column with the channel minimum values to the search results. Click Channel.Minimum to move the column in the search results list. Click the two triangles on the right side of the column title to sort the channels in ascending or descending order according to the channel minimum values. Select Configure Results List from the context menu to edit the display of the properties.

The logical operations line below the search conditions displays the logical operators of the individual search conditions. DIAdem numbers the search conditions in decrementing order with C1 to Cn and uses the AND operator to connect the search conditions. If you search for two values of one property, use the OR operator to link the values. If you enter several channel names in the Value column, DIAdem automatically uses OR to link the names. You can use the logical operations line to search simultaneously for files that contain the channel Name1 (search condition C1) or Name2 (search condition C2), and which are not older than one day (search condition C3):

(C1 OR C2) AND C3

Search condition entries

If you enter text in the search input area of the Quick Search or in the Value column of the Advanced Search, DIAdem automatically displays a list of the previous entries that start with the same letters. You can enter the search text in uppercase or lowercase because the DIAdem search is not case-sensitive.

In the Quick Search, you can also search for fragments if you do not know the complete filename or if the filenames all start with the name of a test series. For example, enter the fragment tr_m17_qt for a test series in which the test program automatically generates the filenames of the measurement results from the test series identification. DIAdem finds all the data sets that have the fragment tr_m17_qt at the beginning of a name or at the beginning of another property.

In the Advanced Search, you must use wildcards to search for fragments. Use a question mark (?) as a wildcard for one character, and an asterisk (*) as a wildcard for any number of characters. For example, to search for the fragment m17, enter *m17*. DIAdem finds all the data sets that have the fragment m17 at the beginning, in the middle, or at the end of a property. If you search for files with a specific filename extension, enter *.TDMS, for example, for the filename.

In the Advanced Search, depending on the type of property you search for, you can use the various operators listed in the following table. For text properties, such as Name, you use the = operator, and for numeric properties, such as Maximum values you use the usual comparative operators. For date properties, such as Creation date, you can use the # operator with Relative range, where you can select today, yesterday, the last 7 days, the last 14 days, or the last 30 days.

Operator Meaning Numeric Date Text
= Equal to ? ? ?
<> Not equal to ? ?
> Greater than ?
>= Greater than or equal to ? ?
< Less than ?
<= Less than or equal to ? ?
# Relative range ?

If you load data from a test series into the Data Portal and you want to find and compare more data sets from the same test series, you can drag and drop a file, a group, or a channel from the Data Portal into the search input area. DIAdem enters the name of the file, of the group, or of the channel as a search condition. You can also drag and drop properties from the properties display of the Data Portal or of the file browser into the search input area.

If you select several channels in the Data Portal and drag these channels into the search input area, DIAdem combines the channel names with the OR operator and creates a search condition. If you then drag a property such as the maximum value from the properties display of the Data Portal into the search input area, DIAdem combines the maximum values of these channels with OR to a second search condition. If you look for channels of a test series with a maximum value of, for example, between 4 and 12, you must define two search conditions. Combine the search conditionsMaximum>4 and Maximum<12 in the logical operations line with AND.

If you define a query you want to use later, you can save the entries of the current search with Save Query As. DIAdem saves the parameters of the current query together with the configuration of the search results list in a file with the filename extension .tdq. Use Load Query to load saved queries or the configuration of the search results list, or both. Click Delete Query on the toolbar to specify a new query in the search input area. Click Previous Query or Next Query on the toolbar if you want to reuse a previous query. To delete individual search conditions from a query, select the search condition and press <Del>. The next time you start DIAdem, the search input area contains the query you entered last. If you want to change this setting, click Settings»Current Data Provider»Default Settings and specify that DIAdem does not show a query, or that DIAdem reads the query from a file when the program opens.

Configuring DataFinders

You execute the searches described above with the local DataFinder MyDataFinder located on your computer. You install the local DataFinder with DIAdem. The DataFinder function group contains MyDataFinder and three more buttons to which you can assign DataFinder servers if you use the Predefine Setting in the context menu.

Configuring the Local DataFinder

Use MyDataFinder to search folders on your computer with the local DataFinder. You cannot delete the local DataFinder and you cannot create additional local DataFinders. To edit the settings of the local DataFinder, select Settings»My DataFinder»Configure. In the configuration of the local DataFinder you specify the search areas and the indexing of the data files. You can also see which file formats the DataFinder includes in the search.

Search areas are the file system folders in which the DataFinder searches for files. Select these folders on the Search Areas tab. You can add and delete folders, exclude subfolders from the search, and change the order in which DIAdem displays the search areas in the file browser. You can also use the context menu of the file browser to create and to limit search areas. For example, if LabVIEW is installed on your computer, the LabVIEW Data folder is in the default search area My Files or My documents. However, the folders My Pictures, My Music, and My Videos are not included in the search area because these folders do not usually contain data relevant for the DataFinder.

The DataFinder only can search files in the search area if the DataFinder indexes the files first. Click the Indexer tab in the DataFinder configuration dialog box to specify when and how often the DataFinder indexes the search areas. In the file browser, DIAdem displays an hourglass next to files and search areas that the DataFinder has not yet indexed.

The File Extensions tab has two lists that contain all the filename extensions that the local DataFinder recognizes. The Indexable extensions list shows all the indexable file types that correspond to the TDM data model. In indexable files the DataFinder can search for properties of data sets, groups, and channels. If the data is indexed, DIAdem can navigate to channel level. Click a filename extension to see which DataPlugins MyDataFinder uses to index these files. If several DataPlugins are available for one data format, you specify the sequence of the DataPlugins. The Non-indexable extensions list shows all the non-indexable file types. These file types do not correspond to the TDM data model. In non-indexable files, the DataFinder cannot search for properties of groups or channels. The DataFinder only can search for properties that the file system provides, such as the filename and the creation date. Specify which DataPlugins the DataFinder uses on the DataPlugins tab.

You can search for properties in data files, as well as in documents, and graphics. You can search for basic properties such as filename, description, title, author, and storage date, and also for custom properties. If you want to search for a custom property, for example, the name of a test sequence, it must be created when exporting a report. If you drag one of the documents you found onto the root in the Data Portal, the properties display lists all document properties. If DIAdem is not able to search in documents and graphics, check under Settings»Extensions»DataPlugins whether a DataPlugin for documents is registered.

You can optimize the search for custom properties for the local DataFinder. Custom properties are user-defined properties that you can create for data sets, groups, and channels of the data that is saved in the TDM data model. Open Settings»MyDataFinder»Optimize Custom Properties and select the custom properties to be optimized on the tabs File, Group, and Channel. DIAdem lists these custom properties in the selectable properties of the Advanced Search. If you select an optimized custom property, click the button with the three dots at the end of the entry field to include a suggested value in the search. You only can search for Date type custom properties if these custom properties have been optimized.

Generally, you should not change the index location in which MyDataFinder saves the index, and which is set up during installation. However, if the drive with the index has less than 20MB free memory space, the DataFinder server stops indexing and displays an error message. You must then either delete files to make more space available, or change the index location as follows. Open Settings»MyDataFinder»Configure»Advanced and change the Index location. If you change the index location, the DataFinder reindexes the search areas.

If you delete large search areas or if the DataFinder can no longer access its index, the DataFinder must create a new index. To do so, select Settings»MyDataFinder»Reset and delete the previous index. Indexing the search areas may take some time.

In the bottom right corner in the information area of the Windows task bar, you can open the context menu of the registered MyDataFinder. At this point you can configure or close every MyDataFinder, stop or restart indexing, and also reset the index. Select About My DataFinder to see when the last complete indexing was executed and how many files, groups, and channels are indexed.

DataFinder Server

Use TDMServer to install DataFinder servers on a network server in order to index files in the network. DIAdem communicates as a client with DataFinder servers in order to search the index and to load the files it finds. In TDMServer you can specify several DataFinder servers in order to separately index the data areas of different work groups. You cannot do this with My DataFinder. DataFinder servers index faster because of the parallel processes and manage larger volumes of data because of the more powerful database connections.

Use TDM Server to find and to analyze data on other computers and central servers in the network. To register a DataFinder Server in DIAdem, copy the client configuration created by TDM Server onto your computer. TDMServer saves the connection parameters in a file with the filename extension .urf. Double-click this URF file to register the connection parameters and the DataPlugins on your computer. Use Settings»DataFinder Server to import, configure and delete further DataFinder servers.

Use the context menu to assign one of the free buttons in the DataFinder function group to this DataFinder server in order to access the DataFinder server quickly. If you click this button, DIAdem opens the search areas of the DataFinder server in the file browser. In the search areas of the DataFinder server, you can search, navigate, and drag and drop the data into the Data Portal. You can test the connection to the DataFinder server in DIAdem in Settings»DataFinder Server and, if necessary, change the connection parameters.

If several work groups access the central data areas of a network server, you can use the Windows user accounts to manage the access rights of clients to the DataFinder server and data. TDMServer uses Windows access rights so that members of a work group can search for and load data only in the shared folders.

You can use DataFinder servers as an ASAMODS Server in order to read the data through the Corba interface of ASAMODS. The ASAMODS interface makes it possible to connect ASAM conform programs as clients with this server. If you register the connection settings on a DIAdem computer, DIAdem can either access the data from the DataFinder server directly or can open the data as ASAM ODS data in the file browser in DIAdem NAVIGATOR. Register the DataFinder server under Settings»Data Stores as a data store in order to use the DataFinder server as an ASAM ODS server in DIAdem. The data browser in DIAdem NAVIGATOR uses a model hierarchy configured on TDMServer to display the data.

Navigating in Data Stores

DIAdem NAVIGATOR displays external data stores as a hierarchical structure. You can navigate in data stores on your computer, on connected drives in the network, and in every registered database, in order to drag and drop individual channels and channel groups into the Data Portal. In the Data Stores function group in DIAdem NAVIGATOR, you can register file-based data stores and server-based data stores.

To load an example for ASAM data, click ASAM Pass Fail Analysis Example. DIAdem NAVIGATOR opens the ASAM example database in which you can browse in tests, measurements, matrices, and submatrices and drag and drop data into the Data Portal. A level which you can load into the Data Portal has a colored symbol in the file browser and, when it is selected, is highlighted in color. The ASAM model allows the definition of further levels, such as engine, test condition, or operator, the display of which you can define and save in the file browser by clicking Settings»Current Data Provider»Browse Settings.

When an ASAM data store is open in the file browser, you can search the data. You specify the level, such as TestRun, the property, such as Name, the operator, and the value searched for, such as TR_M17_QT_33*. You can search for TestRun, Measurement, and Meaquantity and configure the search results list in order to, for example, show a test description.

Adding Data Stores

Settings»Data Stores lists all data stores available in DIAdem. Double-click a data store in order to open it in DIAdem NAVIGATOR Select Create Data Store to access further data stores in DIAdem and then, for example, the data store type AOP to register an ASAM database. Enter your user name and password, and the server for the ASAM database. Assign this database and further data stores to buttons in the Data Stores function group in order to have fast access to frequently used data stores.

Loading Data from Files and Data Stores

You drag and drop files, channel groups, or channels from the file browser and from the search results into the Data Portal. If you drag a file, a channel group, or channels onto the root element of the Data Portal, DIAdem deletes the internal data and loads the new data. If you drag a file or a channel group to another position in the Data Portal, DIAdem generates new groups with the loaded channels. If you drag single channels into the Data Portal, DIAdem adds the channels to the group you worked with last.

When you drag and drop data into the Data Portal, DIAdem automatically uses the loader associated with the file type. If DIAdem cannot automatically load a file, a dialog box opens in which you specify the loader. Click Open With to see which file formats your DIAdem installation can load. To do so, disable the setting Display only Plugins with same extension. Select, for example, Import Wizard Text to load a specfic text format with this wizard.

Click Search to find and load a loader on the NIDataPlugin website if DIAdem does not have a loader for a specific file format. If DIAdem does not have an appropriate loader on the DataPlugin website, you can create a DataPlugin for the specific file format. Click Wizard to open the DataPlugin wizard. A DataPlugin is based on a VBS script that analyzes files of a specific type and provides the data for DIAdem.

To refresh the DataPlugins available in DIAdem and to add further DataPlugins, open Settings»Extensions»DataPlugins and click Update. This dialog box contains all DataPlugins registered in DIAdem. You can import DataPlugins, export them encrypted to a different computer, and also create new DataPlugins, for example, for your documents.

In earlier versions, DIAdem loaded all the data from a file to an internal storage area. Files with numerous and long channels took some time for this. Now DIAdem does not load the bulk data of a channel until you write to it for the first time. You can change the loading behavior by clicking Settings»DIAdem Settings»NAVIGATOR. For faster processing of large files, only register the channels in DIAdem or execute reduced loading on data from long term measurements.

To register data in DIAdem, right-click the file in the file browser and select Register Data from the context menu. DIAdem creates write-protected channels in the Data Portal that refer to the data without loading the data into the internal memory. To edit a registered channel you must load the data later with the function Expand Channel from the Data Portal context menu.

For reducing loading of large files in which the consecutive values differ only slightly, select Data Reduction from the context menu in the file browser. You specify the width of the interval that DIAdem applies to each data channel in order to load one value from each interval. You can load the first value from each interval or you can calculate the mean, the maximum, or the minimum for each interval. If you use several reduction methods simultaneously on one channel, DIAdem creates a new data channel for each reduction method.

Use, for example, the Bus Log Converter to convert logfiles from CAN, LIN, and FlexRay buses into the TDM format and to load them into the Data Portal. The CAN bus (Controller Area Network), LIN bus (Local Interconnect Network), and FlexRay bus are used for the communication of intelligent sensors and actuators in vehicles. The data packages transported in the bus are saved in bus logfiles. In order to interpret the data packages, the converter also needs the descriptive information from a database file, such as NI-CAN (.ncd) or LIN signal description (.ldf). Use the buttons in the Bus Log Converter function group to simplify the conversion of logfiles with specific database files. To convert into the TDM format, connect the logfile with at least one database file. You save the settings necessary for the conversion of a logfile in a user file with the filename extension .tcc, which you assign in the Predefine Setting context menu of a button.


Not Helpful