Topic outline

• Analyzing Text with InterSystems Products

  Analyzing Text with InterSystems Products

  See how the support for Natural Language Processing (NLP) text analytics found in InterSystems IRIS® data platform and InterSystems IRIS for Health™ provides semantic analysis of unstructured text data in a variety of natural languages. This enables you to discover useful information about the contents of a large number of text documents without any prior knowledge of the contents of the texts.

  This simple procedure walks you through the basic steps of generating NLP metrics. In this exercise, you will walk through the initial tasks associated with indexing text data for semantic text analysis to determine: 

  • The most common entities in those texts
  • Metrics about those entities
  • Various relationships found between those entities
  • The contexts in which an entity appears

  These activities are designed to use only the default settings and features, so that you can acquaint yourself with the fundamentals of NLP text analysis.
  
• Collapse allExpand all
  Try It

  Try It

  This example is provided to give you initial experience with InterSystems IRIS Natural Language Processing. 
  

  
  

  •  You must be logged in to launch an InterSystems IRIS instance.

    • Log in with your existing account.
    • Don't have an InterSystems account? Create a new account.
  • Create a domain and add data locations
    

    There are several ways to create, populate, and build a domain to group related source texts for collective analysis. The following example uses the Domain Architect interface to create a domain in the SAMPLES namespace. If you would like to see other methods of working with domains, see Alternatives for Creating an NLP Environment.
    

    1. Open the Management Portal for your instance in your browser, using the link and credentials from the launcher above.
    2. Verify that you are in the SAMPLES namespace (select Switch if necessary) and then navigate to the Domain Architect page by selecting Analytics > Text Analytics > Domain Architect.
    3. Click the New button to define a domain and fill the pop-up window with the following domain values:
       • Domain name: MyTest
       • Definition class name: Samples.MyTest
       • Allow Custom Updates: leave cleared
    4. Click the Finish button to create the domain. This displays the Model Elements selection screen.
    5. In the Model Elements pane, select Data Locations.
    6. Select Add data from table and use the drop-down menus to select the following options. You may need to zoom the browser view out if you cannot see the drop-down button.
       
       • Name: Table_1
       • Schema: Aviation
       • Table Name: Event
       • ID Field: ID
       • Data Field: NarrativeFull
       Leave other fields set to their default values.
    7. Click Save to save your changes.
       Note:

       The Domain's name in the navigation menu is followed by an asterisk (*) if there are changes to the current domain definition that have not been saved.
    8. Click Compile to compile the domain. Once the compilation has finished, click OK.
       
    9. Build the NLP indices for the data sources by pressing the Build button. When the message Successfully built your domain appears, click OK. This may take a few minutes.
       
  • Explore the data

    Once the domain has been created and compiled, you are ready to perform text analysis. In this section, we will do a quick summary of the most relevant tools. Note that not all of the features available in InterSystems IRIS will be mentioned here. To learn more, please visit the What's next? section below.

    
    

    1. On the Domain Architect page, select the Tools tab on the right side of the screen, then click Domain Explorer.
    2. In the Top Concept table, click pilot. You will see that the Sources section shows everywhere the concept pilot is mentioned.
       
       Note:

       The numbers in the table represents the quantity of occurrences and sources containing that concept, respectively. For example, the concept pilot appears 6195 times in 1084 sources.
    3. Select dominance tab in the Top Concepts table. Note how the table’s name changes to Dominant Concept. The dominance tab displays the dominant concepts in descending order by dominance calculation.
    4. The Similar Entities table shows the selected concept and any other concepts that contain that that concept (for example, pilot and private pilot). Click the private pilot concept and note how the Sources section updates.
    5. Find the Related Concepts table, which shows other concepts that are related to the selected concept. We can use this to understand concept relationships that we may not have been able to otherwise identify, such as airplane and specimens. Select the passenger concept, and note that the Source section now highlights that concept.
    6. Select the proximity tab in the Related Concept table. Note that the pilot concept has the highest proximity score. This means that the concepts pilot and passenger appear in the same line or paragraph most often, at 9635 times.
    7. Type clouds in the search box located at the upper left. Then click Explore! to search your concepts.
    8. Select the Paths tab.
       Note:

       The Paths tab lists all of the paths that contain the selected concept highlighted to show NLP indexing using colors:

       • Orange: Selected concept
         
       • Blue: Other concepts
       • Light blue: Path-relevant concepts (commonly pronouns)
       • White: Relations, or related concepts
       • Red: Negation
    9. Click the eye icon () to the right of the first line. The complete text of the source where the concept appears will be displayed.
    10. Click indexing. The text will be displayed using the NLP indexing colors.
  • Apply the skiplist filter

    You can apply the skiplist filter to exclude undesired concepts, including general concepts (such as airplane or fly) or concepts that are not relevant to your use of the data (such as a list of cities), without affecting the NLP indexing.
    

    1. If the domain Samples.MyTest is closed, reopen it. In the Domain Architect click Open and select User > MyTest.
    2. In the Model Elements pane, select Skiplists. 
    3. Click Add skiplist in the Details tab on the right side of the screen.
    4. Type MyFirstSkiplist in the Name field.
    5. Add the following to the Entries list:
       
       • pilot
       • student pilot
       • co-pilot
       • passenger
       • instructor
       • flight instructor
       • certified flight instructor
       Note:

       The concepts must be listed one per line. Entries are not case-sensitive.
    6. Save and compile the domain. Note that you do not need to build the domain to add, modify, or remove skiplists.
    7. Go back to the Domain Explorer by clicking Tools > Domain Explorer.
    8. Click the skiplist icon   in the upper-right corner.
    9. Select MyFirstSkiplist and click Apply. Note how the data from the Top Concepts table changes.
    10. Click the skiplist icon, clear MyFirstSkiplist, and click Apply to remove the filter.
  • Summary

    In this example, you have seen how the NLP semantic analysis engine in InterSystems products analyzes unstructured data written as text. Using these techniques, you can find patterns and relationships between concepts, even if you did not know that they existed.
• What's Next?

  What's Next?

  • Learn how to handle unstructured texts with the related tool InterSystems SQL Search (documentation, 40m).
    
  • See the full InterSystems IRIS Natural Language Processing (NLP) Guide (documentation, 9h30m).

  • Share Your Feedback!