http://docs.openlinksw.com/virtuoso/freetext.html Free Text Search OpenLink Virtuoso Universal Server: Documentation virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com 2008-09-09T11:44:52Z OpenLink Software Documentation Team http://docs.openlinksw.com/virtuoso/../images/misc/logo.jpg http://docs.openlinksw.com/virtuoso/txtidxquickstart.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Basic Concepts 2008-09-09T11:44:52Z Basic Concepts A text index is created with the CREATE TEXT INDEX statement. This creates a number of stored procedures and triggers which will transparently manage the text index. A text index is dropped by dropping the generated words table, called <table>_<column>_WORDS, where <table> and <column> are the table and column over which the index is made. The contains predicate is a normal SQL predicate and can be used together with other predicates in the where clause. Contains may however not figure inside an OR or NOT. Hence: is OK but http://docs.openlinksw.com/virtuoso/creatingtxtidxs.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Creating Free Text Indexes 2008-09-09T11:44:52Z Creating Free Text Indexes http://docs.openlinksw.com/virtuoso/queryingftcols.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Querying Free Text Indexes 2008-09-09T11:44:52Z Querying Free Text Indexes http://docs.openlinksw.com/virtuoso/txttrig.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Text Triggers 2008-09-09T11:44:52Z Text Triggers The text trigger mechanism allows implementing a broad range of content tracking functionality. The idea is storing free text or XPATH queries in association to a text indexed column of a table. When the content of the table changes through inserts or updates, the new data is matched against a base of stored queries and hits are marked into a separate table. The data being tracked may either be plain text or XML. In the event of XML, both free text and XPATH queries can be stored. The benefit of the text trigger system as opposed to other forms of periodic content tracking is that the incoming data itself indexes a base of stored queries instead of a base of stored queries repeatedly indexing the database. This means that only the changes are compared to the stored queries and that queries that could not even in principle match will not be tried. This results in a qualitatively better performance and scalability than repeatedly running a batch of queries over updated data and thus makes possible personalized information filtering applications that would be impractical with other approaches. http://docs.openlinksw.com/virtuoso/tablesandinternals.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Generated Tables and Internals 2008-09-09T11:44:52Z Generated Tables and Internals http://docs.openlinksw.com/virtuoso/droptxtindex.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Removing A Text Index 2008-09-09T11:44:52Z Removing A Text Index A text index is dropped by dropping the words table with DROP TABLE. This will drop all triggers, procedures and auxiliary tables. The words table is in the qualifier and owner of the indexed table and is named <table>_<column>_WORDS. http://docs.openlinksw.com/virtuoso/droptxttrig.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Removing A Text Trigger 2008-09-09T11:44:52Z Removing A Text Trigger Used to drop text trigger definition on text indexed table. The operation also drop all tables created by create text trigger statement. Syntax: Or using the stored procedure: http://docs.openlinksw.com/virtuoso/ftinternationalization.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Internationalization & Unicode 2008-09-09T11:44:52Z Internationalization & Unicode The text being indexed and the text query expression may both be wide strings. The word boundaries used to cut the text in words in both queries and index maintenance may depend on a language declared for the text index. The default language has white space and punctuation as word delimiters and will recognize Unicode ideographic characters as self standing. A single non-ideographic character will always be considered noise and not indexed. Non-ASCII Unicode values are converted to UTF8 before being stored into the word table as narrow strings. Narrow 8 bit strings are stored in the words table as is. http://docs.openlinksw.com/virtuoso/ftperformance.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Performance 2008-09-09T11:44:52Z Performance For indexing large volumes it is critical to run the indexing process over large volumes of data. This is accomplished by using the batch update mode. This is activated with the VT_BATCH_UPDATE procedure. When this mode is on the index will be updated after a settable interval, doing all the updates accumulated since the last batch in a single go. The size of a single batch is configurable in the virtuoso.ini file using the FreeTextBatchSize option. This is the number of bytes of text which will be processed as one sweep over the index. A value of 1MB is often suitable. Even All changes are processed in a batch at the set interval. If there are more characters of text to index / unindex than the batch size, the operation is broken into several transactions, each processing about FreeTextBatchSize bytes worth of text. This improves concurrency and cuts down on locking. The command http://docs.openlinksw.com/virtuoso/fttfuncs.html virtuoso.docs@openlinksw.com virtuoso.docs@openlinksw.com Free Text Functions 2008-09-09T11:44:52Z Free Text Functions Text index log table. In case of batch update mode this table is used to store a log of actions over text indexed table. Text index batch procedure. This procedure log, process and stores already filled _vt_batch, caused storing of words in index table. Text trigger hook function. This function, for existing text trigger definition, applies the filtering queries (if defined) to the _strings, and if the _vt_batch matches any of them then add new record in HITS table (see Text trigger)