Page source:

<h1>Advanced Programming in WubHub</h1>

<p>
This page describes how to program new functions in WubHub.
Read the introductory
<a href="/webl/WubHub_DoIt?cmdline=site.help">help</a>
page first, and the introduction to <a href="/webl/WubHub_DoIt?cmdline=site.programming">programming</a>.
</p>


<h2>Working with Lucene</h2>

<p>Simple functions have been created to let you work with Lucene, both for search and local search.  Generally, the pattern is:

<ul>
   <li> You get a lucene index for reading -- <i>Wub_ReadLuceneIndex(indexName)</i> -- or writing -- <i>Wub_NewLuceneIndex(indexName, create)</i> </li>
   <li> You create a document, add fields to it, and then add the document to the index --  <i>Wub_NewLuceneDocument()</i>, <i>Wub_AddLuceneField(doc, name, content, store, analyze)</i>, <i>Wub_AddLuceneLatLongField(doc, lat, lng)</i>, <i>Wub_AddLuceneDocument(writer, doc)</i></li>
   <li>  You read the index for search or local search -- <i>Wub_LuceneSearch(reader, defaultField, query, hitsPerPage)</i>, <i>Wub_LocalSearch(reader, lat, lon, miles defaultField, query, hitsPerPage, calculateDistances, sortByDistances)</i></li>
   <li> You optimize (if writing) and then close the index -- <i>Wub_OptimizeLucene(writer), <i>Wub_CloseLuceneIndex(writerOrReader)</i></li>
   <li> A few miscellaneous other functions exist -- <i>Wub_DeleteLuceneDocuments(writer, term, doc)</i>,  <i>Wub_UpdateLuceneDocument(writer, term, doc)</i>, <i>Wub_NewLuceneTerm(fieldName, searchStr)</i> </li>
</ul>

Here is an example of usage:

<h4>Writing example</h4>
<pre>/* Adds a person object to the search index  */
addPerson = fun(person)
   var ok = true;

   // create is false, add to existing index.  "standard" analysis (no stemming). For stemming: "kstem" and "english" (porter)
   var writer = Wub_NewLuceneIndex("resumes", false, "standard"); 
   var doc = Wub_NewLuceneDocument();

  // Add fields to doc
   ok = ok and Wub_AddLuceneField(doc, "id", person["id"], true, true, 0); // 0 = no special boost
   ok = ok and Wub_AddLuceneField(doc, "name", person["name"], true, true, 0);
   ok = ok and Wub_AddLuceneField(doc, "headline", (person["headline"] ? ""), true, true, 0);
   ok = ok and Wub_AddLuceneField(doc, "industry", (person["industry"]?""), true, true, 0);
   ok = ok and Wub_AddLuceneField(doc, "salary", (ToString(person["salary"])?""), true, true, 0);

   // Add coordinates for local search
   ok = ok and Wub_AddLuceneLatLongField(doc, ToReal(person["lat"]), ToReal(person["lng"]));

   ok = ok and Wub_AddLuceneDocument(writer, doc);
   ok = ok and Wub_OptimizeLucene(writer);
   ok = ok and Wub_CloseLuceneIndex(writer);
end;

</pre>
</p>

<h4>Searching Example</h4>
<pre>   var reader = Wub_ReadLuceneIndex("resumes");
   var docs = Wub_LuceneSearch(reader, "skills", queryString, 20, "standard"); // use same analyzer as was indexed with
   Wub_CloseLuceneIndex(reader);

   var i = 0;
   var sz = (docs.size() ? 0);

   while (i < sz) do
      var doc = docs.get(i).getDoc();
      var explaination = docs.get(i).getExplanation(); // explains score

      var d = [. .];
      d["id"] := doc.getField("id").stringValue();
      d["name"] := doc.getField("name").stringValue();
      d["headline"] := doc.getField("headline").stringValue();
      d["industry"] := doc.getField("industry").stringValue();
      d["salary"] := doc.getField("salary").stringValue();
      d["explanation"] := explanation;

      res = res + [d];
      i = i + 1
   end;
   res;

</pre>
</p>

<h2>Working with Tries</h2>
<p>Simple functions have been created to let you work with Trie data structure:
<ul>
   <li> <tt>addToTrie(trie, term, dataType, data)</tt>: Returns a trie that adds a new <tt>term</tt> to an existing <tt>trie</tt>, with dataType=[data] set at the terminal value.  If the same term is added more than once, each of the data elements are added to the dateType list.</li>
   <li> <tt>lookupInTrie(trie, term, dataType)</tt>: Returns the list of data values associated with dataType for a term, or nil if term is not found.</li>
</ul>
The empty trie is [. .].  Here is an example of usage:
<pre>
var trie = [. .];   // initial trie to empty

trie = WubCall("addToTrie", [ trie, "ab", "data", "1"] );
trie = WubCall("addToTrie", [ trie, "ab", "data", "2"] );
trie = WubCall("addToTrie", [ trie, "a", "data", "3"] );
trie = WubCall("addToTrie", [ trie, "def", "data", "4"] );

// Returns some values;
WubCall("lookupInTrie", [ trie, "ab", "data"] );    // returns [1, 2]
WubCall("lookupInTrie", [ trie, "def", "data"] );   // returns [4]
WubCall("lookupInTrie", [ trie, "xyz", "data"] );   // returns nil
</pre>
</p>

<h2>Working with MongoDB</h2>

To work with the MongoDB database, follow the following steps:

<UL>

<li> Get a reference to a named database.  This command either creates a new one or accesses an existing one.<p>
<code>
var db = Wub_GetDB("humanResources");
</code>
</li>

<li> Get a reference to a collection (like a table).<p>
<code>
var coll = Wub_GetCollection(db, "employees");
</code>
</li>

<li> Create a DB Object (row) to insert into the collection.<p>
<code>
var dbobj = Wub_NewDBObject([. name="Adam Cheyer", salary=100000.00, age=45 .]);
<br>
Wub_InsertDB(coll, dbobj);
</code>
</li>


<li> To query the database, create a query object and call the QueryDB function:<p>
<code>
var dbobj = Wub_NewDBObject([. name="Adam Cheyer" .]);
<br>
var res = Wub_QueryDB(coll, dbobj, 20);  // 20 = max results
</code>
</li>

<li> Query objects can search fields using comparative operators ($all, $gt, $gte, $in, $lt, $lte, $ne, $nin):<p>
<code>
var dbobj = Wub_NewDBObject([. "age"=[. "$gt"=40 .] .]); 
<br>
var res = Wub_QueryDB(coll, dbobj, 20);  
</code>
</li>

<li> You can update records using a search object and an update object with update operators ($set, $unset, $inc, $rename).
If the Upsert parameter is true, it will insert an object if a replace doesn't occur.  If Multi is true, it will make
the update on all changes, otherwise just on the first:<p>
<code>
var upobj = Wub_NewDBObject([. "$set" = [. "age"=46 .] .]); 
<br>
var res = Wub_UpdateDB(coll, dbobj, upobj, false, false);  
</code>
</li>
  
<li> You can drop an entire database:<p>
<code>
var upobj = Wub_DropDB("employees"); 
</code>
</li>  
  
<li> Or remove individual records matching a query:<p>
<code>
   Wub_RemoveDB(collection, queryObj); 
</code>
</li>    
  
<li>You can get the number of records in a collection:<p>
<code>
   coll.count(); 
</code>
</li>    
    
</UL>



<h2>More Information</h2>
<ul>
   <li><a href="WubHub_DoIt?cmdline=internalapi">WubHub Internal API (cmd=internalapi)</a>)</li>
   <li><a target="_blank" href="http://www.hpl.hp.com/downloads/crl/webl/library.html">WebL Homepage</a></li>
   <li><a target="_blank" href="http://www.hpl.hp.com/techreports/Compaq-DEC/SRC-TN-1997-029.pdf">WebL Overview Paper</a></li>
   <li><a target="_blank" href="http://www.hpl.hp.com/downloads/crl/webl/webl.pdf">WebL Documentation</a> <a target="_blank" href="http://www.hpl.hp.com/downloads/crl/webl/htmldocu/WebL.html">[HTML]</a> <a target="_blank" href="http://www.wubhub.com:4110/static/webl.pdf">[PDF]</a></li>
</ul>