pkg

SEARCH

1. Goals

   i.   Provide relevant information

   ii.  Provide a consistently fast response

   iii. Make responses consistent between local and remote search

   iv.  Provide the user with a good interface to the information

   v.   Allow seamless recovery when search fails

   vi.  Ensure the index is (almost) always in a consistent state

2. Approach

   From a high level, there are two components to search: the 

   indexer, which maintains the information needed for search; the 

   query engine, which actually performs a search of the information 

   provided. The indexer is responsible for creating and updating the 

   indexes and ensuring they're always in a consistent state. It does this 

   by maintaining a set of inverted indexes as text files (details of which

   can be found in the comments at the top of indexer.py). On the server 

   side, it's hooked into the publishing code so that the index is updated 

   each  time a package is published. If indexing is already happening when 

   packages are published, they're queued and another update to the indexes 

   happens once the current run is finished. On the client side, it's 

   hooked into the install, image-update, and uninstall code so that each 

   of those actions are reflected in the index.

   The query engine is responsible for processing the text from the user, 

   searching for that token in its information, and giving the client code 

   the information needed for a reasonable response to the user. It must 

   ensure that the information it uses is in a consistent state. On the 

   server, an engine is created during the server initialization. It reads 

   in the files it needs and stores the data internally. When the server gets

   a search request from a client, it hands the search token to the query

   engine. The query engine ensures that it has the most recent information

   (locking and rereading the files from disk if necessary) and then searches

   for the token in its dictionaries. On the client, the process is the same

   except that the indexes are read from disk each time instead of being stored

   because a new instance of pkg is started for each search.

3. Details

   Search reserves the $ROOT/index directory for its use on both the client

   and the server. It also creates a TMP directory inside index which it stores

   indexes in until it's ready to migrate them to the the proper directory.

   indexer.py contains detailed information about the files used to store the

   index and their formats. 

   3.1 Locking

       The indexes use a version locking protocol. The requirements for the

       protocol are: 

		the writer never blocks on readers

		any number of readers are allowed

		readers must always have consistent data regardless the

			writer's actions

       To implement these features, several conventions must be observed. The

       writer is responsible for updating these files in another location,

       then moving them on top of existing files so that from a reader's

       perspective, file updates are always atomic. Each file in the index has

       a version in the first line. The writer is responsible for ensuring that

       each time it updates the index, the files all have the same version

       number and that version number has not been previously used. The writer

       is not responsible for moving multiple files atomically, but it should

       make an effort to have files in $ROOT/index be out of sync for as short

       a time as is possible.

       The readers are responsible for ensuring that the files their reading 

       the indexes from are a consistent set (have identical version 

       numbers). consistent_open in search_storage takes care of this

       functionality.