Archiving Protocol

• Last updated: September 29, 2008.
• Author: Heather Trumbower, Robert Kuhn, and Brooke Rhead
• List of Archived Databases

Procedures

Determine connections from other assemblies

• Chain/Nets, Conservation, hgGene, {Human,Fly,Worm} Proteins and liftOver all have potential connections to as assembly being archived. To get the list of assemblies involved, run

    findDbPartners.csh db
  

Set up organism database on hgarchive1

• Request database on hgarchive1. Push all tables from RR (or hgwbeta). It's a good idea to compare update times for all machines to see if any RR machines have tables with strange update times (then, of course, do not push from *that* RR machine). The checkSync.csh script will compare table update times for all tables in a database between two machines. It is being run on a cron job for a db a day, so the results are available HERE.

  Remember to turn off Genbank updates first if pushing in the afternoon. Updates start ~4:30 p.m.

• Turn off Genbank updates on hgwbeta and the RR. Do NOT do this while they are running!:
  • cd to hgwdev:~/kent/src/hg/makeDb/genbank/etc/
  • comment out the database name in hgwbeta.dbs and rr.dbs
  • commit the changes
  • ssh hgwbeta
  • cd /genbank/etc
  • cvs up -dP

• Check that all tables exist on hgarchive1. If you are not able to ssh to hgarchive1, ask cluster-admin for an account there, or ssh as qateam. Note that the hgsql script does not exist on hgarchive1. In order to see the mysql tables (not just the databases), run mysql as user 'hgcat':

  [rhead@hgarchive1 /]$ mysql --user=hgcat --password -A

  The hgcat password should be in your .hg.conf file (in your home directory). The path to the mysql tables in the filesystem is hgarchive1:/var/lib/mysql/ .
• Compare tableName values in the trackDb table between hgarchive1 and the RR. ('Base Position' and 'Short Match' do not have entries in the trackDb table.)
• Push proteins databases as necessary. Include both proteinsYYMMDD and spYYMMDD. Drop from RR if no longer used by any assemblies (see gdbPdb). Ask for reset of the pointer on hgarchive for both proteome and uniProt.

Create an hgcentral database on hgarchive1

• A new copy of hgcentral will be unique to the newly archived database and called 'hgcentarch[DbName]'. Update hgcentarch[DbName], using hgcentralbeta (or hgcentral on genome-centdb) as the baseline. Include blatServers, clade, dbDb, defaultDb, gdbPdb, genomeClade, liftOverChain, sessionDb, and userDb. Get a copy for loading as follows:

On hgwbeta:
hgsqldump --all -d -c hgcentralbeta blatServers sessionDb userDb > hgcentral.sql
hgsqldump --all -c hgcentralbeta clade dbDb dbDbArch defaultDb gdbPdb genomeClade liftOverChain >> hgcentral.sql

(if using hgcentral, add '-h genome-centdb' after the '-c' above)
• Push and prune hgcentral table. The blatServers, sessionDb, and userDb tables should exist on hgcentarch[DbName], but should be empty. (No BLAT servers are available for archived dbs.) The remaining tables should be pruned so that only the relevant information is retained. Notes from archiving rheMac1 (and modified during archiving of mm6 and others in Aug 2008) follow.

hgcentral setup Refer to the table for a summary of the various combinations from the archiving project of 2008-08-25.
Case 1	Case 2	Case 3
No connections of any kind left to the database from remaining dbs on RR. E.g., ce1, rn2. See details for discussion and MySQL commands.	Only liftOver connections remain to the database from remaining dbs on RR. E.g., hg15, bosTau1. See details for discussion and MySQL commands.	Connections remain to the database from remaining dbs on RR, including net/chains, multiz, otherOrgs in hgGene, Human (or Fly or Worm) Proteins. E.g., mm5, mm6, danRer2. See details for discussion and MySQL commands.

Push the current CGIs

• Each assembly has a CGI directory on hgarchive1, e.g., hgarchive1:/usr/local/apache/cgi-bin-mm2 . Ask for a new directory to be created for the assembly being archived, and then for a push of the current CGIs to that directory on hgarchive1. It may be helpful to find the most recent CGI push request and edit it. NOTE: hgarchive1 is MACHTYPE=i386, but hgwdev, hgwbeta, and hgw1-8 are all MACHTYPE=x86_64. Therefore, the CGIs need to be pushed from titan at titan:/scratch/releaseBuild/cgi-bin-32 , not from hgwbeta or the RR machines. We are currently still building 32-bit CGIs on titan for mirror sites.
• Link multiple cgi-bin-db directories. If more than one assembly is being archived, you can make a single db for all of them (e.g, cgi-bin-2008-08) and hard-link all the binaries and ancillary files and directories (all.joiner, hgGeneData, etc) from the various cgi-bin-db directories. The only file that needs to be unique to each directory is hg.conf
• Request push of cgis

  sample push-request:

  
  pushers:
  
  a four-part request:
  
  1.  could you please create the following directories on hgarchive1:
  
          /usr/local/apache/cgi-bin-2008-08
          /usr/local/apache/cgi-bin-mm5
          /usr/local/apache/cgi-bin-mm6
  
  and push the 32-bit cgis from 
  
          titan:/scratch/releaseBuild/cgi-bin-32
  
  to cgi-bin-2008?  Then make hardlinks in the other two directories
  so all the files EXCEPT hg.conf are the same file.
  
  please push these cgis only:
          
  cartDump
  cartReset
  das
  hgConvert
  hgCustom
  hgGateway
  hgGene
  hgGenome
  hgLiftOver
  hgNear
  hgPcr
  hgSession
  hgTables
  hgTrackUi
  hgTracks
  hgVisiGene
  hgc
  pbGateway
  pbGlobal
  pbTracks
  phyloGif
  
  and 
          
  2.  also push this directory and make the hardlinks:
   
    titan:/scratch/releaseBuild/cgi-bin-32/loader/* 
  
    to:
          /usr/local/apache/cgi-bin-2008-08
  
  3. and these configuration files to the same place 
  and make the hardlinks:
  
   /usr/local/apache/cgi-bin/all.joiner
   /usr/local/apache/cgi-bin/galaAvail.tab
   /usr/local/apache/cgi-bin/hgNearData/*
   /usr/local/apache/cgi-bin/hgGeneData/*
   /usr/local/apache/cgi-bin/hgcData/*
   /usr/local/apache/cgi-bin/hgCgiData/*
   /usr/local/apache/cgi-bin/visiGeneData/*
   /usr/local/apache/htdocs/js/*
   
          from hgwbeta 
  
  4. also, please grab the hg.conf from the cgi-bin-hg15 directory 
  on hgarchive1 and copy it to the two new dirs.  This should be a 
  real file in each place NOT HARDLINKS.
  
          /usr/local/apache/cgi-bin-hg15/hg.conf
   to
          /usr/local/apache/cgi-bin-mm5
          /usr/local/apache/cgi-bin-mm6
  
  reason: archiving
  
  

Set up virtual host and htdocs

• Each assembly has a CGI directory on hgarchive1. Ask Cluster-Admin to set up a virtual host on genome-archive for the newly archived assembly. Also ask them to add genome-db to the DNS. Verify that the admins have edited the file hgarchive1:/usr/local/apache/conf/httpd.conf to include a block containing the new database. For database 'db' it will look like http://genome-db.soe.ucsc.edu/ and point to /usr/local/apache/cgi-bin-db .
• Check that hgarchive1:/usr/local/apache/cgi-bin-db/hg.conf points to hgcentarch[DbName] on localhost. Look for two lines that look like this (this is for the cgi-bin-rn1/hg.conf file):

  central.db=hgcentarchRn1 
  central.host=localhost

  If this file does not exist for the newly archived assembly, it can be copied from one of the other assemblies on hgarchive1 and edited.
• Note that no ".htaccess" file needs to be present in the cgi-bin-db directory. (This has previously been found to prevent the new server from working.)
• Copy all of htdocs from RR to htdocs-YYYY-MM. This will allow older archived assemblies to keep the docs they have and the newly archived assemblies get current docs. If archiving more than one assembly, ask the admins to set up DocumentRoot in conf/httpd.conf to point to it from the new cgis from each new assembly. NOTE: See the README in htdocs-2008-08 for notes about several files that do not work porperly like this (they are hard-coded in the custom-track cgi). These files were symlinked from the main htdocs directory. Erich says the following about getting htdocs to use included properly:

    Just fixed it.  It appears there were some allow/deny directives
    specific to the 'htdocs' directory that had to be explicitly mentioned
    for the 'htdocs-2008-08' directory as well for includes to be honored.
  

  Note that there will be one index file for all assemblies, so the file, htdocs/index.html, should be updated to reflect all archived assemblies. Then set up symlink in the new htdocs-YYYY-MM dir to point to it:

    htdocs-YYYY-MM>ln -s ../htdocs/index.html index.html
  

Push any related dbs/tables

• Consider: are there any needed tables from other organism databases? If the assembly being archived contains nets and chains to other organisms, the 'chromInfo' table from the other orgs need to be pushed to hgarchive1. Also, if there is a 'Fly Proteins' or 'Human Proteins' track, there are tables that are referenced in the corresponding fly or human databases. To check for these tracks:

   [rhead@hgwdev ~]$ getAssemblies.csh blastHg%
   [rhead@hgwdev ~]$ getAssemblies.csh blastDm% 
  

  or see the output from findDbPartners.csh. The 'blastHg%' table references the human 'blastKGRef%' and 'blastKGPep%' tables. The 'blastFb%' table references the fly 'blastFBRef%' and 'blastFBPep%' tables. If needed, these should be pushed from their databases to hgarchive1.
• Determine whether the 'chromInfo' table for any other assembly needs to be pushed to hgarchive1. The 'chromInfo' table is needed if there are nets and chains to other organisms. To locate them:

    hgarchive1:>mysql -A --user=hgcat db
    SHOW TABLES LIKE "net%";
  

• Consider: are there any needed tables from the hgFixed or go databases? What about any needed protein databases, such as for human, mouse, or rat Known Genes tracks? If needed, ask for a push of these auxilliary database tables hgarchive1.
• Push download content from hgdownload:/usr/local/apache/htdocs/goldenPath/db to hgarchive1:/usr/local/apache/htdocs/goldenPath/db/ .
• Add links to index and download pages on genome-archive (Donna usually does this).
• Ask Donna to write static content for RR (includes downloads.html for hgdownload; indexNews.html and goldenPath/newsarch.html for beta and RR). These should not be pushed until the archived database is working and tested on hgarchive1.
• Add archived assembly information to the top of hgwdev:~/browser/qa/archiveCgis.html (that's the "Archived Databases" link at the top of this page) and check in the change. This is our record of CGI versions for archived databases.

Test newly archived assembly on hgarchive1

• Try clicking through to the details page for each track, and compare to the non-archived version. Look for links that could break on the archive server, such as links to other organisms. Currently there is a problem with a link on the Human Proteins track (which should be fixed in a later cgi version).

• Check nets and chains -- there should be data on hgc, but no active links to any other assembly, though if the other assembly is also archived, these may be activated on archive by adding a row in dbDb with active = 0 and a row in dbDbArch. "View alignments" should be available and working. If "View alignments" is not available, check to see that the 'chromInfo' table for net/chain organism has been pushed to hgarchive1.

• Genomic alignments for all other tracks (such as mRNA alignment tracks) should work, although they will quit working as genbank is updated and the files are replaced.

  Note from Mark: An error will occur any time the browser needs to access the FASTA files for the genbank mRNAs, ESTs, or RefSeq genes. This happens with:

  • the alignment display in hgc
  • "Color track by bases" == mRNA bases, not dense and zoomed into to the base level
  • "Color track by bases" == different mRNA bases, not dense, and at any zoom level

  Turning off "Color track by bases" by default should keep the hgTracks display functioning. To do this, change the setting field in trackDb to use "baseColorDefault none".

• Look for any missing .html or image files.

• Check to see that "get DNA" still works.

• hgConvert should give the message "Sorry, no conversions available from this assembly."

Announce the archive

• Find the size of the archived database so that the mirrors can be informed of the size of the drop. Use the getDbSize.csh script (be sure to use the "index" option, like so):

      [rhead@hgwdev ~]$ getDbSize.csh danRer1 index
  
            db Gbytes index
       ======= ====== =====
       danRer1    4.6   2.1
  

  Alternatively, use dbSnoop:

       hgwbeta> cd /tmp
       hgwbeta> dbSnoop -unsplit $db $db.dbSnoop
       hgwbeta> head $db.dbSnoop
  

• Determine whether the 'chromInfo' table needs to be retained on hgwbeta and RR (and announce this to the mirrors the mirrors, if they are not going to drop /gbdb info for the archived organism). The 'chromInfo' table is needed if there are nets and chains from other organisms pointing to the archived assembly. To locate them:

      [rhead@hgwdev danRer1]$ cd /usr/local/apache/htdocs/goldenPath/
      [rhead@hgwdev goldenPath]$ ls */vsDanRer1
  

  or:

      getAssemblies.csh netDb
  

  (also included in output of findDbParterns.csh if you ran it above).
• Notify Genome-Mirror and ask Donna to notify Genome-Announce of the impending change and ask if it will be a problem, especially the lack of a BLAT server. Example of a Genome-Announce email. Example of a Genome-Mirror email.

Drop assembly from hgwbeta and test

• Edit hgcentral. Remove reference to the archived database from blatServers. Keep the reference to the archived database in dbDb, but set active = 0 (unless there are no more back-refs from any other tracks on other assemblies. In that case, the dbDb rows can be dropped after being copied to dbDbArch).

    DELETE FROM blatServers where db = 'rheMac1';
    UPDATE dbDb SET active = 0 WHERE name = 'rheMac1';
  

• Copy rows to dbDbArch. Moving rows for seven dbs:

   rm -f dbDbArch.rows
   foreach db (hg15 mm5 mm6 ce1 bosTau1 danRer2 rn2)
     hgsql -N -e "SELECT * FROM dbDb WHERE name = '$db'" hgcentraltest >>  dbDbArch.rows
   end
  

  load into hgcentralbeta

   hgsql -h hgwbeta -e "LOAD DATA LOCAL INFILE 'dbDbArch.rows' INTO TABLE dbDbArch" hgcentralbeta
  

• There is no need to change clade, defaultDb, sessionDb, or userDb. GenomeClade only needs editing if there are no other assemblies remaining of the same organism type. The gdbPdb table may need editing for some organisms. The entries in liftOverChain should be kept in order to keep nets/chains from other browsers that point to the now missing assembly from breaking. Also, they will make it possible for hgLiftOver and hgConvert to continue to work for the archived assembly (on the RR, not on hgarchive1).
• If nets and chains from other organisms are going to continue to point to the archived assembly, drop all tables from the assembly on hgwbeta EXCEPT for the 'chromInfo' table. Otherwise, drop the entire assembly from hgwbeta. To drop all tables except 'chromInfo', it is easiest to drop the entire database and then push 'chromInfo' from hgwdev to hgwbeta. To drop the entire assembly from hgwbeta:

    mysql> drop database rheMac1;
  

  Be very careful with this command!
• Test hgwbeta to look for errors caused by the missing assembly. Test nets and chains from other organisms that point to the missing assembly.

Drop assembly from RR and test

• This section may not be complete. Check that you do everything to RR that you did on beta to put them both to bed.
• Update dbDb on hgcentral (-h genome-centdb) to set active = 0 for assembly on RR.

• Check to see that hgLiftOver and hgConvert are working as expected on the RR.

• To make the assembly available for hgLiftOver and hgConvert, copy the the line from hgcentral.dbDb to hgcentral.dbDbArch and set active = 1 for dbDbArch. (All assemblies in dbDbArch should be set to active = 1.) Note that the liftOverChain entry in hgcentral needs to be kept for liftOver to work. As of CGI version 145, hgConvert has logic to keep from drawing active links to an assembly if active = 0 in hgcentral.dbDb.
• If not previously done, push static pages to RR.
• Check that static pages are correct on RR.
• Drop assembly from hgw1-8, in its entirety if there are no other net and chain tracks pointing to the archived assembly, or leaving 'chromInfo' if net and chain tracks from other assemblies will still use the table. NOTE: If asking for a drop of all tables from a database except the 'chromInfo' table, it is easier for the admins to drop the entire database first, and then push 'chromInfo' from hgwbeta.
• We no longer drop downloads from hgdownload. The archive server is pointing to hgdownload. In the past, we moved downloads to hgarchive and made them available only from there. Now we create the links on genome-archive/downloads.html but they point back to the hgdownload machine.

  If any assemblies are being stripped down to chromInfo only, have the admins set up the rsync so it no longer removes missing tables from the annotation link. Do this before dropping the dbs from the RR.

• Check to see that the database has been removed from the release log. This should happen automatically once the database entry in hgcentral.dbDb is inactive or removed.

After the archive is complete

• Be sure the admins know to liberate any blatServers that are no longer in use after the archive.

• Check to see if, now that this assembly is archived, any other assemblies that contain only the 'chromInfo' table can be dropped. Such a database can be dropped if no other nets and chains or Conservation tracks point to it. The list of databases that are archived, but still alive via the single chromInfo table is found in the trackDbGlobal output or in trackDb/makefile under ARCHIVED_DBS.
• Remove any dbDb entries in hgcentral that are no longer being used. A dbDb entry is still needed for a no-longer-existing assembly if:
  1. Any other organism's nets and chains are still pointing to it.
  2. Any other organism's Conservation tracks are still pointing to it.
  The row may be removed from dbDb even if:
  1. Another organism's hgGene.otherOrgs are still pointing to it (rename the dbBlastTab table in the other to DbBlastTab --- mmBlastTab to MmBlastTab -- to disable the functionality until the dbs are together again on hgarchive).
  2. The assembly is still wanted for hgLiftOver and hgConvert on the RR (as long as the row has been added to dbDbArch). The conservation track needs the dbDb entry in order to find the organism name for the hgc page (it uses it to write "Rhesus" rather than "rheMac1," for instance). Use getConservation.csh to determine the assemblies whose conservation tracks use a particular database.

• Check that the db was dropped from genome-mysql (this should happen automatically when the db is dropped from RR).
• Remove the listing of the assembly from DBS list in the trackDb/makefile. If present, also remove from MGC_DBS list. If the assembly is staying with only chromInfo table to support comparatives, add it to ARCHIVED_DBS list.
• If the assembly is removed from hgwdev, remove references to it in all.joiner, remembering that there are references to each asembly thorough out the document. Remove even the !db references