============================================================================================
Version 2.2.1 - 2014-02-13
============================================================================================
	Updates:
    	SnpEff - users can now upgrade the SnpEff tool version and genome builds. See UserGuide for more info
    	The bior.properties file has been updated to include a SnpEffCmd property which allows the SnpEffect tool to be upgraded easily. However, if users have added a bior.properties file to their home directory to override the default one in the bior_pipeline*/conf/ directory, this will need to be updated to include the new property. See the file in the bior_pipeline*/conf/ directory for how to specify the new property.
		bior_annotate: Added all columns for config file to the help output when using the --help or -h flag
  
  	New Commands:
  		Added bior_trim_spaces command to remove spaces around columns. This command skips any metadata header lines that begin with "##".
  			bior_trim_spaces was added by default within bior_vep, bior_snpeff, and bior_annotate commands. Spaces around columns could cause problems for tools that operate on numeric columns like POS, and would cause VEP to eat up memory at a rate of 2-3MB/sec.

	Bug Fixes:
  		Bug fix: bior_same_variant will no longer match alts when they are "." - this was a problem when there was no rsId specified in either the input or catalog file, and variants were matching on chrom, position already, and then matched incorrectly on the "." in the Id column.
  		Bug fix: bior_index_catalog - was throwing a NullPointerException when trying to create an index in the current directory.
  		Bug fix (internal): LineCounterPipe can now handle either Strings or History objects. This was causing a problem with the bior_create_catalog command. 



============================================================================================
Version 2.2.0 - 2013-11-27
============================================================================================

    bior_annotate updates:
        Resurrected single-threaded bior_annotate with metadata functionality. This will reduce the amount of memory used by bior_annotate.
        Single-threaded operation is now the default, with flag to call multi-threaded version
        Flag to output status (# lines in, out, failed, exit code)
        Improved exit code handling so it only returns 1 if the code did not run to completion. If some lines could not be processed, but command ran to completion, then errors on those lines are shown in log, with summary to stderr, but exit code = 0 

    Added LineCounterPipe to pipes project 


============================================================================================
Version 2.1.0 - 2013-09-11
============================================================================================

    Metadata is now included with all catalogs  one properties file to denote the data source name, version, build; one to specify column names, types, number of occurrences, and descriptions. 

    Metadata ##BIOR lines added for all commands when a new column is added (or one is modified). This required a significant change to the code base as all commands were affected, and the metadata handling was centralized into one location. This also affected many test cases as well that had to be updated to match the new command outputs. 

    New commands added:
        bior_bed_to_tjson - converts a bed file to a tab-delimited JSON file.
        bior_compress - compresses multiple lines into one. If values in a compressed column differ from one line to the next, those are combined into the same column, but separated by some delimiter.
        bior_create_catalog - creates a catalog from a given file that contains a JSON column. This performs a bgzip compression on the output file, and then runs tabix to create a positional index on that bgzip file.
        bior_create_catalog_props  creates the two properties files for a catalog. To do this, the catalog is crawled to calculate data types and whether there are multiple values within a field. The user can also point to a VCF file from which the catalog was derived in order to parse the correct data types and multiplicities.
        bior_create_config_for_tab_to_tjson - helps to create the config file that is used by the bior_tab_to_tjson command
        bior_tab_to_tjson  for each line in a tab-delimited file it creates a JSON object from this, and appends the JSON to the end of the line. This requires a config file that can be constructed using the bior_create_config_for_tab_to_tjson command.
        bior_tjson_to_vcf  takes lines in a tab-delimited JSON file and converts them to VCF-compatible lines with the data going into the INFO column. The metadata header is also modified to include ##INFO lines describing these fields added to the INFO column. 

    Commands renamed:
        bior_index renamed to bior_index_catalog to signify that it is indexing a catalog.
        bior_bed_to_json renamed to bior_bed_to_tjson
        bior_vcf_to_json renamed to bior_vcf_to_tjson 

    Reduced memory usage on SGE (Sun Grid Engine) for all commands
        Gathered all license information from tools and data sources 

    Created a "Bior 2.1.0 User Guide" in GoogleDocs so external viewers can read it and select reviewers can edit and update it.
        https://docs.google.com/document/d/17abGGHKJU6BizDLanI58enTRXmRh7yLGt69GurObjxo/edit?pli=1 

    Bug Fixes:
        bior_index_catalog now shows error if the path is a directory. Shows warning if the output index name does not have the same prefix as the catalog.
        bior_vcf_to_tjson - when processing files, if a given line contains a data type that is inconsistent with the ##INFO line in the header, it will just ignore that field now rather than crashing (example: data: "MQRankSum=..03" - the ##INFO line says the data type is Integer, but the value in the INFO column is messed up and then treated as a string)
        bior_annotate more gracefully handles blank or malformed lines in the config file. 

    RefData
        HTTP Downloader fixes - now handles symlinks and MD5sums
        New data model - Updated refdata_search command, download daemon 

    BioR is now officially using the support org (Remedy) to submit and track bugs and feature requests.
        http://helpdesk.mayo.edu/remedyessforms/remedyessincident.aspx?ext_sys=Bioinformatics+Systems+Unit&ext_event=BioR 



============================================================================================
Version 0.0.3 - 2013-06-20
============================================================================================

Changes:

    General:
        Logs only written now when --log flag used (instead of every time) 

    bior_vep:
        --all flag now outputs a valid JSON object (that starts with "{") that contains an array of values
        Resolved timeouts that were slowing down VEP processing and possibly causing some out of memory errors 

    bior_annotate
        Increased memory for bior_annotate command from 512M to 2G to reduce the number of out-of-memory errors
        In the future, we may need to have special handling of variants that are thousands of base-pairs long (which can still cause out-of-memory errors) 

    bior_bed_to_json
        This new script is now available
        Example: 

$ cat /data4/bsi/BIOR/example.bed | bior_bed_to_json 
#chrom	chromStart	chromEnd	name	score	strand	thickStart	thickEnd	itemRgb	blockCount	blockSizes	blockStarts	BED2JSON
chr22	1000	5000	cloneA	960	+	1000	5000	0	2	567,488,	0,3512	{"chrom":"chr22","_landmark":"22","chromStart":"1000","_minBP":1001,"chromEnd":"5000","_maxBP":5000,"name":"cloneA","score":"960","strand":"+","thickStart":"1000","thickEnd":"5000","itemRgb":"0","blockCount":"2","blockSizes":"567,488,","blockStarts":"+"}
chr22	2000	6000	cloneB	900	-	2000	6000	0	2	433,399,	0,3601	{"chrom":"chr22","_landmark":"22","chromStart":"2000","_minBP":2001,"chromEnd":"6000","_maxBP":6000,"name":"cloneB","score":"900","strand":"-","thickStart":"2000","thickEnd":"6000","itemRgb":"0","blockCount":"2","blockSizes":"433,399,","blockStarts":"-"}

============================================================================================
Version 0.0.3 - 2013-05-29
============================================================================================

Changes:

    bior_annotate command updated:
        New --config option allows a file to be used to limit the columns that are outputted, as well as avoids commands that do not need to be run if their output is not required.
        same_variant used instead of overlap for several catalogs (BGI, ESP, Hapmap, 1000Genomes), since overlap returned some large indels that were not the same variant
        Passes GOLD VCF file (a vcf file where several variants have been verified against known catalog values)
        4 HapMap columns added to output 

    bior_vep command fix:
        Fix to avoid indefinite hang when variant_effect_predictor returns no output (now times out after 10 sec on a bad line instead of ending) 

    bior_compress command updated:
        bior_compress has new --reverse and --escape options. (data can sometimes contain the delimiter character we use to separate values, so escaping it when it occurs in the original data) 

    bior_lookup command updated:
        bior_lookup is no longer case sensitive. Note that the default behavior has changed. You can now search for a key with any case. To make case-sensitive, just use the appropriate flag (-s) 

    SGE fix:
        Sun Grid Engine fix for bior toolkit commands (avoids out of memory exceptions and using too much memory) 

    $BIOR_CATALOG variable updated:
        $BIOR_CATALOG has been switched to the new directory (from /data4/bsi/refdata-new/catalogs/ to /data4/bsi/catalogs/bior/) 

    $USER_CATALOG variable added:
        There is a new $USER_CATALOG variable that points to the user-created catalogs. It is a good idea to use this instead of the hard-coded paths (especially since /data4 may move to /data5 soon) This should be used for all new catalogs that users (not the bior team) create 

============================================================================================
Version 0.0.3 - 2013-05-07
============================================================================================

    New bior_annotate
        New command that will replace the legacy TREAT annotation module. This is an early test version that has 56/60 of the final TREAT output columns. Documentation and validation against the GOLD file are not complete, but you can take it for a test-drive if you like. Please wait until the full production 2.0 release prior to replacing the legacy perl-based module. Be aware that SNPEff requires a lot of RAM and takes time to iinitialize prior to any data being processed. 

    New bior_compress
        New command that compresses multiple rows to a single row. This is built-in to the bior_annotate command, but can also be run independently after other toolkit commands such as bior_overlap, bior_lookup, and bior_same_variant that may cause multi-row output. 

    Updated bior_vep
        Updated command so that it no longer is required to be the 1ST command in your pipeline. 

    New dbSNP Catalog
        Added new clinvar catalog as a requirement for TREAT. See http://bsiweb.mayo.edu/dbsnp for details. 

    Updated OMIM Catalog
        OMIM separates out the Title, Comments, and Disorders fields into multiple continued fields. The original rationale behind this is not clear, but the usability of the Catalog has been enhanced so that the continued fields have been removed and their respective content has been merged. ** For example, the fields Disorders and Disorders, cont. have been merged to just be Disorders.
        For details, see: http://bsiweb.mayo.edu/omim 

    Updated ESP Catalog
        ESP reports allele frequencies as a percentage 0-100%. New fields have been added to the catalog that divide these percentages by 100 so that the frequencies are normalized along the same lines as HapMap and 1000 Genomes.
        For details, see: http://bsiweb.mayo.edu/esp 

    NEW Catalog space
        The current space for catalogs in the RCF environment is /data4/bsi/refdata-new/catalogs/v1 . The $BIOR_CATALOG environment variable also points to this space. In 2 weeks (May 21), this space will be going away. The new /data4/bsi/catalogs will take its place. We will update $BIOR_CATALOG at that time to point to the new space. We highly encourage users to utilize $BIOR_CATALOG instead of hard-coding the path to make the transition seamless.
        The /data4/bsi/catalogs/user space is open to the BioR user community for publishing their own catalogs. Its recommended that you mirror the folder structure and naming conventions used for existing BioR built catalogs for your own catalogs. This will help everyone from a usability standpoint. 


============================================================================================
Version 0.0.3 2013-03-13
============================================================================================

The following 8 catalogs are based on UCSC annotation tracks required by the TREAT workflow. These are now available in the RCF environment and are fully compatible with the bior_overlap command.

    Regulation Catalog:
        Track for regulatory regions from ORegAnno.
        http://bsiweb.mayo.edu/regulation 

    Conservation Catalog:
        Vertebrate Multi-z alignment and conservation.
        http://bsiweb.mayo.edu/conservation 

    Enhancer Catalog:
        Vista HMR-Conserved Non-coding Human Enhancers from LBNL.
        http://bsiweb.mayo.edu/enhancer 

    TFBS Catalog:
        HMR Conserved Transcription Factor Binding Sites.
        http://bsiweb.mayo.edu/tfbs 

    TSS Catalog:
        SwitchGear Genomics Transcription Start Sites.
        http://bsiweb.mayo.edu/tss 

    Blacklisted Region Catalog:
        Blacklisted Region  Browser extensible data.
        http://bsiweb.mayo.edu/blacklisted-region 

    Repeat Region Catalog:
        Repeat Masker .out data.
        http://bsiweb.mayo.edu/repeat-region 

    Uniqueness/Alignability Catalog:
        Duke excluded regions.
        http://bsiweb.mayo.edu/uniquenessalignability 


NOTE: Although UCSC has 8,344 annotation track files, only 8 have been documented and vetted by the BioR team. These 8 catalogs are part of TREAT and are fully supported by the BioR Team. The remaining 8,336 UCSC annotation tracks have also been published as catalogs and are available "as is" with no support or documentation at this time. These tracks will be available in the RCF environment at /data4/bsi/refdata-new/catalogs/beta/ucsc/hg19 by March 15 as the upload is still running.


============================================================================================
Version 0.0.3 - 2013-03-11
============================================================================================

The following catalog has been changed:

    BGI Catalog:
        The fields calculated_minor_allele_freq 

and calculated_major_allele_freq have been removed from the catalog as of March 8th. These fields were calculated by the BioR team as a convenience and are not in the original LuCAMP_200exomeFinal.maf.gz raw flat file. Thanks to some helpful feedback from Greg Dougherty, it was brought to our attention that these calculated values are not correct due to ambiguity in the exact meaning of values in the LuCAMP_200exomeFinal.genotype.gz file. We made the decision to pull these fields from the catalog. These changes are reflected in the rebuilt Catalog file and the Drupal page http://bsiweb.mayo.edu/bgi. Please use the fields estimated_minor_allele_freq and estimated_major_allele_freq for allele frequencies. These fields are carried over verbatim from the original source data.


============================================================================================
Version 0.0.3 - 2013-03-06
============================================================================================

The following catalogs are now available:

    OMIM genemap Catalog:
        This catalog is accessible with the bior_lookup command using the pre-built index for the MIM_Number field. Please see the Drupal documentation page http://bsiweb.mayo.edu/omim for more details. 

    HGNC Catalog:
        This catalog is accessible with the bior_lookup command using the pre-built indexes for either the Entrez_Gene_ID or UniProt_ID fields. Please see the Drupal documentation page http://bsiweb.mayo.edu/hgnc for more details. 

NOTE: Each of the above catalogs have numerous fields. Please let the team know at bior@mayo.edu if we are missing pre-built indexes for commonly used fields.


============================================================================================
Version 0.0.3 - 2013-02-22
============================================================================================

The BioR toolkit currently supports coordinate based search with the bior_overlap and bior_same_variant commands. We are now introducing a new search capability to the BioR toolkit that searches for annotation by unique identifiers. This ID based search is implemented with the following 2 new commands:

    bior_lookup
        The bior_lookup command searches for a given ID or string in a catalog. We currently have an index built for the following catalogs:
            dbSNP RS identifier for the dbSNP catalog [$BIOR_CATALOG/dbSNP/137/00-All_GRCh37.tsv.bgz]
            HGNC identifier for the NCBI Entrez Gene catalog [$BIOR_CATALOG/NCBIGene/GRCh37_p10/genes.tsv.bgz] 

    bior_index
        The bior_index command builds a new index on a catalog that can be used by bior_lookup for fast searches. This gives users the ability to create their own index if desired. 

The bior_lookup and bior_index commands are now available in the RCF environment. The Drupal page http://bsiweb.mayo.edu/lookup-and-indexing has more information on getting familiar with the commands. We appreciate any feedback you can provide. We would especially like to know of new indexes that we should built that would be useful to the BioR user community.

NOTE: Please logout of your RCF terminal and log back in to pick up the latest changes.


============================================================================================
Version 0.0.2 - 2013-02-21c
============================================================================================

A new 1000 Genomes catalog is available for release 20110521. This catalog is a direct "out of the box" translation of the data in the original source files. The calculated subpopulation frequencies are not available in this catalog, but are available in the older BioR 1.0 Legacy system (see http://bsiweb.mayo.edu/user-guide-bior-10-legacy ). The BioR team would like to build a full-featured catalog that does include the subpopulation frequencies and retire the BioR 1.0 Legacy System. The timing of this new catalog is still up in the air and should be discussed in a future Stakeholder meeting.

Please visit the drupal documentation page at http://bsiweb.mayo.edu/1000genomes for details.


============================================================================================
Version 0.0.2 - 2013-02-21b
============================================================================================

A new miRBase catalog is available for release 19. Please visit the drupal documentation page at http://bsiweb.mayo.edu/mirbase for details.


============================================================================================
Version 0.0.2 - 2013-02-21a
============================================================================================

Bug fixes

    Bug fix: The bior_overlap and bior_same_variant commands had an issue where a single variant with ZERO matches in a catalog would cause all subsequent variants to also show ZERO matches regardless of whether a match exists or not. See http://bsu-bugs/default.asp?1932 for details. 

    Bug fix: The bior_overlap and bior_same_variant commands did not correctly handle the input JSON column being a blank JSON document "{}". All subsequent variants after the variant with the blank JSON column show ZERO matches regardless of whether a match exists or not. See http://bsu-bugs/default.asp?1943 for details. 

The latest bug fixes are now available in the RCF environment.


============================================================================================
Version 0.0.2 - 2013-02-11
============================================================================================

The following catalog is now available:

    COSMIC v63 Catalog:
        Drupal documentation page http://bsiweb.mayo.edu/cosmic 


============================================================================================
Version 0.0.2 - 2013-02-06
============================================================================================

The following catalogs are now available:

    Hapmap Allele Frequencies Catalog:
        Drupal documentation page http://bsiweb.mayo.edu/hapmap
        Please take special note on the LiftOver process section and how a very small percentage of variants had difficulty lifting over to GRCh37. 

    BGI LUCAMP Allele Frequencies Catalog:
        Drupal documentation page http://bsiweb.mayo.edu/bgi 


============================================================================================
Version 0.0.2 - 2013-02-05b
============================================================================================

A new ESP6500SI catalog is available for the NHLBI GO Exome Sequencing Project (ESP). Please visit the drupal documentation page at http://bsiweb.mayo.edu/esp for details.


============================================================================================
Version 0.0.2 - 2013-02-05a
============================================================================================

New command (bior_vep) and bug fixes

    bior_vep
        Based on feedback from the Jan 21 stakeholder meeting, the team has finished development of the command named bior_vep. The default behavior of the command has been changed to suit the needs of the TREAT/GenomeGPS workflow. Here's a snippet from the command help:
        NOTE: By default this command will select a single transcript that has the worst possible 

outcome as predicted by Sift/PolyPhen. This differs from VEP's behavior of producing multiple transcripts per input variant. To retain VEP's behavior of a single variant row becoming multiple rows, each row containing the variant plus transcript specific data, use the --all option.

    Bug fix: The bior_vcf_to_json command had an issue handling VCF files with blank columns.
        For details, see http://bsu-bugs/default.asp?1882 

The latest bior_vep command and bug fix for bior_vcf_to_json are now available in the RCF environment. Please take a look and give it a test drive. We appreciate any feedback you can provide.


============================================================================================
Version 0.0.2 - 2013-01-10
============================================================================================

The team has finished development of a new command named bior_vep. This command allows you to take variants from a VCF file and stream them into the Variant Effect Predictor (VEP) http://useast.ensembl.org/info/docs/variation/vep/index.html to get SIFT and Polyphen functional prediction data. A more detailed breakdown on the command can be found in the help text included below.

One other noteworthy item is that we have removed the ".sh" suffix from all commands to improve usability. For example, "bior_drill.sh" is now just "bior_drill".

The bior_vep command is now available in the RCF environment. Please take a look and give it a test drive. We appreciate any feedback you can provide.
Version 0.0.1 - 2012-11-28

There will be an outage of the BioR system this Friday, 11/28 from 12PM-5PM.

The purpose of this outage is to upgrade the server bior.mayo.edu to the newest release (1.2). Please review the Release Notes at http://bsiweb.mayo.edu/release-notes for more details on version 1.2.

NOTE: Some bug fixes in version 1.2 have made legacy BioR client versions incompatible. Legacy BioR client versions will be required to upgrade to 1.2. 