MCarts Documentation
From Zhang Laboratory
Contents
Introduction
mCarts is a hidden Markov model (HMM)-based methods to predict clusters RNA motif sites.
Many RBPs recognize very short and degenerate sequences, with targeting specificity achieved by mechanisms such as synergistic binding to multiple clustered sites and modulation of site accessibility through different RNA-secondary structures. mCarts integrates the number and spacing of individual motif sites, their accessibility and conservation, which substantially improves signal to noise ratio. This algorithm learns and quantifies rules of these features, taking advantage of a large number of in vivo RBP binding sites obtained from high throughput sequencing of RNAs isolated by cross-linking and immunoprecipitation (HITS-CLIP). We applied this algorithm to study two representative RBPs, Nova and Mbnl. Despite the very low information content in individual motif elements, our algorithm made very specific predictions for successful experimental validation.
Citation:
Zhang, C. †, Lee, K.-Y., Swanson, M.S., Darnell, R.B. † 2013. Prediction of clustered RNA-binding protein motif sites in the mammalian genome. Nucleic Acids Res, in press.
Versions
- v1.2.0 ( 02-03-2015 ), current
- included PatternMatch and RegExpMatch as part of the software
- v1.0.2 ( 05-14-2013 )
- Minor bug fixes in the pipeline
- some minor internal restructuring
- v1.0.1 ( 09-20-2012 )
- minor bug fixes
- v1.0.0 ( 09-20-2012 )
- The initial Public release
Download
Source code:
- mCarts (perl): the core algorithm. (download from github)
- czplib (perl): a required perl library with various functions for genomic/bioinformatic analysis. (download from github)
Note: A separate download is no longer required for the following starting with mCarts version 1.2.0 (Feb 2015):
- PatternMatch (c/c++): a handy tool to search individual motif sites based on consensus. It supports degeneracy and mismatches. (download from github)
- RegExpMatch (c/c++): a handy tool to search individual motif sites based on regular expression (download from github)
Library data:
- mm10 (24 GB compressed / 32 GB uncompressed): download
- hg19 (30 GB compressed / 39 GB uncompressed): download
Previous libraries:
- mm9 (9.8 GB compressed / 61 GB uncompressed): download
- hg18 (24 GB compressed / 119 GB uncompressed): download
Sample data files:
- Nova unique CLIP tags (mm10, for Springer protocol) (40 MB compressed / 167 MB uncompressed): download
- RepeatMasker database at time of publication (mm10, for Springer protocol) (56 MB compressed / 180 MB uncompressed): download
- Nova training data (mm9, for NAR paper) (1.8 MB compressed / 5.7 MB uncompressed): download
| File | md5 |
|---|---|
| mCarts_lib_data_mm10.tgz | 79ee0842eeda2bf737caa31bf7e91737 |
| mCarts_lib_data_hg19.tgz | 83b3734b6e64be5ac4396ab5be78beee |
| mCarts_lib_data_mm9.tgz | 36240de47d00e29c87de0459b1c41962 |
| mCarts_lib_data_hg18.tgz | d01a4ed04a1bcb59f9b795d2ebed0b57 |
| Nova_CLIP_uniq_mm10.bed.gz | b3cc2adf0c8b4b5ad0af17996a346b63 |
| mm10.rmsk.bed.gz | 630bd2e0adf98db97101250d967bfba4 |
| Nova_train_data.tgz | 253446b978fd8685fc3076d91fd932c2 |
Installation
Prerequisites
This software is implemented with perl and c/c++. It also relies on several standard linux/unix tools such as grep, cat, sort, etc. We have tested the software on RedHat Linux, although it is expected to work on most unix-like systems, including Mac OS X.
Steps to install the software
- Download the perl library files czplib, if not already.
Decompress it and move it to a place you like
$tar zxvf czplib.v1.0.x.tgz $mv czplib /usr/local/lib
Add the library path to the environment variable, so perl can find it.
PERL5LIB=/usr/local/lib/czplib
- Download mCart codes, if not already.
Decompress it and move it to a place you like
$tar zxvf mCarts.v1.0.x.tgz $cd mCarts $chmod 755 *.pl $mv mCarts /usr/local/mCarts
Add the dir to your $PATH environment variable.
- Download and compile PatternMatch and RegExpMatch
$tar zxvf PatternMatch.v1.0.x.tgz $cd PatternMatch $make $chmod 755 PatternMatch $mv PatternMatch /usr/local/bin $tar zxvf RegExpMatch.v1.0.x.tgz $cd RegExpMatch $make $chmod 755 PatternMatch $mv RegExpMatch /usr/local/bin
If your system does not have the boost c++library, which is required by RegExpMatch, check here for details: http://www.boost.org/doc/libs/1_51_0/libs/regex/doc/html/index.html
Make sure /usr/local/bin is already in your $PATH
- Download and decompress library files
If the library files were provided in split pieces (*.tgz.part*) to reduce the size of each part, concatenate them first by using the follow command:
$cat mCarts_lib_data_hg18.tgz.part-* > mCarts_lib_data_hg18.tgz
$tar zxvf mCart_lib_data_mm9.tgz $mv mCart_lib_data_mm9 /home/czhang/data/mCart_lib_data_mm9
Get started
mCarts requires genomic sequences of genic regions, multiple alignments, exon/intron/UTR annotations, RNA accessibility information, etc, which are all provided in the library files (that's why it is huge!!). The list of library files is describe below (using mouse as an example):
- mm9.exon.uniq.bed: a collection of unique exons
- mm9.genic.bed: a collection of genic regions
- mm9.genic.ext10k.bed: a collection of genic regions, with 10 kb extension on both sides
- mm9_genic.ext10k_maf_split: multiple sequence alignments of extended genic regions
- mm9_genic.ext10k_pu4: pre-calculated single strandedness of all tetramers in extended genic regions
- refGene_knownGene.3utr.bed: 3' utr regions based on refSeq and UCSC known genes
- refGene_knownGene.5utr.bed: 5' utr regions based on refSeq and UCSC known genes
- species: list of 20 mammalian species (change the symbolic link to the 30 vertebrate species if necessary)
- tree.nh: phylogenetic tree of the 20 mammalian species (change the symbolic link to the 30 vertebrate species if necessary)
In addition, mCarts takes two sets of genomic regions, provided in two BED files, to obtain motif sites in the positive and negative training datasets of the HMM. The positive training regions are typically (several thousand) regions of robust CLIP tag clusters. The negative training regions are typically genic regions without any CLIP tags. Only motif sites in genic regions (as defined by library files) are actually used.
A real example to predict Nova binding YCAY clusters.
To get training data, download.
There are two files in the compressed package: mm9.Nova.train.pos.bed specifies 6,231 non-repetitive, genic Nova CLIP tag clusters with peak height (PH)≥15, and located in exons or 1 kb flanking intronic sequences on each side (exon+ext1k sequences). mm9.Nova.train.neg.bed specifies 110,998 exon+ext1k sequences, in which no CLIP tags were present.
It is recommended to divide the process to two steps, although they can also be combined into one single step:
Model training
$mCarts -v -ref mm9 -w YCAY -f ./Nova.train.pos.bed -b ./Nova.train.neg.bed -lib /home/zhangc/data/mm9_mammal_input_data --train-only ./mm9_Nova_out
This command specifies the verbose mode (-v), reference genome (-ref mm9), the consensus motif to search (-w YCAY), foreground or positive training regions (-f ./Nova.train.pos.bed), background or negative training regions (-b ./Nova.train.neg.bed), directory with library files (-lib /home/zhangc/data/mm9_mammal_input_data), model training only (--train-only), and the output dir (./mm9_Nova_out)
see a complete list of options.
This command will do the following things:
- Search for individual motif sites in the genic regions (with 10 kb extension on each side) in the reference genome (mm9) and many additional genomes (e.g., 19 other mammalian genomes aligned to mm9) and evaluate their conservation using branch length scores (BLS).
- Retrieve the RNA accessibility information as measured by probability of unpairedness (PU) or single strandedness. These scores for all tetramers in genic regions were pre-calculated, because calculating PU scores is quite slow.
- intersect motif sites with positive and negative training regions to get training motif sites for the HMM
- estimate parameters of the HMM.
The following out put files are of particular interest:
- model.txt:
This file saves parameters of the HMM, including emission probabilities and data to calculate transition probabilities. The emission probability is represented by histograms (nonparametric), and can show how each feature contrast RBP bound motif sites vs. background motif sites. This information is specified in the following lines:
distance_positive 3.59E-20 3.59E-20 3.59E-20 0.078482739 ... distance_negative 2.90E-22 2.90E-22 2.90E-22 0.044107727 ... # 0 0.05 0.1 0.15 ... conservation_positive_0 0.179478553 0.182674516 1.68E-19 0.023044575 ... conservation_positive_1 0.02999663 0.047691271 1.69E-19 0.005729693 ... conservation_positive_2 0.101576182 0.126094571 8.76E-19 0.014886165 ... conservation_positive_3 0.12325902 0.163378809 4.75E-20 0.019869753 ... conservation_negative_0 0.341826256 0.282508127 3.69E-22 0.033694368 ... conservation_negative_1 0.11093915 0.126552235 2.04E-21 0.01781323 ... conservation_negative_2 0.237788887 0.22985303 6.83E-21 0.022994864 ... conservation_negative_3 0.291355883 0.259434326 4.82E-21 0.029970126 ... accessibility_positive 0.014387222 0.057225909 0.073668448 0.082271419 ... accessibility_negative 0.059879224 0.132628229 0.118651224 0.106571071 ...
The probability of distance are for spacing from 0,1,2, ...
Conservation scores and accessibility scores are in the range between 0 and 1. For conservation scores, the suffix represents different regions (0-intron, 1-CDS, 2-5'UTR, 3-3'UTR).
This information can be used to produce a figure as shown below:
Note that the distance distribution for the positive training sites is censored at 30 nt (--max-dist 30, default). For Nova, all features contribute to the discrimination of Nova bound YCAYs and background sites.
- BLS/mm9.genic.ext10k.motif.bls.chrom.bed
This is the bed file of individual motif sites, which could be quite big if the motif is short and degenerate (like Nova). You can load this file to the genome browser for visualization. The 5th column if each row is the BLS conservation score of each site, which is in the range of 0 and 1, and should be re-scaled to 0-1000 for the best visualization.
$awk '{print $1"\t"$2"\t"$3"\t"$4"\t"$5*1000"\t"$6}' BLS/mm9.genic.ext10k.motif.bls.chrom.bed > mm9.genic.ext10k.motif.bls.chrom.2.bed
Prediction
After the model is generated, and verified (or even modified if you like by editing model.txt) to make sense, one can move forward for prediction using the command line below.
$mCarts -v --exist-model ./mm9_Nova_out
This will produce the results in the bed file cluster.bed. The 5th column is the motif cluster score, which can be used to rank the clusters. High scoring clusters are in general more reliable than low scoring clusters.
This file can be converted into bedGraph, which in combination with the individual motif sites, will give the best visualization.
Additional options
If you run mCarts without any parameters, it prints the usage information:
$perl ~/src/czsrc/mCarts/mCarts search clustered RNA motif sites Usage: mCarts [options] <out dir> Example1: mCarts -v -ref mm9 -f CLIP.pos.bed -b CLIP.neg.bed -lib mm9_mammal_input_data -w YCAY -m 3 --min-site 3 --max-dist 30 out_dir Example2: mCarts -v --exist-model out_dir_from_prev_run
Options:
| Option | Description |
|---|---|
| -ref [string] | reference species to search (mm9)
The data libraries for reference species of mm9 and hg18 are provided |
| -f [string] | a BED file specifying positive (foreground) training regions |
| -b [string] | a BED file specifying negative (background) training regions |
| -lib [string] | dir with data library files |
| -w [string] | the consensus motif to search
e.g., YCAY |
| -m [int] | number of mismatches (0) |
| -r | the motifs provided are regular expressions (will disable -n and -m)
The -w specifies a regular expression (e.g., TTTT+). More details about syntax is available at http://www.boost.org/doc/libs/1_51_0/libs/regex/doc/html/index.html |
| --min-site [int] | minimum sites in clusters (3) |
| --max-dist [int] | max distance allowed in clusters (30)
The max spacing between neighboring sites in a cluster |
| --train-only | train the model only, no prediction |
| --exist-model | prediction based on existing model specified in out dir |
| --check-maf | check maf files in the library dir
This option is reserved, and should not be used |
| -v | verbose |
Running jobs in parallel
Support for running jobs in parallel using a queue system (tested for SGE) is already included. The program will try to find if SGE is available. If yes, jobs will be dispatched to the default queue, and the results will be collected and combined when all jobs are done. Otherwise, the job will run locally.