Exploring the 1000 genome dataset with Hail on Amazon EMR and Amazon Athena

 Blog post from Roy Hasson

https://aws.amazon.com/blogs/big-data/genomic-analysis-with-hail-on-amazon-emr-and-amazon-athena/?nc1=b_rp

Genomics analysis has taken off in recent years as organizations continue to adopt the cloud for its elasticity, durability, and cost. With the AWS Cloud, customers have a number of performant options to choose from. These options include AWS Batch in conjunction with AWS Lambda and AWS Step Functions; AWS Glue, a serverless extract, transform, and load (ETL) service; and of course, the AWS big data and machine learning workhorse Amazon EMR.

For this task, we use Hail, an open source framework for exploring and analyzing genomic data that uses the Apache Spark framework. In this post, we use Amazon EMR to run Hail. We walk through the setup, configuration, and data processing. Finally, we generate an Apache Parquet–formatted variant dataset and explore it using Amazon Athena.

Slides for the 1000 Genomes Project tutorial at ASHG 2015

The 1000 Genomes project is holding a tutorial at ASHG 2015.

This tutorial will cover and overview of the 1000 Genomes Data, how to access it and how it can be used.

1. 7:15 – 7:30 pm Overview of the data set Hyun Min Kang - University of Michigan
2. 7:30 – 7:45 pm How to access the data Holly Zheng Bradley - European Bioinformatics Institute
3. 7:45 – 8:00 pm Structural variants Eugene Gardener - University of Maryland
4. 8:00 – 8:15 pm Functional variation Stephen Montgomery - Stanford University
5. 8:15 – 8:30 pm Admixture Alicia Martin - Broad Institute
6. 8:30 – 8:45 pm Using LDLink Mitch Machiela - National Cancer Institute7. 8:45 – 9:00 pm Q&A

No registration is needed for this meeting. Slides will be posted after the meeting is finished.

Source link: http://www.1000genomes.org/ashg-2015-1000-genomes-tutorial-thursday-october-8th-7-9pm-baltimore-convention-center-room-327-leve

CORTEX update contains scripts for assembly of large numbers of samples with large genomes - i.e. for the 1000 Genomes project.

cortex_var is a tool for genome assembly and variation analysis from sequence data. You can use it to discover and genotype variants on single or multiple haploid or diploid samples. If you have multiple samples, you can use Cortex to look specifically for variants that distinguish one set of samples (eg phenotype=X, cases, parents, tumour) from another set of samples (eg phenotype=Y, controls, child, normal). See our Nature Genetics paper and the documentation for detailed descriptions.

http://cortexassembler.sourceforge.net/index_cortex_var.html

The Cortex paper is now out in Nature Genetics! 

cortex_var features

• Variant discovery by de novo assembly - no reference genome required
• Supports multicoloured de Bruijn graphs - have multiple samples loaded into the same graph in different colours, and find variants that distinguish them.
• Capable of calling SNPs, indels, inversions, complex variants, small haplotypes
• Extremely accurate variant calling - see our paper for base-pair-resolution validation of entire alleles (rather than just breakpoints) of SNPs, indels and complex variants by comparison with fully sequenced (and finished) fosmids - a level of validation beyond that demanded of any other variant caller we are aware of - currently cortex_var is the most accurate variant caller for indels and complex variants.
• Capable of aligning a reference genome to a graph and using that to call variants
• Support for comparing cases/controls or phenotyped strains
• Typical memory use: 1 high coverage human in under 80Gb of RAM, 1000 yeasts in under 64Gb RAM, 10 humans in under 256 Gb RAM

23rd August 2012: Bugfix release v1.0.5.11. Get it here.. The main change in this release is in the scripts/1000genomes directory, which I have not advertised previously. It contains scripts for running Cortex on large numbers (tens, hundreds) of samples with large genomes - i.e. for the 1000 Genomes project. These are to allow collaborators across the world to reliably run a consistent Cortex pipeline on human populations. However this is the first time people other than me have done this, so I expect there may be some smoothing-out of issues in the near future. You can see aPDF describing the pipeline here. I've had enough people ask me about running Cortex on lots of samples with big genomes, that I thought people would find it useful to see the process. This release is a bugfix for a script in that 1000 Genomes directory, plus fixes for a few potential bugs-in-waiting (array overflow errors) in Cortex itself.

I love IGV!

Well sorry for the proclamation

But I think it's really nice that you have a relatively small footprint for a application that's cross platform.

anyway I needed to view a few regions of a merged bam file that's aligned to hg18 from 1000 genomes project.
You would think I might have copies of the hg18 lying around on every server so that I can just use 'samtools tview in.bam ref.fasta'

But unfortunately I don't and I am not really looking forward to downloading the hg18 just for a verification exercise.

Luckily I remembered that IGV has prebuilt human genome references that's available on the fly.
with a few clicks, I was happily viewing the regions of interest. (I downloaded the compiled binary instead of using the 'launch from browser' option but that would have been cool too)

among the list of reference human genomes avail there was actually one available for 1KG specifically. How cool is that?


Go on take the application for a run if you haven't!

http://www.broadinstitute.org/software/igv/download

VCFtools BEDtools compare, intersect, merge

A Good computational biologist is only as good as the tools he uses (or maybe how good he is at google) rofl kidding ...
Life is always easier when you find the correct tool.
I also adopt the path of least resistance when trying to solve problems that are more common than I imagine.
There's always the good old linux tools for comparing SNPs called from different programs / options

   grep | sed | awk | cut | diff | comm
   see http://ged.msu.edu/angus/tutorials-2011/snp_tutorial.html


and if you are working with NGS data, you most probably already have samtools installed on your system and you might have used bcftools
Did you also know that there's also a (unrelated) set of tools called vcftools?
http://vcftools.sourceforge.net/

The VCFtools package is broadly split into two sections: 

• The vcftools binary program, generally used to analyse VCF files.
• The Vcf.pm perl module, which is a general Perl API containing a core of the utilities vcf-convert, vcf-merge, vcf-compare, vcf-isec, and others.

  Documentation

  • Documentation for the binary executable vcftools
  • The Perl API and scripts
  • Useful shell one-liners

  Examples of usage by topic

  • Installation
  • Annotating
  • Comparing
  • Concatenating
  • Converting
  • Intersections, complements
  • Merging
  • Querying
  • Reordering columns
  • Stats
  • Stripping columns
  • Useful shell one-liners
  • Validating

Then there's  also the highly used BEDTools http://code.google.com/p/bedtools/

which I highly recommend to keep as part of your tools collection. Check out the link below

Do watch out for this 'oversight' in vcftools as pointed out in seqanswers.
Overlap number discrepancy between VCFTools and BEDTools

Usage

Examples of common usage.   Featured

1000 Genomes Project Tutorial Videos

The 1000 Genomes Project has released the data sets for the pilot projects and for more than 1000 samples for the full-scale project. A tutorial for how to use the data was held at the 2010 American Society of Human Genetics (ASHG) annual convention on November 3.

Videos for each of the tutorial sessions are now available. The tutorial describes 1000 Genomes Project data, how to access it and how to use it. Each of the speakers and their topics are listed below along with their tutorial videos and PowerPoint slides.

Introduction Gil McVean, Ph.D. Professor of Statistical Genetics University of Oxford
Description of the 1000 Genomes Data Gabor Marth, D.Sc. Associate Professor of Biology Boston College
How to Access the Data Steve Sherry, Ph.D. National Center for Biotechnology Information National Library of Medicine National Institutes of Health. Bethesda, Md.
How to Use the Browser Paul Flicek, Ph.D. European Molecular Biology Laboratory Vertebrate Genomics Team European Bioinformatics Institute (EBI)
Stuctural Variants Jan Korbel, Ph.D. Group Leader, Genome Biology Research Unit Joint Appointment with EMBL-EBI European Molecular Biology Laboratory (Heidelberg, Germany)
How to Use the Data in Disease Studies Jeffrey Barrett, Ph.D. Team Leader, Statistical and Computational Genetics Wellcome Trust Sanger Institute Hinxton, United Kingdom