Tracking the astounding pace of digital storage

Tracking the astounding pace of digital storage

from Boing Boing by Cory Doctorow

9 people liked this

Ivan Smith maintains a page tracking the price of digital storage over the years. This is one of technology's least appreciated growth stories -- we hear a lot about Moore's Law and the doubling of processing capacity, but storage-density's growth makes the pace of processor improvements look glacial. Every now and then I realize that the 32GB SD card in my camera costs less than the 16k memory upgrade I put in my Apple ][+ in 1980, even without accounting for inflation, and I am croggled. Here are David Isenberg's benchmarks, calculated from Smith's records:

YEAR -- Price of a Gigabyte1981 -- $300,000
1987 -- $50,000
1990 -- $10,000
1994 -- $1000
1997 -- $100
2000 -- $10
2004 -- $1
2010 -- $0.10

It would be interesting to do the same chart for a megabyte -- you'd go from six figures to fractional pennies in a damned short period.
Cost of Hard Drive Storage Space (via Isen.blog)

Quality control and preprocessing of metagenomic datasets

Quality control and preprocessing of metagenomic datasets

from Bioinformatics - current issue by Schmieder, R., Edwards, R.

Summary: Here, we present PRINSEQ for easy and rapid quality control and data preprocessing of genomic and metagenomic datasets. Summary statistics of FASTA (and QUAL) or FASTQ files are generated in tabular and graphical form and sequences can be filtered, reformatted and trimmed by a variety of options to improve downstream analysis.
Availability and Implementation: This open-source application was implemented in Perl and can be used as a stand alone version or accessed online through a user-friendly web interface. The source code, user help and additional information are available at http://prinseq.sourceforge.net/

OICR's Genomics Pathway-Sequencing Project Highlights Genomic Data's Steady March Into Clinics

OICR's Genomics Pathway-Sequencing Project Highlights Genomic Data's Steady March Into Clinics

The Ontario Institute for Cancer Research has kicked off a project that promises to address some key informatics challenges related to clinical sequencing.

Lincoln Stein, platform leader of informatics and biocomputing at the OICR, described the project in a presentation at the fourth Summit on Translational Bioinformatics in San Francisco this week.

Stein said that the project, conducted in partnership with Canada's Princess Margaret Hospital and dubbed the Genomics Pathway Sequencing project, or GPS, will sequence genes in normal and tumor samples excised from patients.

.....

“The biggest challenge,” Stein said, “is trying to keep the amount of information in this report to a minimum without keeping potentially important information back [from physicians].”

....

For each individual, OICR researchers will sequence about 1,000 cancer-related genes in control and tumor samples and attempt to identify mutations that are of immediate relevance to the patient’s cancer care, or that are targeted by drugs that are currently being tested in trials.

...

The genes in the study — which include the usual suspects such as KRAS, p53, and B-Raf — were selected based on suggestions from oncologists in the Toronto area as well as data stored in knowledgebases at the Memorial Sloan Kettering Cancer Center, the Wellcome Trust Sanger Institute, and the National Cancer Institute.

To sequence patient samples, OICR will use Pacific Biosystems' single-molecule sequencing as a first step to discover novel mutations. It will then sequence known mutations on Sequenom's MassArray platform. The sequence variants will be confirmed using a Sanger sequencer housed at the clinical sequencing lab at Princess Margaret hospital.

Stein explained that the OICR selected the PacBio system because “it allows us to do very long reads at high coverage … currently 1,000-base-pair reads in a circular consensus, which gives high accuracy for targeted genes.”

In addition, he said, the platform has a rapid turnaround time, taking only “15 minutes per run to completely analyze … a single typical gene.”

Kevin: fascinating! Would their's become the de facto standard for clinical sequencing?

Would the closing of SRA affect you?

Short Read Archive Canned

from Pathogens: Genes and Genomes by Nick Loman

4 people liked this

This email apparently from NCBI head-honcho David Lipman was posted in the comments section atTree of Life:

Dear Staff Members of NCBI,
As you are aware, the federal government as well as NIH is facing a period of budgetary uncertainty that is resulting in ongoing program reviews throughout the government.   At NCBI our senior staff have been giving serious consideration to our own projects and staffing levels in order to prepare for and adjust to new fiscal constraints.
NCBI had received a significant adjustment in its appropriated funding in the proposed FY2011 President’s Budget.  The President’s Budget, however, has not been enacted and we are being required to operate at last year’s (FY2010) level under a Continuing Resolution (CR) from Congress.  Upon the CR’s expiration on March 4, 2011, there is little likelihood the budget picture will improve.  The NIH Office of the Director has provided us with stop-gap funding to alleviate some of our FY2011 and FY2012 funding needs.

Wow, big news. And not just because the fact that NCBI is downsizing is a worry in terms of funding priorities – given the ongoing explosion in production of genomics data.
I, like many others, won’t necessarily be sorry to see the back of the Short Read Archive as it was a bit of a pain to upload to and a massive pain to retrieve from.
But the question is now, what will become the de facto place to find short-read data, or pointers to data? Certainly the SRA was useful – particularly for getting data to test bioinformatics applications against.




Kevin: like others, the SRA was a useful place to get demo data for testing software, but moving forward I haven't gone back to SRA (partly as my work takes me away from testing more stuff ) but I think it points to the fact that  raw sequencing data might be too expensive for anyone to archive for long.

Statistics Meeting Focused on RNA-Seq Data Handling from RNA-Seq Blog

Statistics Meeting Focused on RNA-Seq Data Handling

from RNA-Seq Blog by admin

Joint Statistical Meeting July 30^th-August 4^th, 2011
Miami Beach, Fl
Sharing Information Across Genes to Estimate Overdispersion in RNA-seq Data 
Steven Peder Lund, Iowa State University; Dan Nettleton, Iowa State University31/2011
Differential expression analysis for paired RNA-seq data 
Lisa M Chung, Department of Epidemiology and Public Health, Yale University ; John Ferguson, Department of Epidemiology and Public Health, Yale University ; Hongyu Zhao, Yale University
How to characterize dynamic bayesian networks across multiple species from time series mRNA-Seq count gene expression profiles:An intelligent Dynamic Bayesian Networks (IDBNs) 
sunghee OH, Yale University; Hongyu Zhao, Yale University; James P. Noonan, Yale University
Statistical methods for the analysis of next-generation sequencing data 
Karthik Devarajan, Fox Chase Cancer Center
MEAN-VARIANCE MODELING OF RNA-SEQ TRANSCRIPTIONAL COUNT DATA 
Yihui Zhou, Univ North Carolina; Fred Andrew Wright, Univ North Carolina
On Differential Gene Expression Using RNA-Seq Data 
Ju Hee Lee, University of Texas, MD Anderson Cancer Center; Yuan Ji, MD Anderson Cancer Centre – University of Texas; Shoudan Liang, University of Texas, MD Anderson Cancer Center; Guoshuai Cai, University of Texas, MD Anderson Cancer Center; Peter Mueller, MD Anderson Cancer Center
A Bayesian nonparametric method for differential expression analysis of RNA-seq data 
Yiyi Wang, Department of Statistics, Texas A&M University; David B. Dahl, Department of Statistics, Texas A&M University
Statistical strategy for eQTL mapping using RNA-seq data 
Wei Sun, University of North Carolina, Chapel Hill
Joint analyses of high-throughput DNA and RNA-seq data from cancer samples 
Su Yeon Kim, University of California, Berkeley; Terence Speed, University of California, Berkeley
Significance Analysis of time-series gene expression profiles :via differential/trajectory models in temporal mRNA-Seq data 
sunghee OH, Yale University; Hongyu Zhao, Yale University; James P. Noonan, Yale University
Model-Based Clustering for RNA-Seq Data 
Yaqing Si, Iowa State University; Peng Liu, Iowa State University
An integrative approach to comparing and normalizing gene expression data generated from RNA-seq, microarray, and RT-PCR technologies 
Zhaonan Sun, Department of Statistics, Purdue University; Yu Zhu, Department of Statistics, Purdue University
Normalization, testing, and false discovery rate estimation for RNA-sequencing data 
Jun Li, Department of Statistics, Stanford University; Daniela Witten, University of Washington; Iain M Johnstone, Stanford University; Robert Tibshirani, Dept of Health Research and Policy, & Statistics, Stanford University
Statistics Meeting Focused on RNA-Seq Data Handling is a post from: RNA-Seq Blog More information about RNA-Seq can be here.

JCVI Supports Human Mircrobiome Body Site Experts with Shotgun Data Analysis from JCVI Blog by Johannes Goll

The current survey comprises more than 700 samples from hundreds of individuals taken from up to 16 distinct body sites. Illumina sequencing has yielded more than 20 billion Illumina reads and annotation data produced from the sequences exceeds 10 terabytes. In anticipation of such data volumes, we have developed JCVI Metagenomics Reports (METAREP), an open source tool for high-performance comparative analysis, in 2010. The tool enables users to slice and dice data using a combination of taxonomic and functional/pathway signatures. To demonstrate how the tool can be used by body site experts, we picked and loaded sample data from 17 oral samples and presented a quick tutorial on how users can view, search, browse individual samples and compare multiple samples (see video). The functionality was very well received and body site experts asked JCVI to make all the 700+ samples available. As a result of the Jamboree, JCVI in agreement/collaboration with the HMP Data Analysis and Coordination Center and the rest of the HMP consortium, will soon set-up a dedicated HMP METAREP instance that will allow body-site experts and eventually other users to analyze the DAWG data in a user-friendly way via the web.

Tip of the Week: World Tour of Genomics Resources from The OpenHelix Blog

Tip of the Week: World Tour of Genomics Resources

from The OpenHelix Blog by Jennifer

Most weeks our tip is a five-minute movie that quickly introduces you to a new resource, or a cool new function at an established resource. Occasionally we feature one of our full resource tutorial that is being made freely available through resource sponsorship of our training suite. In this week’s tip we provide access to one of our tutorials that is especially near and dear to our heart. It is a World Tour of Genomics Resources in which we explore a variety of publicly-available biomedical, bioinformatics and bioscience databases and other resources.
This tutorial is quite different from our usual ones. Generally we focus on a specific software resource and describe step-by-step how to use its functions such as how to do basic and advanced searches, how to understand and modify displays, where to find specific types of data such as FASTA sequences, etc. and even provide tips on ‘hidden features’ that power users even find useful and informative.  This type of software training is absolutely critical.
But many people need an even earlier step: just the *awareness* that resources are available that might serve their needs. This tutorial fills that niche. We present a sampling of resources, all free to use, from each of 9 categories including: Analysis & Algorithms, Expression, Genome Browsers (for Eukaryotes and for Prokaryotes and Viruses), Genome Variation,  Literature, Nucleotides, Pathways and Proteins. After the World Tour, which is the majority of the tutorial, we then describe how to use OpenHelix’s free search and learn portal to find bioscience resources most appropriate for your research needs. From this the tour transitions into a brief discussion of the format of our training materials and how to use them, and then ends with information about other learning resources that we provide.
This tutorial has been wildly popular whenever we’ve done it as a live seminar. At the NIH they actually had to lock the doors because we’d hit the capacity of the room, and people were turned away. In fact, it has been so popular that we decided to produce it as a full tutorial suite and release it as one of our free trainings so that anyone and everyone could learn about the breadth of great public software options available for free use.
In addition to this free tutorial, we also have published a paper entitled “OpenHelix: bioinformatics education outside of a different box” in a special issue of Briefings in Bioinformatics entitled “Special Issue: Education in Bioinformatics“. This paper describes a plethora of sources where researchers can access informal educational sources of learning on publicly available bioinformatics resources. The sources of information include a wide variety of formats including lists of resources, journals that regularly feature tool descriptions, and eLearning resources sources such as the MIT OpenCourseWare effort. If you know of other such resources that aren’t covered in our tour or paper, comment & let us know about them – we love to learn as much as we love to teach!
Quick link to World Tour of Genomics Resources tutorial here.

• Williams, J., Mangan, M., Perreault-Micale, C., Lathe, S., Sirohi, N., & Lathe, W. (2010). OpenHelix: bioinformatics education outside of a different box Briefings in Bioinformatics, 11 (6), 598-609 DOI:10.1093/bib/bbq026