Bioinformatics File Formats Explained: FASTA, FASTQ, SAM/BAM, VCF, GTF and BED

Chapter 1 of 5: Bioinformatics file formats and how your data flows

Every bioinformatics project is a chain of files. The sequencer writes FASTQ. The aligner reads FASTQ and a FASTA reference and writes SAM, BAM or CRAM. The variant caller writes VCF. Genes and exons come as GFF3 or GTF, and regions of interest as BED. If you understand these bioinformatics file formats, you can read any pipeline, check any result by eye, and fix most errors in minutes instead of days.

In this free step-by-step tutorial you will open each format line by line, on small real files that you create or download yourself. You will learn what every column means, which formats count from 0 and which from 1, why .gz is not always the same .gz, and how index files (.fai, .bai, .tbi) make huge files fast. You need only a terminal and the tools from the Conda and Bioconda tutorial.

What you will learn

• Why file formats matter, and the three silent errors they cause (off-by-one, chromosome names, genome build)
• Text, binary and compressed files: plain gzip vs bgzip, and what index files do
• The difference between 1-based and 0-based coordinates, with a rule you will never forget
• FASTA and FASTQ: headers, wrapping, IUPAC codes, quality encoding and the .fai index
• SAM/BAM/CRAM: the header, the 11 mandatory columns, FLAG bits, CIGAR strings, sorting, indexing and region queries
• VCF/BCF: meta-lines, the 8 fixed columns, genotypes, indel padding and tabix indexing
• GFF3, GTF and BED: gene models, the ID/Parent hierarchy, CDS phase, and safe conversion between them
• How to check, convert and troubleshoot files with samtools, bcftools, bedtools and seqkit

Why bioinformatics file formats matter

A file format is a contract. The tool that writes the file and the tool that reads it agree on what each line and each column means. When both sides follow the contract, pipelines just work. When they do not, you rarely get a crash. You get a wrong answer that looks right.

Three mistakes cause most of these silent errors:

• Off-by-one coordinates. BED counts from 0, VCF and GFF count from 1. Mix them up and every exon, SNP or primer is shifted by one base.
• Chromosome names. Ensembl writes 1, UCSC writes chr1. A BED file with chr1 will match nothing in a BAM that uses 1, and many tools just return zero results.
• Genome build. Position 1,000,000 on GRCh37 (hg19) is a different base from position 1,000,000 on GRCh38 (hg38). The file looks fine; the biology is wrong.

This tutorial teaches you to spot all three by reading the files themselves.

Text, binary, compressed and indexed files

Some formats are plain text: you can open them with less, search them with grep and cut columns with cut. Others are binary: they are smaller and faster but need a tool to read them. Most large text files are also compressed.

Two ideas explain the whole table. BGZF (block gzip, made by bgzip) is gzip cut into small independent blocks, so a tool can jump into the middle of the file. An index is a small side file that says which block holds which genomic region. Together they let IGV or samtools view file.bam chr7:55,000,000-55,300,000 answer in under a second, even for a 100 GB BAM.

Coordinates: 1-based vs 0-based (the rule that saves you)

Every genomic format must say where a base is. There are two systems.

• 1-based, closed [start, end]: the first base is 1, and both start and end are included. Used by SAM, VCF, GFF3/GTF and by region strings you type, such as chr1:100-200.
• 0-based, half-open [start, end): the first base is 0, start is included, end is excluded. Used by BED, and inside the binary BAM and BCF files (tools convert for you when they print).

The rule: to go from 1-based to BED, subtract 1 from the start and keep the end. To go back, add 1 to the start. The end never changes. The length is end − start + 1 in 1-based coordinates and simply end − start in BED.

Worked example. In this tutorial’s practice gene, exon 1 covers bases 1 to 15 of a sequence called ref. In GFF3 that is ref … 1 15. In BED it is ref 0 15. Both describe exactly 15 bases: 15 − 1 + 1 = 15 and 15 − 0 = 15.

# Converting coordinates (pseudo-code)
bed_start = one_based_start - 1
bed_end   = one_based_end          # unchanged
length    = bed_end - bed_start    # = one_based_end - one_based_start + 1

# A single 1-based position p (for example a SNP in a VCF) as BED:
bed_line  = chrom, p - 1, p

Chromosome names and genome builds

Every file that holds positions also holds sequence names: @SQ SN: lines in SAM, ##contig lines in VCF, the first column of BED and GFF. All files in one analysis must use the same names and the same lengths, taken from the same reference FASTA. Later in this tutorial you will compare names across files with one command.