Linux Command Line for Bioinformatics

Chapter 2 of 5: Set up your terminal and practice folder

What you need

• A Linux terminal. On Windows 10 (version 2004 or later) or Windows 11, use WSL2 with Ubuntu. On macOS, use the built-in Terminal app. On Linux, open any terminal.
• About 1 GB of free disk space for this tutorial. Real projects need much more: plan for tens to hundreds of GB.
• No special software. Every command here is part of a standard Linux system (GNU coreutils, grep, gzip and awk).

Windows: install WSL2 and Ubuntu

Open PowerShell as Administrator (right-click the Start button, choose Terminal (Admin) or Windows PowerShell (Admin)) and run:

wsl --install

Restart when asked. Ubuntu opens and asks you to choose a Linux user name and password. This password is only for Linux; you will need it for sudo commands later. From now on, open Ubuntu from the Start menu to get your terminal.

Step 1: Create your project folder

A clear folder layout keeps raw data safe and results easy to find. We use three folders: data for inputs you never edit, results for everything you create, and scripts for your code.

cd ~
mkdir -p bhl-linux/{data,results,scripts}
cd bhl-linux
ls

• mkdir: make a directory (folder)
• -p: create parent folders as needed, and do not complain if they already exist
• {data,results,scripts}: brace expansion; the shell turns it into three folder names before mkdir runs

Step 2: Create the practice files

We create five small files that look exactly like real ones: a genome (FASTA), sequencing reads (FASTQ), genes (GTF), variants (VCF) and a sample sheet. They are tiny on purpose, so you can check every answer by eye. The cat > file << 'EOF' form writes everything you type until the line EOF into the file.

The genome (FASTA):

cat > data/genome.fa << 'EOF'
>chr1 example chromosome 1
AAGCCCAATAAACCACTCTGACTGGCCGAATAGGGATATAGGCAACGACATGTGCGGCGA
CCCTTGCGACAGTGACGCTTTCGCCGTTGCCTAAACCTATTTGAAGGAGTCTAGCAGCCG
CAGTAAGGCACAATACCTCGTCCGTGTTAC
>chr2 example chromosome 2
CAGACCAAACAAGACGTCCTCTTCAATGTTTAAATGACCCTCTCGTCATAAAACCTTTCT
ACTATGTGTTCCGCAAGAATCAACAACTAC
>chrM example mitochondrion
AATGGCGCGTCGTGAATAACGCGACGGCTGAGACGAACGGCGCGT
EOF

The reads (FASTQ), six reads of 36 bases:

cat > data/reads_R1.fastq << 'EOF'
@read1/1
GAATGAAGCGCTTAAACAGCTCAGGAGCCAGTCCCC
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FF
@read2/1
GAAGCGAATGGTATCGATACGTAGGAGGTGTGCCTT
+
F::FFFFFFFF::FFFFFFFFFFF:FFFFF,,:F,#
@read3/1
CCAACTATTCTTTCCAATCCTACAGATCGGAAGAGC
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FF
@read4/1
CACGAAGTTCACAGAAGTGAGATTATGTCTCGTTTG
+
FFFFFFF::FFFFFFFFFFFF:FFFFFFFF,,:F,#
@read5/1
TCTTTAAGCTNGGTGTGGTGGGCACGACCCTGGACG
+
FFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFF:FF
@read6/1
AATGGAACAGACTTGCTTATGTGGACGTTGTATAGG
+
FFFFFFFFFFFF:FFFFFFFFFFFF:FFFFFFF:FF
EOF

GTF, VCF and sample sheets are tab-separated. Tabs are invisible and easily turn into spaces when text is copied, so we type a | between columns and let the tr command (translate characters) turn every | into a real tab:

cat > data/genes.txt << 'EOF'
chr1|bhl|gene|5|120|.|+|.|gene_id "G001"; gene_name "abcA";
chr1|bhl|exon|5|40|.|+|.|gene_id "G001"; gene_name "abcA";
chr1|bhl|exon|70|120|.|+|.|gene_id "G001"; gene_name "abcA";
chr1|bhl|gene|125|148|.|-|.|gene_id "G002"; gene_name "xyzB";
chr1|bhl|exon|125|148|.|-|.|gene_id "G002"; gene_name "xyzB";
chr2|bhl|gene|10|80|.|+|.|gene_id "G003"; gene_name "lmnC";
chr2|bhl|exon|10|35|.|+|.|gene_id "G003"; gene_name "lmnC";
chr2|bhl|exon|50|80|.|+|.|gene_id "G003"; gene_name "lmnC";
EOF

cat > data/variants.txt << 'EOF'
##fileformat=VCFv4.2
##contig=<ID=chr1,length=150>
##contig=<ID=chr2,length=90>
##INFO=<ID=DP,Number=1,Type=Integer,Description="Read depth">
##FILTER=<ID=LowQual,Description="Low quality">
#CHROM|POS|ID|REF|ALT|QUAL|FILTER|INFO
chr1|12|.|A|T|58|PASS|DP=31
chr1|47|.|G|A|12|LowQual|DP=6
chr1|88|.|T|A|71|PASS|DP=40
chr1|133|.|A|C|45|PASS|DP=22
chr2|21|.|CT|C|38|PASS|DP=18
chr2|64|.|A|C|9|LowQual|DP=4
EOF

cat > data/samples.txt << 'EOF'
sample|condition
S01|control
S02|control
S03|treated
S04|treated
EOF

tr '|' '\t' < data/genes.txt > data/genes.gtf
tr '|' '\t' < data/variants.txt > data/variants.vcf
tr '|' '\t' < data/samples.txt > data/samples.tsv
rm data/*.txt

Finally, compress the reads, because real FASTQ files always arrive compressed:

gzip data/reads_R1.fastq
ls -lh data

If a file size is different, compare your file with the text above; a missing or extra line is the usual cause. Buyers of the study material get these files (and a larger practice dataset) ready to download.

Step 3: Copy, move, rename and delete

cp data/genome.fa results/genome_copy.fa
mv results/genome_copy.fa results/backup.fa
ls results
rm -i results/backup.fa

• cp source destination: copy a file (cp -r copies a whole folder)
• mv source destination: move a file; moving within the same folder is how you rename
• rm file: delete a file; -i asks for confirmation (type y)
• rmdir folder: delete an empty folder

Warning: the command line has no recycle bin. rm deletes immediately and permanently, and rm -r deletes whole folders. Never run rm -r with a * until you have checked the exact list with ls first. mv and cp also silently overwrite a destination that already exists; add -i to be asked first.

Step 4: Look inside files without opening them

head -n 4 data/genome.fa
tail -n 2 data/genome.fa
wc -l data/genome.fa data/genes.gtf data/variants.vcf

• head -n 4: first 4 lines (default 10)
• tail -n 2: last 2 lines; tail -n +2 means “from line 2 to the end”, handy for skipping a header
• wc -l: count lines (wc alone also counts words and characters)
• less -S file: scroll through a file page by page; -S stops long lines wrapping; press q to quit and / to search
• cat file: print the whole file; fine for tiny files, a bad idea for a 10 GB FASTQ

Compressed files need one extra step. zcat unzips a .gz file on the fly and prints it, without creating an unzipped copy on disk:

zcat data/reads_R1.fastq.gz | head -n 4