Linux Command Line for Bioinformatics

Chapter 5 of 5: Troubleshooting, practice, FAQ and next steps

Paid videoThis video is part of the paid study material.Buy the study material or join membership to watchAlready bought or a member? Sign in

Troubleshooting

Problem	Likely cause	What to do
command not found	Typo, tool not installed, or its conda environment not active	Check spelling; run which tool; conda activate the right environment
No such file or directory	Wrong folder or wrong relative path	pwd and ls; use Tab completion; try the absolute path
Permission denied when running a script	No execute permission	chmod +x script.sh, or run bash script.sh
A file is suddenly empty	> overwrote it, or an unquoted > in a grep pattern	Restore from backup or re-download; keep raw data read-only (chmod a-w)
The terminal seems frozen	The command waits for input (no file given) or is still running	Press Ctrl+C; check the command has an input file
Binary garbage on screen	head or cat on a .gz file	Use zcat file.gz | head
awk gives wrong columns	Spaces inside a column, default field splitting	Add -F'\t'
Counts look one too low, or ^M characters appear	Windows line endings (\r\n) or no newline at the end	cat -A file | head shows ^M$; fix with sed -i 's/\r$//' file or dos2unix
No space left on device	Disk full	df -h, du -sh *; keep outputs gzipped; delete intermediate files you can recreate

▸ Go deeper: Windows line endings in detail

Windows ends each line with two characters, carriage return and line feed (\r\n); Linux uses only \n. A sample sheet saved from Excel on Windows can therefore carry an invisible \r at the end of every line. Then $2 == "treated" never matches, because the field is really treated\r. The file command reports “with CRLF line terminators” for such files. Fix them once, right after copying them into data/.

Practice exercise

Use the files in ~/bhl-linux/data. Write one command (or a short pipeline) for each question, then check your answers below.

1. How many sequences are in genome.fa, and which one is the longest?
2. How many reads in reads_R1.fastq.gz have # (the lowest quality) as their last quality character?
3. How many exons are on chr1 in genes.gtf?
4. Save every PASS variant with QUAL of at least 40 to results/high_quality.vcf, keeping the header lines. How many variants does the new file contain?

Quick check: what is the difference between grep -v ‘^#’ and grep -v ‘^##’ on a VCF file?

grep -v '^#' removes every header line, including the #CHROM column-name line. grep -v '^##' removes only the meta-information lines and keeps #CHROM, which is handy when you want to see the column names above the data.

Study material

Complete study material: Linux for bioinformatics

• DATAPractice dataset (larger FASTA, FASTQ, GTF and VCF files)
• DATAReady-made scripts and command sheet
• IPYNBColab notebook: every command in this tutorial
• PDFSlides: all figures and tables (PDF)
• PDFFull solutions with explanations (PDF)

• Every file above, yours to keep
• Notebooks open in Google Colab (free)
• Certificate with QR verification after your project report is approved

Linux for bioinformatics: frequently asked questions

Do I need to install Linux on my computer?

No. On Windows, WSL2 runs a real Ubuntu system inside Windows. On a Mac, the Terminal app already gives you a Unix shell. For big data you will eventually work on a Linux server anyway.

Bash or zsh: does it matter?

Not for anything in this tutorial. macOS uses zsh by default, and every command here works the same way in both. Scripts start with #!/usr/bin/env bash, so they always run with Bash.

Should I learn Python or R instead of the command line?

You need both, for different jobs. The shell runs tools and moves and inspects files; Python and R are for statistics, tables and plots. Nearly every pipeline starts in the shell and ends in R or Python.

Is awk worth learning when Python exists?

Yes, for quick questions on large text files. A one-line awk command answers “how many PASS variants?” in seconds without writing a program, and it streams files of any size.

How do I avoid deleting something important?

Keep raw data in data/ and make it read-only with chmod a-w data/*. Use rm -i, check wildcards with ls before rm, and keep a backup of raw data somewhere else.

How do I write this in my Methods?

Name the system, the shell and any non-standard tools, and put your scripts in a public repository. A template:

“Data processing was performed on Ubuntu 22.04 (under WSL2) using GNU bash 5.1 and GNU coreutils 8.32, with grep, awk and gzip. All commands are available as scripts at [GitHub link].” Replace the versions with the ones you actually used (bash --version, ls --version, grep --version).

Glossary

Term	Meaning
Terminal	The window where you type commands
Shell (Bash, zsh)	The program that reads and runs your commands
Prompt	The text (ending in $) that shows the shell is ready
Path	The address of a file; absolute from /, relative from where you are
Option / flag	A setting that changes a command, written with a dash (-l, --help)
Argument	What a command works on, such as a file name or a pattern
Pipe (|)	Sends one command’s output into the next command
Redirection	Sends output to a file (>, >>, 2>) or reads input from one (<)
stdin / stdout / stderr	The standard input, output and error streams of every program
Exit code	A number a command returns: 0 for success
Wildcard (glob)	A pattern such as *.fa that the shell expands to matching file names
Regular expression	A search pattern for text, such as ^> (“lines starting with >”)
Script	A text file of commands that runs as one program

References

1. Pearson WR, Lipman DJ (1988). Improved tools for biological sequence comparison. PNAS 85(8):2444–2448. doi:10.1073/pnas.85.8.2444
2. Ewing B, Green P (1998). Base-calling of automated sequencer traces using phred. II. Error probabilities. Genome Res 8(3):186–194. doi:10.1101/gr.8.3.186
3. Kent WJ, et al. (2002). The human genome browser at UCSC. Genome Res 12(6):996–1006. doi:10.1101/gr.229102
4. Noble WS (2009). A quick guide to organizing computational biology projects. PLoS Comput Biol 5(7):e1000424. doi:10.1371/journal.pcbi.1000424
5. Li H, et al. (2009). The Sequence Alignment/Map format and SAMtools. Bioinformatics 25(16):2078–2079. doi:10.1093/bioinformatics/btp352
6. Cock PJA, et al. (2010). The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Res 38(6):1767–1771. doi:10.1093/nar/gkp1137
7. Quinlan AR, Hall IM (2010). BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics 26(6):841–842. doi:10.1093/bioinformatics/btq033
8. Danecek P, et al. (2011). The variant call format and VCFtools. Bioinformatics 27(15):2156–2158. doi:10.1093/bioinformatics/btr330
9. Leinonen R, Sugawara H, Shumway M (2011). The Sequence Read Archive. Nucleic Acids Res 39(Database issue):D19–D21. doi:10.1093/nar/gkq1019
10. Sandve GK, et al. (2013). Ten simple rules for reproducible computational research. PLoS Comput Biol 9(10):e1003285. doi:10.1371/journal.pcbi.1003285
11. Wilson G, et al. (2014). Best practices for scientific computing. PLoS Biol 12(1):e1001745. doi:10.1371/journal.pbio.1001745
12. Taschuk M, Wilson G (2017). Ten simple rules for making research software more robust. PLoS Comput Biol 13(4):e1005412. doi:10.1371/journal.pcbi.1005412
13. Wilson G, et al. (2017). Good enough practices in scientific computing. PLoS Comput Biol 13(6):e1005510. doi:10.1371/journal.pcbi.1005510
14. Grüning B, et al. (2018). Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods 15:475–476. doi:10.1038/s41592-018-0046-7
15. Danecek P, et al. (2021). Twelve years of SAMtools and BCFtools. GigaScience 10(2):giab008. doi:10.1093/gigascience/giab008
16. GNU Coreutils manual. gnu.org/software/coreutils/manual
17. GNU Awk user’s guide. gnu.org/software/gawk/manual
18. GNU Bash reference manual. gnu.org/software/bash/manual
19. Microsoft. Install Linux on Windows with WSL. learn.microsoft.com/en-us/windows/wsl/install

What to learn next

You can now move around a Linux system, inspect every common bioinformatics file, and automate work with pipes, loops and scripts. Next steps:

• Check your sequencing data: Quality Control of NGS Reads with FastQC and MultiQC uses everything from this tutorial on real reads.
• Coming next in the series: Conda, Bioconda and environments (installing tools), and Bash scripting and loops for pipelines.
• Full guided path: our Introduction to NGS Sequencing & QC course, then Whole Exome Sequencing (WES) Data Analysis or RNA-seq Data Analysis.
• Browse all free tutorials on the Bioinformatics Tutorials page.