Part 4 — Files you will meet

Markdown, JSON, tables, and why text files misbehave

Most files that you work with are plain text: files that a person can read and any program can process. This part covers the formats that you will meet in every subject.

NoteThis page grows

These are the formats that you will meet first. We add other formats when they become useful. The page does not need to be complete to be useful.

Plain text, and what “plain” means

A plain text file contains characters and nothing else: no fonts, no colours, no hidden formatting. This is why it lasts. A text file from thirty years ago still opens today, and every programming language can read it without an extra library.

A .docx or .xlsx file is not plain text. It is a zip archive full of markup. This is why a program that expects text fails on it.

printf "just some text\n" > plain.txt
cat plain.txt
just some text

The file extension is a hint for you. It does not define the file. data.csv, data.txt and data can contain exactly the same bytes. If you rename a file, you do not convert it.

Markdown

Markdown (.md) is text with a few simple formatting marks that are still easy to read as text. READMEs, documentation and notes are written in Markdown, and GitHub shows it formatted on the main page of a repository.

cat > README.md <<'EOF'
# My project

A short description of what this does.

## Requirements

- Something you need
- Something else

Run it with `pixi run analysis`, and see [the notes](notes.md) for detail.

**Important:** the data is not in this repository.
EOF
head -n 3 README.md
# My project

A short description of what this does.

This is all the syntax you need at the start:

you write you get
# Title, ## Section headings, one # per level
*text* or _text_ italic
**text** bold
`code` inline code
- item a bullet list
1. item a numbered list
[label](target) a link
> quoted a quotation

Code with several lines goes in a block between three backticks. Name the language, so that the code is highlighted:

```python
print("hello")
```
TipEvery repository should have a README.md

It is the first thing that anyone sees, including you in a year. It needs four things: what the project is, what you need to run it, how to run it, and where the data is. That is enough. A short README that exists is better than a long one that does not.

NoteMarkdown is not one standard

Different tools support slightly different extensions. Tables, footnotes, task lists and strikethrough are common, but not every tool supports them. You will meet GitHub’s version most often. If something looks right on GitHub but not in another tool, this is usually the reason, and it is not your mistake.

JSON

Programs use JSON (.json) to pass structured data to each other. Many tools write a JSON report in addition to their output for people. It is usually the easiest file to get numbers from.

cat > report.json <<'EOF'
{
  "sample": "A1",
  "passed": true,
  "count": 1420,
  "score": 0.97,
  "flags": ["short", "duplicated"],
  "source": {
    "instrument": "bench-2",
    "operator": null
  }
}
EOF
cat report.json

JSON has six data types, and that is the whole language:

string "A1" — always double quotes, never single
number 1420, 0.97 — no quotes
boolean true / false — lowercase
null null — a value that is missing on purpose
array [...] — an ordered list
object {...} — named fields, like the whole file

An object can contain other objects. This is how JSON shows structure: source above is an object inside an object.

Reading it

Python reads JSON without any extra installation:

python3 -c "
import json
d = json.load(open('report.json'))
print(d['sample'], d['count'])
print(d['source']['instrument'])
print(len(d['flags']), 'flags')
"
A1 1420
bench-2
2 flags

Note how you read nested values: d['source']['instrument'] goes down one level at a time, in the same structure as the file.

Checking whether a file is valid

A JSON file is either valid or not, and one missing comma makes the whole file invalid. Instead of searching for the error by eye, run:

python3 -m json.tool report.json > /dev/null && echo "valid JSON"
valid JSON

This command also formats the file for reading. So it is the quickest way to make a long single-line file readable:

printf '{"a":1,"b":[2,3]}' > tight.json
python3 -m json.tool tight.json
{
    "a": 1,
    "b": [
        2,
        3
    ]
}
WarningThe three things that break JSON
  • A comma after the last item. [1, 2, 3,] is invalid. Most programming languages allow it, but JSON does not.
  • Single quotes. {'a': 1} is not JSON, it is Python. JSON needs "a".
  • Comments. JSON has no comments. A line with # or // makes the file invalid.

All three cause an error message with a line number. This line number is often after the real mistake, because the program stops reading only there.

Tables: CSV and TSV

Tables usually come as one row per line, with the columns separated by a character: a comma (CSV) or a tab (TSV).

cat > samples.csv <<'EOF'
id,group,value
A1,control,0.42
A2,control,0.51
B1,treated,0.88
EOF
column -s, -t samples.csv
id  group    value
A1  control  0.42
A2  control  0.51
B1  treated  0.88

The tools from Part 1 work directly on these files:

wc -l samples.csv
head -n 1 samples.csv
grep "treated" samples.csv
4 samples.csv
id,group,value
B1,treated,0.88

cut selects columns:

cut -d, -f2 samples.csv | sort | uniq -c
      1 group
      2 control
      1 treated

-d, sets the separator, and -f2 selects field 2. Note that the header was counted too. These tools see lines, not tables, and they do not know what a header is.

WarningPrefer TSV when you get the choice

Real data contains commas: in names, in descriptions, in any free text. CSV puts such a field in quotes. Then cut -d, gives a wrong result, because cut does not understand quotes.

Tabs almost never appear inside a value. So TSV avoids the problem completely.

ImportantNever open data in Excel “only to look at it”

Excel changes data without telling you, and when you save, the changes become permanent:

  • text that looks like a date becomes a date, and you cannot undo it
  • zeros at the start are removed, so 007 becomes 7
  • long identifiers become scientific notation
  • the separator and the encoding can change when you save

This problem is well documented, and it happens again and again in real published datasets. Use head, less or VS Code instead.

Two things that break text files

You cannot see either of them on the screen. This is why you should know them.

Line endings

Windows ends a line with two characters (\r\n). Linux and macOS use one (\n). So a file written on Windows and read on Linux can have an extra \r at the end of every line. You cannot see it, but it makes "control" and "control\r" different strings, and comparisons fail without an error message.

printf 'value\r\n' > windows.txt
cat -e windows.txt
value^M$

cat -e makes it visible: ^M is the carriage return, and $ is the end of the line. To fix it:

tr -d '\r' < windows.txt > fixed.txt
cat -e fixed.txt
value$
Tip

VS Code shows LF or CRLF in its status bar, and a click on it converts the file. So open a strange file in VS Code first.

Encoding

Text is stored as bytes, and an encoding defines which bytes mean which characters. Today, UTF-8 is used everywhere, and it covers every language and symbol.

You meet this problem when a name with an umlaut or an accent appears as ü or ?. The file was written in one encoding and read in another. Save the file as UTF-8, and the problem disappears.

TipThe safest habit

Use only plain ASCII in file names, column headers and identifiers: letters, digits, underscores and hyphens. Umlauts, spaces and accents in a file name are allowed, but sooner or later they break a tool that was not tested with them.

Inside the content of a document, use any characters you like. The names are what programs pass to each other.

What you should be able to do now

Where to go next

This is the end of the material. If you can do everything on the checklists at the end of all four parts, you have the basics that a computational practical assumes, and more than most people have at the start.

Do not memorise this material. Come back to it when you need it: that is what a reference is for.