Try it: identify a file by its first bytes
The tool reads the first 32 bytes of a file in the browser and matches them against the signature table further down. Nothing is uploaded.
No file read yet. Only the first 32 bytes are examined and nothing leaves the device.
Ask three people what a file called report.doc is and the likely answers are “a Word document”, “a Word document” and “it depends”. The third person is right. File format identification is the work of finding out what a file really contains, and it starts by distrusting the name. The extension says what someone wanted the file to be; the bytes say what it is. This guide covers how those bytes are read, which tools read them, and how to write the answer down so that someone can rely on it in twenty years.
To test the idea on a real file, the tool above reads its first 32 bytes in the browser and compares them with a small set of known signatures. The full list it checks is in the signature table at the end of this page.
Why the file extension misleads
An extension is three or four characters after a dot in the file name. It lives in the file system, not in the file, and every copy, rename, download or email can change it. Most of the time it happens to be correct, which is exactly why it causes trouble when it is not.
The failure modes repeat across every collection that has been around long enough:
- Renamed files. A user changes
photo.pngtophoto.jpgbecause a web form only accepts JPEG. The file is still a PNG. - Lost extensions. Files recovered from old Mac volumes, mainframe exports or email attachments often arrive with no extension at all.
- Shared extensions.
.doccovers the binary Word format of Word 97–2003, older Word versions, plain text saved by other programs and WordPerfect files renamed by hand..datand.binsay almost nothing. - Version blindness.
.pdfdoes not distinguish PDF 1.4 from PDF 2.0 or from PDF/A, and.tifdoes not reveal whether the image is compressed with LZW, JPEG or nothing at all.
For a desktop user, a wrong extension is a nuisance. For an archive deciding which files need migrating, or for a server deciding whether an uploaded file is safe to process, it can mean the wrong decision applied to thousands of objects. That is why every serious approach to file format identification reads the content.
Magic bytes: the signature at the start of a file
Many formats begin with a fixed sequence of bytes chosen by the people who designed them. These sequences are called magic numbers or magic bytes, and a byte pattern that identifies a format is called a file signature. They are written in hexadecimal, two digits per byte, because many of them are not printable characters.
Here are the ones most people meet first:
PNG 89 50 4E 47 0D 0A 1A 0A .PNG....
PDF 25 50 44 46 2D %PDF-
ZIP 50 4B 03 04 PK..
GIF 47 49 46 38 37 61 GIF87a
47 49 46 38 39 61 GIF89a
JPEG FF D8 FF ...
The PNG signature is a small piece of engineering. The first byte, 89, has its high bit set, so a transfer that strips the eighth bit corrupts it visibly. 50 4E 47 spells “PNG”. 0D 0A is a Windows line ending and the later 0A a Unix one, so a transfer that converts line endings changes the signature and the damage is detected before the image is decoded. 1A is the old DOS end-of-file character, which stops the type command from printing binary data to the screen.
PDF is friendlier to the eye: %PDF- is followed by a version number such as 1.7 or 2.0, so the first eight bytes often name both the format and its version. One complication matters for identification software. Readers tolerate junk before the header, and the header may appear a short distance into the file; a strict “byte 0 only” rule for these magic bytes misses some real PDFs.
GIF also carries its version in the signature (GIF87a or GIF89a). JPEG starts with FF D8, the Start of Image marker, followed by another marker that begins with FF, which is why most tools test the three bytes FF D8 FF. The fourth byte hints at the flavor: E0 usually introduces a JFIF segment, E1 an Exif segment from a camera.
ZIP begins with PK (50 4B), the initials of Phil Katz, who wrote PKZIP, followed by 03 04, which marks a local file header. As later sections show, that signature is where identification of a large group of formats starts, not where it ends.
Magic bytes are evidence, not proof. Nothing stops a program or a person from writing %PDF at the start of a file that is not a PDF, and so-called polyglot files are built to satisfy two formats at once. For honest files, though, they are fast and very reliable.
Offsets: when the signature is not at byte zero
Some formats put a generic wrapper at the start and the specific identifier a few bytes later. The position of a signature, counted in bytes from the start of the file, is its offset. Counting starts at 0.
The RIFF family is the standard example. A RIFF file begins with the four characters RIFF, then a 32-bit size field stored in little-endian byte order, then a four-character form type at offset 8:
offset 0 4 8
RIFF (file size) WAVE -> WAVE audio
RIFF (file size) WEBP -> WebP image
RIFF (file size) AVI -> AVI video (the code is "AVI" plus a space)
Those form types are four-character codes, four ASCII bytes that read as a word in a hex dump. A tool that only checks RIFF at offset 0 can say “some RIFF file” and nothing more. Checking offset 8 tells sound, images and video apart.
The ISO base media file format (ISO/IEC 14496-12), which underlies MP4, QuickTime MOV, M4A, 3GP, HEIC and AVIF, works the same way with different numbers. The file is a sequence of boxes, each beginning with a four-byte length and a four-character type. The first box is normally ftyp, so the string ftyp appears at offset 4, after the length. The next four bytes, the major brand, narrow it down: isom or mp42 for MP4, M4A for audio, qt for QuickTime, heic for HEIF images, avif for AVIF. Because the length field varies, a signature for these formats has to ignore bytes 0 to 3 and test from byte 4.
Other formats sit further in. TAR archives written in the POSIX ustar format carry the string ustar at offset 257. ISO 9660 CD images carry CD001 at offset 32769, well past the first sector. Any identification rule that reads only the first few bytes misses both.
End-of-file signatures
A signature can also sit at the end. JPEG files close with FF D9, the End of Image marker. PDF files end with %%EOF, usually preceded by the startxref keyword and a byte offset pointing back to the cross-reference table. A ZIP archive keeps its central directory at the end, closed by an end of central directory record beginning 50 4B 05 06.
These trailing markers do two jobs. They strengthen identification, since a file that starts like a JPEG and ends like one is more convincing than a file that only starts like one. And they expose truncation: a JPEG missing its FF D9 is very often a file that was cut off during transfer, which matters more to an archive than the format name does. The PRONOM signature language used by DROID and Siegfried can anchor a byte sequence to the end of the file (an “EOF” offset), so a format entry can require both ends to match.
Trailing data is not always damage. Some cameras and editing programs append thumbnails or extra metadata after the JPEG end marker, and incremental updates add new sections after the first %%EOF of a PDF, so a PDF can legitimately contain several of them. A sound workflow treats a missing end marker as a flag for review, not an automatic rejection.
Containers: one file signature, many formats
A large share of modern formats are packages: a ZIP archive or an older compound file holding several internal parts. For these, the outer file signature identifies the packaging and nothing else. File format identification has to open the container and look inside.
ZIP-based formats. Office Open XML (DOCX, XLSX, PPTX), OpenDocument (ODT, ODS, ODP), EPUB, Java JAR files and many others all begin with 50 4B 03 04. They are told apart by their contents:
- Office Open XML packages contain a part called
[Content_Types].xmland a main part such asword/document.xml,xl/workbook.xmlorppt/presentation.xml. - OpenDocument and EPUB files store a first entry called
mimetype, uncompressed, whose content is the media type of the package. In an EPUB, the file namemimetypebegins at offset 30, directly after the 30-byte local file header, and the stringapplication/epub+zipfollows immediately after it. OpenDocument text documents normally putapplication/vnd.oasis.opendocument.textin the same place.
That fixed mimetype entry is a deliberate design choice: it lets simple byte-level tools identify the package without unzipping it.
OLE2 compound files. Before the XML formats, Microsoft Office used the Compound File Binary Format, a small file system inside a file. Every such file begins with D0 CF 11 E0 A1 B1 1A E1. Word 97–2003 documents, Excel workbooks, PowerPoint presentations, Outlook .msg messages and a number of non-Microsoft formats all share that header. Telling them apart means reading the internal directory and finding streams such as WordDocument, Workbook or PowerPoint Document.
DROID and Siegfried handle both families with a second set of rules, the PRONOM container signature file, which describes what must be present inside a ZIP or OLE2 container for each format. A tool that has only the byte-level signature file reports a DOCX as a ZIP archive, which is technically true and practically useless.
Text formats without a signature
Plain text, CSV, TSV, JSON, Markdown, source code and many log formats have no required header at all. Their first bytes are simply the first characters of the content, which could be anything. File format identification for them is heuristic, because there are no magic bytes to find.
The one reliable byte-level clue is the byte order mark (BOM), an optional marker at the start of some Unicode text:
EF BB BF UTF-8 (the BOM is optional and often absent)
FF FE UTF-16, little-endian
FE FF UTF-16, big-endian
A BOM says how the characters are encoded. It says nothing about what the text is. A UTF-8 file with a BOM can be a CSV export from a spreadsheet, a subtitle file or a Python script.
Beyond that, tools fall back on patterns. A file that is valid UTF-8 or ASCII with no control characters is probably text. Consistent counts of commas or tabs per line suggest CSV or TSV. A leading <?xml declaration identifies XML, and the root element plus its namespace often identifies the specific XML format (an XHTML page, an SVG drawing, a METS document). HTML is recognized by tags such as <!DOCTYPE html> or <html near the start; the WHATWG MIME Sniffing standard describes exactly how browsers do it, and the X-Content-Type-Options: nosniff response header exists to switch that behavior off on servers that send correct types.
For these formats the extension regains some weight, and most tools combine it with the content tests. DROID, for example, records whether a match came from a signature, a container rule or the extension alone. An extension-only match on a CSV file is normal. An extension-only match on a file that claims to be a TIFF is a warning.
Tools for file format identification
Several mature tools handle file format identification at scale, and they differ more in scope than in method.
The file command and libmagic. On Linux, macOS and the BSDs, file is the quickest check there is. It reads the file against a magic database of byte patterns, offsets and string rules maintained alongside the program, then falls back to text heuristics. file --mime-type report.doc prints a media type; file -b drops the file name from the output. The libmagic library behind it is embedded in many other programs, including scripting-language bindings used in upload filters.
DROID. The Digital Record Object Identification tool from The National Archives UK is the reference implementation for the PRONOM registry. It runs as a desktop application or from the command line, scans whole directory trees, and exports a profile listing each file’s PUID, format name, version, media type, identification method and any extension mismatch. It uses two signature files from PRONOM: the binary signature file and the container signature file.
Siegfried. Siegfried, written by Richard Lehane, is a separate command-line tool written in Go that uses the same PRONOM signatures and can also load the freedesktop.org MIME-info database and the Library of Congress format descriptions. It is fast on very large collections, outputs YAML, JSON, CSV or DROID-style CSV, and includes a basis field for every result explaining which bytes matched and where. That field is very useful when a result looks wrong.
Apache Tika. Tika is a Java toolkit used in search engines and content pipelines. Its detector combines magic-byte rules from its own media type database with container-aware detectors for ZIP and OLE2 packages and, optionally, the file name.
The PRONOM-based tools return PUIDs, persistent identifiers for each format version. PNG alone has three: fmt/11 for PNG 1.0, fmt/12 for 1.1 and fmt/13 for 1.2. Older entries carry an x-fmt/ prefix. PRONOM signature entries can declare that one format has priority over another, which is how a tool decides between a general match and a more specific one; when no priority rule applies, a file can receive several results, and someone has to look.
Media types and how to record the result
An identification result is only as useful as the record that keeps it. Two kinds of name are worth storing.
The first is the media type (formerly MIME type, and called an Internet Media Type or IMT in Dublin Core), from the IANA registry: image/png, application/pdf, audio/flac, application/vnd.openxmlformats-officedocument.wordprocessingml.document. Registration follows RFC 6838. Media types are what web servers, browsers and most software understand, and they are the recommended value for dc:format. In DCMI Metadata Terms, dcterms:format takes a value of the class dcterms:MediaType (or an extent), and the IMT encoding scheme states that the value comes from the IANA list. A digital file format in particular is modeled as dcterms:FileFormat.
Media types are coarse. application/pdf covers every PDF version and every PDF/A conformance level. So the second name is a registry identifier, normally a PUID, which pins down the version.
A practical record for a single file therefore holds:
- the media type, for example
image/png - the PUID, for example
fmt/13, with the format name and version it stands for - the tool name and version, and the signature file version it used
- the identification basis (signature, container or extension)
- the date of the identification
- a checksum of the file at that moment, such as SHA-224 or SHA-256
In Dublin Core the descriptive part might look like this:
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix dcterms: <http://purl.org/dc/terms/> .
<https://example.org/objects/0042/scan-001.png>
dc:format "image/png" ;
dcterms:format <http://www.nationalarchives.gov.uk/pronom/fmt/13> .
The process data (tool, version, signature file, date, basis) does not fit Dublin Core well. Preservation systems usually keep it in PREMIS, which models the result as format designation and format registry entries on the object and the run itself as a “format identification” event linked to the software agent that performed it. Without that event record, a result cannot be reproduced: a newer signature file can change the answer for the same bytes.
Identification is not validation
Identification answers one question: which format does this file appear to be in? It does not ask whether the file is any good.
Validation does. A validator reads the whole file and checks it against the format specification. JHOVE, maintained by the Open Preservation Foundation, has modules for formats such as PDF, TIFF, JPEG, JPEG 2000, GIF, WAVE and XML, and reports whether each file is well-formed (it follows the basic syntax) and valid (it also meets the higher-level rules). Specialized validators go further for single formats: veraPDF checks PDF/A conformance, for example.
The two steps answer different questions, and both answers can disagree with the extension. A file can be identified with confidence as PDF 1.4, from a perfectly good header, and still fail validation because its cross-reference table is broken.
A step-by-step identification guide for archive intake
The order in this part of the guide suits a typical transfer into an archive or repository: a donor’s hard drive, a project’s shared folder, an export from a records system. The details change with local systems; the sequence rarely does.
- Fix the bits first. Before touching anything, compute a checksum for every file and store the manifest with the transfer. Everything afterwards can be checked against it.
- Work on a copy. Run tools against a read-only copy or a mounted image, never the donor’s only original.
- Run a PRONOM-based identifier over everything. Siegfried or DROID, with current binary and container signature files, recording the tool and signature versions in the output.
- Sort the results. Separate files with a single signature or container match, files with multiple matches, files identified by extension only, and files with no identification at all. Note every extension mismatch.
- Look at the hard cases. Open unknown files in a hex viewer and compare their magic bytes with known signatures (the signature table on this page covers the common ones). Check context: the software the donor used, neighboring files, dates. Record any manual decision with the reason.
- Validate where it matters. Run JHOVE or a format-specific validator on formats the archive commits to preserving actively, starting with the ones that will be migrated.
- Record the result. Write media type, PUID, tool, version, basis and date into the preservation metadata, and the media type into the descriptive record.
- Plan a re-run. Files left as unknown today are worth trying again when PRONOM publishes new signatures, which happens several times a year.
Step 5 is where most of the time goes. In a typical mixed transfer, the large majority of files identify cleanly on the first pass; the remainder, often old word processor files, database fragments and anything with a .dat extension, take far more effort per file than everything else combined.
Signature table: magic bytes of common file formats
The signatures the tool above checks, with their offsets and registered media types. Offsets count from the first byte of the file, starting at 0.
| Format | Extension | Offset | Bytes (hex) | Media type |
|---|---|---|---|---|
| PNG image | .png | 0 | 89 50 4E 47 0D 0A 1A 0A | image/png |
| JPEG image | .jpg | 0 | FF D8 FF | image/jpeg |
| GIF image | .gif | 0 | 47 49 46 38 | image/gif |
| TIFF image, little-endian | .tif | 0 | 49 49 2A 00 | image/tiff |
| TIFF image, big-endian | .tif | 0 | 4D 4D 00 2A | image/tiff |
| WebP image | .webp | 8 | 57 45 42 50 | image/webp |
| PDF document | 0 | 25 50 44 46 | application/pdf | |
| Rich Text Format | .rtf | 0 | 7B 5C 72 74 66 31 | application/rtf |
| ZIP container (also DOCX, XLSX, EPUB, ODT) | .zip | 0 | 50 4B 03 04 | application/zip |
| 7-Zip archive | .7z | 0 | 37 7A BC AF 27 1C | application/x-7z-compressed |
| gzip stream | .gz | 0 | 1F 8B | application/gzip |
| WAVE audio | .wav | 8 | 57 41 56 45 | audio/wav |
| FLAC audio | .flac | 0 | 66 4C 61 43 | audio/flac |
| Ogg container | .ogg | 0 | 4F 67 67 53 | audio/ogg |
| MP3 audio with ID3 tag | .mp3 | 0 | 49 44 33 | audio/mpeg |
| ISO base media (MP4, MOV, M4A) | .mp4 | 4 | 66 74 79 70 | video/mp4 |
| Matroska / WebM | .mkv | 0 | 1A 45 DF A3 | video/x-matroska |
| SQLite database | .sqlite | 0 | 53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 | application/vnd.sqlite3 |
| HDF5 data | .h5 | 0 | 89 48 44 46 0D 0A 1A 0A | application/x-hdf5 |
Questions and answers
What is file format identification?
File format identification is the process of working out which format a file is actually written in, usually by reading characteristic byte sequences inside it. The result is a format name, often a version, a media type and, in archives, a registry identifier such as a PRONOM PUID.
What are magic bytes?
Magic bytes (or magic numbers) are a fixed sequence of bytes that a format places at a known position, usually the very start of the file. PNG files begin with 89 50 4E 47 0D 0A 1A 0A, PDF files with %PDF, ZIP archives with PK followed by 03 04.
Can magic bytes be faked?
Yes. Anyone can write any bytes at the start of a file, and polyglot files deliberately satisfy two formats at once. Magic bytes make honest files easy to recognize; they are not a security control. Upload filters that need certainty should parse or validate the whole file.
Why is a file extension not enough for file format identification?
Extensions are part of the file name, not the file. They get dropped by email gateways, changed by users, mangled by old operating systems, and shared by unrelated formats (.dat, .bin, .doc for both Word binary and Word XML). Identification reads content, so it survives renaming.
What is a PUID?
A PUID (PRONOM Unique Identifier) is the persistent identifier The National Archives UK assigns to each format version in its PRONOM registry, for example fmt/11, fmt/12 and fmt/13 for PNG 1.0, 1.1 and 1.2. Tools such as DROID and Siegfried report PUIDs as their result.
What is the difference between DROID and Siegfried?
Both use the PRONOM signature files and report PUIDs. DROID is the Java application from The National Archives UK with a graphical interface and profile exports. Siegfried is a separate command-line tool written in Go, fast on large collections, and it explains each match in a basis field.
How does the Unix file command identify files?
The file command, built on the libmagic library, tests the file against a magic database of byte patterns, offsets and string rules, then falls back to heuristics for text. file --mime-type prints a media type, which makes it handy in scripts and web upload checks.
How are DOCX, XLSX and EPUB told apart if they are all ZIP files?
They share the ZIP signature, so a tool has to look inside the archive. Office Open XML files contain [Content_Types].xml and a part such as word/document.xml; EPUB and OpenDocument files store a first entry named mimetype whose content names the exact format.
Do plain text and CSV files have magic bytes?
No. Text formats have no required header, so identification relies on heuristics: character encoding checks, line structure, delimiter counts and sometimes the extension. A byte order mark such as EF BB BF reveals the encoding, not whether the text is CSV, Markdown or source code.
What is the difference between identification and validation?
Identification names the format a file claims to be. Validation checks whether the file actually conforms to that format's specification. JHOVE, for instance, reports whether a PDF or TIFF is well-formed and valid; a file can be identified correctly as PDF and still fail validation.
Which media type should be recorded for a file?
Use the registered IANA media type where one exists, such as image/png or application/pdf. In Dublin Core it goes in dc:format or dcterms:format with the IMT encoding scheme. Avoid unregistered x- types when a registered one covers the format.
Why do identification tools sometimes return several matches?
Some signatures are generic enough to fit more than one format or version, and some files are genuinely ambiguous. When no priority rule separates the candidates, tools report all of them. A person then checks the file, its context and any container contents before choosing one.
Where is the end-of-file marker used in file format identification?
Some formats close with a fixed sequence: JPEG ends with FF D9, PDF with %%EOF, and a ZIP archive keeps its end of central directory record (50 4B 05 06) near the end. PRONOM signatures can test these end-of-file positions to separate complete files from truncated ones.
How often should a collection be re-identified?
Whenever the signature files change in a way that affects the collection, and at least before any migration project. PRONOM releases new signature files several times a year; files reported as unknown or ambiguous are the first candidates for a fresh run.
Should the identification tool and version be recorded?
Yes. The same file can get a different answer from a newer signature file, so a result without the tool name, tool version, signature file version and date cannot be reproduced or audited. PREMIS records this as a format identification event with a linked agent.