MSDT File Format

MSDT is a standardized, schema-rich representation for large-scale AI-driven proteomics. It provides a compact and structured abstraction of tandem mass spectrometry data for storage, exchange, reuse and machine learning.

MSDT v2.0 was released on September 2, 2026. Key updates include:

Batch generation of AI-ready MSDT datasets.
FragPipe-derived MSDT support for SCIEX data.
New fields (`score`, `q-value`, and `PEP`) for FragPipe-derived MSDT (MSDT Schema for details).
Workflow-aware processing through command-line FragPipe workflow integration.

1Overview

MSDT is a logical data representation derived deterministically from raw mass spectrometry data and associated identification results. It complements raw standards such as mzML with an AI-friendly representation optimized for downstream modeling and benchmarking.

MSDT files can be generated using the open-source MSDT-Converter tool.

2Scope and Non-goals

Designed to

Represent MS/MS spectra in a structured, tabular form.
Provide a reusable data layer between raw files and AI models.
Support dataset construction, benchmarking and cross-project reuse.

Not intended to

Replace mzML or vendor-specific raw formats as archival standards.
Preserve every instrument-specific acquisition parameter.
Serve as an execution-time format for highly demanding access patterns.

3Conceptual Data Model

At the conceptual level, MSDT represents MS/MS data as a collection of independent spectra, where:

Each row corresponds to a single MS/MS spectrum.
Each spectrum is associated with:

    * Precursor information (m/z, charge, retention time).
    * Fragment ion peak lists.
    * Optional identification-related annotations.

All entries in MSDT are derived deterministically from raw spectra and, where applicable, database search results.

4Physical Storage Format

MSDT is physically stored using the Apache Parquet format, a columnar data storage standard that provides:

Explicit schema definition and validation.
Efficient compression and encoding.
Fast column-wise access and slicing.

Parquet was selected to enable scalable dataset construction and efficient downstream processing while maintaining strong compatibility with common data processing ecosystems.

5Versioning and Compatibility

The current specification defines MSDT v1.0.
Minor v1.x updates may add optional fields but must preserve backward compatibility.
Major versions may introduce incompatible schema changes and are explicitly versioned.

6MSDT v1.0 Schema

The schema contains required fields for spectrum reconstruction and optional fields for identification and annotation workflows.

Run-Level FDR-Filtered MSDT Globally FDR-Filtered MSDT

Field	Status	Description
scan_id	Required	Unique identifier for the MS/MS spectrum.
precursor_mz	Required	Precursor mass-to-charge ratio in Thomson.
precursor_charge	Required	Precursor charge state.
retention_time	Required	Chromatographic retention time.
mz_array	Required	Fragment ion m/z values.
intensity_array	Required	Fragment ion intensity values aligned with `mz_array`.
peptide_sequence	Optional	Identified peptide sequence.
modifications	Optional	Peptide or protein terminal modifications.
protein_accessions	Optional	Associated protein accessions.
psm_score	Optional	Identification score from the search workflow.
q_value	Optional	FDR-derived q-value for the identification.

7Normalization and Conventions

m/z values are reported in Thomson (Th).
Intensity arrays may contain normalized or raw intensities, depending on dataset construction.
Missing values are explicitly encoded and do not imply absence of the corresponding entity.

8Minimal Working Example

A minimal MSDT table contains one row per MS/MS spectrum, required precursor metadata and fragment ion peak lists stored as array-typed columns. This representation is sufficient for a wide range of AI-based proteomics tasks.

9Relationship to Other Formats

mzMLMSDT is derived from mzML but does not preserve full raw-data fidelity.MGFMSDT generalizes spectrum-level representations beyond flat text.LMDBExecution-layer formats may be derived from MSDT but are outside this specification.

10Intended Use and Future Extensions

MSDT is designed to be extensible. New fields may be introduced in future versions to support additional tasks or annotations while preserving compatibility within major versions. It provides a stable foundation for AI-driven proteomics research and benchmarking.