Analytical Cytometry Standard (ACS) - NetCDF Component

NetCDF Conventions for List Mode Binary Data File

Introduction

We are proposing to address these shortcomings via series of related data standard proposals. Different components describing analytical cytometry experiments would be stored reusing standard formats such as XML, RDF, and others, and all the components would be bundled together into an Analytical Cytometry Standard (ACS) container.

The netCDF proposal demonstrates how the Network Common Data Form (netCDF) data format could be used to capture the most important component of FCS files: the recordings of the fluorescent or light-scattering properties of hundred of thousands of individual particles. NetCDF is a set of freely available software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Many groups, including NASA and the Los Alamos National Laboratory, have adopted netCDF as a standard way to represent some forms of scientific data. Conventions have been developed on how to use netCDF to store data for climate and weather forecast, aviation research, geographic information systems, ocean and atmosphere research and descriptions of molecular dynamics simulations.

Document Status

The proposal is an unapproved draft of a proposed standard that is intended for a review by the International Society for Analytical Cytology (ISAC) Data Standards Task Force (ISAC DSTF). As such, this document is subject to change and must not be utilized for any conformance/compliance purposes.

What is NetCDF

NetCDF is a complex solution to create, store, access, and share array-oriented scientific data. It consists of a general data model, a file format, and a set of Application Programming Interfaces (APIs) and libraries for various programming languages. It allows for the storage and retrieval of data in the form of n-dimensional arrays. Auxiliary information about the data, such as variable labels or what units are used, may be stored with the data. Generic utilities and application programs can access netCDF datasets and combine, analyze, or display specified fields of the data. The development of such applications has led to improved accessibility of data and improved re-usability of software for array-oriented data management, analysis, and display.

The netCDF format is:

Portable and platform-independent – the netCDF file format is XDR-based21 so that data written on one platform can be read on other platforms.
Fast with within file access – the netCDF file format uses indexing structures so that small subsets of a large dataset may be accessed efficiently, without first reading through all the preceding data.
Appendable - data may be efficiently appended to a netCDF file without copying the dataset or redefining its structure.
Self-Describing – a netCDF file may include metadata such as description of variables and dimensions, or units of measurements. Not all options should be part of the ISAC convention of use.
Sharable – one writer and multiple readers may simultaneously access the same netCDF file. With Parallel netCDF, even multiple writers may efficiently and concurrently write into the same netCDF file.
Extensible: Adding new dimensions, variables, or attributes to netCDF files does not require changes to existing programs that read the files.
Archivable: Access to all earlier forms of netCDF data are promised to be supported by current and future versions of the software.

Available NetCDF Software

The netCDF library has been designed to read and write data that has been structured according to well-defined rules and has been ported across various computer platforms. The basic distribution contains the C, C++, F77, and F90 libraries. Further libraries can be downloaded for Ada, IDL, Java, MATLAB, Objective-C, Perl, Python, R, Ruby, and Tcl/Tk. More than 80 independent software applications have been developed for manipulating or displaying netCDF data.

List Mode Data Converter

We have developed a List Mode Data Converter that performs conversion between files conforming to the Data File Standard for Flow Cytometry ( FCS) and the Network Common Data Form ( NetCDF); Conventions for List Mode Binary Data File. Follow this link to try the List Mode Data Converter.

Proposal download

For more information about netCDF and how we propose to use it to store list mode data files, please download the conventions document in our ACS download section at sourceforge.net.

The latest version of the PDF can also be downloaded from http://flowcyt.sourceforge.net/netcdf/latest.pdf.

Last update: December 6th, 2007