More details on the ALICE DAQ

The ALICE Data-Acquisition System (DAQ) system handles the stream of experimental data created by the ALICE detectors until it is published for Offline processing via the Grid.

The 19 ALICE detectors, all located in the underground area of the LHC Interaction Point 2, send their data over optical fibers using specialized point-to-point links, the Detector Data Links (DDLs), to the Data Acquisition farm, which is located near the surface, where the data is received and formatted by a first layer of computers, the Local Data Concentrators (LDCs). When the data is accepted by the High Level Trigger (HLT), the LDCs send it to a second group of computers, the Global Data Collectors (GDCs), which put together all the contributions to create the so-called events, i.e. self-contained blocks of data related to the same collision(s). These events are stored on local disks (the Transient Data Storage - TDS) in files encoded using the AliROOT format, adopted for the Offline data processing. The data files are finally transferred to the CERN Computing Centre, and are ready to be published on the Grid.

A single DDL can transfer data between detectors and DAQ at a speed of up to 6 Gigabits per second, and there are about 500 of such links going into the ALICE DAQ system. The global peak data rate from the ALICE detectors to the LDCs using the architecture that is currently deployed at the ALICE experimental area can exceed 13 Gigabytes per second. This information is then reduced, compressed, filtered, and formatted for a peak output data rate of ~5 Gigabytes per second. One of the main challenges for the ALICE DAQ system is to constantly sustain the recording and publication of an average 2.5 Gigabytes per second during periods of ~28 consecutive days (the so-called Heavy Ion runs). This is equivalent to a DVD being written every two seconds for four consecutive weeks.

The computer farm supporting the ALICE data stream is mainly composed of LDCs and GDCs plus their interconnecting network. The LDCs are machines specialized in the handling of DDLs: they receive up to six links from the ALICE detectors and store the data in their memory, waiting for the decision from HLT. If the data is accepted, the LDCs transfer it to the GDCs, where the events are assembled, formatted, and recorded in the TDS. The ~150 LDCs and the 15 GDCs are all Intel-based rack-mounted PCs running Linux (distribution Scientific Linux CERN - SLC). Their interconnection is based on standard Ethernet using link speeds of one Gigabit, ten Gigabits, and forty Gigabits per second.

<<< Photo of CR1 >>>

When transferred through the DAQ, selected portions of experimental data are sent to dedicated computers supporting multiple data processing modules (Detector Algorithms - DAs and Data Quality Monitorings - DQMs). The online data is used there to achieve detector-specific functions (e.g. to improve the detectors' parameters) and to monitor on-the-fly the quality of events coming from the front-ends.

One of the major challenges of the ALICE DAQ is to handle the huge amount of data (~5 Gigabytes per second) generated by the ALICE detectors during the Heavy-Ion data-taking periods. The ALICE readout must be performed without introducing significant delays to avoid slowing down the data acquisition chain. The data must then be published on the Grid, whose entry points are not optimized for online data transfers and are physically located away (several kilometers) from the ALICE experimental area. As this procedure generates even more delays, a pool of high-speed disks (the TDS) is needed to record the data locally at the ALICE experimental site and de-couple the online and offline domains. The TDS also offers enough disk space (about 0.8 Petabytes) to keep the ALICE experiment running in the case of a partial or total temporary failure of the migration procedures. The disks are accessed via FibreChannel over optical links running at forty and ten Gigabits per second. A small set of specialized hosts (the so-called Movers) takes care of the migration of the data sets from the ALICE experimental area to the CERN Computing Centre: this is done via multiple—parallel and redundant—Ethernet links.

 

With its 19 separate detectors, 500+ data links, very sophisticated trigger system, and multiple running modes (Cosmic rays detection, proton physics, heavy-ion collisions), the ALICE experiment is a very complex machine that needs sophisticated configuration and monitoring facilities. The ALICE DAQ provides several dedicated tools to accomplish these functions: the ALICE Configuration Tool (ACT) to handle the various configurations, the ALICE Logbook to record all the details concerning the data taking and the experiment's conditions, the Monitoring ALICE Dataflow (MAD) facility to check online the status of the data streams, the Online Event Display to have a first-hand look at the reconstructed events, etc.

 

The ALICE DAQ is also in charge of the ALICE RunControl Centre, the operational base for all the control and acquisition activities performed at the ALICE experimental area. About 50 workstations and 15 status displays are available night and day to support shifters, run coordinators, and system experts in the accomplishment of their duties.

==========================================================================

---- TAB: Technical details ----

- Optical fibers from the LHC underground area to the DAQ: ~1300.

- Optical fibers between DAQ and HLT: ~1200.

- DDL links from the detectors: ~500.

- DDL links to the HLT: ~500.

- CRORCs: ~60 cards.

- D-RORCs: ~250 cards.

- LDCs: ~150 hosts.
... Models: DELL PowerEdge 720, SuperMicro X9SRE-F.
... RAM: 64 GB or 16 GB.
... CPU: Intel(R) Xeon(R) @ 2.5 GHz, 1 or 2 CPUs/LDC, 1 or 8 cores/host.

- GDCs: 15 hosts.
... Model: DELL PowerEdge 720.
... RAM: 64 GB.
... CPU: Intel(R) Xeon(R) @ 2.9 GHz, 2 CPUs/GDC, 8 cores/host.

- DQM/DAs: 10 hosts.
... Model: DELL PowerEdge 720.
... RAM: 64 GB.
... CPU: Intel(R) Xeon(R) @ 2.4 GHz, 2 CPUs/host, 8 cores/host.

- Servers: 15 hosts.
... Models: DELL PowerEdge 720, HP Proliant DL360p Gen8.
... RAM: 64 GB.
... CPUs: Intel(R) Xeon(R) or Intel(R) E5-2643 v2 @ 2.4, 2.9 or 3.5 GHz, 8 or 12 cores/host.

- Ethernet Network: 17 switches.
... Models: DELL S6000, DELL Force10 S4810, DELL S55.
... Ports: ~850 (electric and optical) at 40, 10 or 10/1 Gbps.

- FibreChannel: 2 switches.
... Model: Brocade 6520.
... Ports: ~200 at 16 GBps.

- TDS: 10 arrays and 9 extensions DELL PowerVault MD3660f/MD3660e.
... Hard disks: 1140 of 900 GB each.
... Total available space: ~1 PB.
... Total usable space: ~800 TB.

- Movers: 10 hosts.
... Model: DELL PowerEdge 720.
... RAM: 64 GB.
... CPU: Intel(R) Xeon(R) @ 2.5 GHz, 2 CPUs/mover, 8 cores/host.

==========================================================================

---- TAB: DAQ running figures ----

- Maximum incoming data throughput: 13 GB/s.

- Online data handling rate: minimum 5 GB/s at ~100 Hz.

- Target sustained data handling rate: 2.5 GB/s for 28 days.

- Power consumption: 46 KW.

- Cooling: up to 12 KW/rack, total maximum: 500 KW.

- UPS: 500 KW for a maximum autonomy at full load of 10 minutes.

==========================================================================