The site's hierarchy

Home page :: Private area :: Site Map :: Credits
Science / Simulations / Cosmological simulations / The MareNostrum Simulation


Mar 2009
  • CINES received a 150 TFlops machine
  • GENCI has acquired for the French University Supercomputing Center a 12288 core SGI Altix. It is ranked 14th in the World and first in France.
  • Site Web du CINES
  • IDRIS received a 140 Tflops BlueGene/P machine.
  • The CNRS supercomputing centre has acquired a new BlueGene machine with 40480 processors and 20 To of RAM memory.
  • See the press release.
  • Mare Nostrum simulation down to z=1.55
  • We carried out our fourth run of MareNostrum down to z=1.55 while producing 150 Million "stars". More info is given here
  • Horizon 4Pi 4096^3 down to z=0
  • We have carried a full sky 2Gpc/h 4096^3 Dark Matter simulation down to z=0. More details can be found here
  • Horizon 4Pi run 2048^3 completed
  • We have completed a full sky cone run of DM with RAMSES on platine CEA with 2048^3 particles. See a slice here
  • Mare Nostrum @ z=1.9
  • We carried out our third run of MareNostrum down to z=1.9 while producing 120 Million "stars"
  • Mare Nostrum simulation down to z=2.4
  • We carried our second run on the Mare Nostrum supercomputing center down to z=2.4, thanks to 4-fold improvement in overall performance.
  • 1024^3 500 h-1Mpc simulation completed
  • RAMSES run of the HORIZON initial condition was carried down to z=0 using 1024^3 dark matter particles in a (500 Mpc/h)^3 box with up to 16 levels of refinement (corresponding to local effective resolution of 65536).
  • Horizon 1024^3 100 Mpc/h simulation completed.
  • A RAMSES run of the HORIZON initial condition was carried down to z=0 using 1024^3 dark matter particles in a (100 Mpc/h)^3 box with up to 16 levels of refinement (corresponding to local effective resolution of 65536).
  • A 43 Tflops supercomputer at CEA
  • CCRT, the supercomputing centre of CEA, has just received a new BULL supercomputer with 6800 processors and 13.6 Tb of RAM memory. This facility is dedicated to academic and industrial research.
  • Lire le communiqué
  • Fourth Horizon Workshop 2006b
  • The 4th workshop takes place from December, the 11th to December, the 12th at Paris Observatory. Registration form available here.
  • MareNostrum simulation has started at BSC
  • After 1 week of operations, we have reached redshift 4. The simulation will proceed further more during the next quarters. We have started to post-process these initial data.
  • Horizon Grid
  • The Horizon grid is now in operation. It links the 6 quadri opteron of Meudon, Paris, Lyon, Saclay, IAP and Marseille and is open to all members of the collaboration. Voir ce lien
  • Third Horizon Workshop 2006a
  • Horizon Workshop 2006a 10, 11 and 12 april 2006 in Lyon. It will be dedicated to progress reports. Registration form
  • Horizon is part of the DEISA "Extreme Computing Initiative"
  • Horizon will launch one extreme application for galaxy formation using the DEISA infrastructure at Mare Nostrum
  • DEISA web site
  • Second Horizon Workshop
  • Horizon Workshop 2005 took place at Paris Observatory the 14th and 15th november. It was dedicated to internal discussions and management issues.
  • November 2005: Horizon méso-machine is fully operational at HPC1
  • October 2005: Horizon was awarded a 500 k€ grant by ANR, the French Science Foundation.
  • Horizon response to «ANR Blanche» Call for Proposals has been succesfull in the «UNIVERS» program.
  • ANR web site
  • July 2005: HP France was selected to host the Horizon "meso-computer"
  • HP company has succesfully answered the "call for proposals" issued by the Hrizon Project for a medium-size dedicated super-computer. The computer will be hosted at HPC1, the HP high performance computing center near Paris.
  • February 2005: INSU and CEA agree to fund the Horizon "meso-machine".
  • Horizon has submitted to INSU a proposal for funding a medium-size computer dedicated to the project. The proposal was accepted as a joint CEA and INSU operation.
  • September 2004: Kick-off meeting in Paris
  • During 3 days (13, 14 and 15 of september), 30 scientists have met at Observatoire de Paris to set up the basic organisation and objectives of the Horizon Project.
  • KO Meeting
  • June 2004: Project report at SF2A in Paris
  • April 2004: Horizon received official support from ASSNA.
  • The french initiative «Action Spécifique pour les Simulations Numériques en Astrophysique» have given to the Horizon project its "label" for outstanding and structuring computational project.
  • April 2004: The french Astroparticule program provides financial support for the Horizon "mini-grid"
  • The Programme Astro-Particule, a joint IN2P3 and INSU initiative, has agreed to finance 6 quad AMD64 servers dedicated to the Horizon Project.
  • January 2004: Review by PNC and PNG
  • Both PNG and PNC scientific comitees are officially supporting the Horizon Project initiative.
  • June 2003: Presentation of the project to SF2A
  • Theoretical Virtual Observatory workshop (IAP April 5-6, 2006)
  • A workshop dedicated to the Theoretical Virtual Observatory will take place at IAP on April 5-6th.

    The goal is to bring together experts of the Virtual Observatory and theoreticians who would like to make results of their simulations (e.g. databases or catalogs) or numerical codes available to the worldwild astronomical community.

  • Program and speakers

The MareNostrum Simulation

From large scale structure to galactic physics.

Voir aussi une version differente en francais en bas de page

JPG - 108.1 kb
Composite image of MareNostrum
red: dark matter gree: gas density blue: temperature. also available at full res here and in false colour

According to the Bible, the universe was created in about a week. Astrophysicists are currently building a virtual universe that will be completed in about four months, using 2048 processors of the MareNostrum supercomputer. Hosted by the Barcelona Supercomputing Center, this 10,240-processor IBM machine is able to perform more than 94 trillion operations per second. This unique facility, the largest in Europe and ironically located inside an old chapel, is the perfect place to compute the formation and evolution of a virtual replica of our own universe.

JPG - 841.9 kb
MN density at z=1.6
Here the projected density is shifted in x by half a boxsize to place the large cluster in the center.

The latest generation of astronomical instruments has allowed astronomers to have a clear view of the universe at its infancy, based on the so-called "cosmic microwave background," as well as a very detailed knowledge of the universe at present, in its fully adult, grown-up age. In order to fill the gap in between, and to prepare for the next generation of astronomical instruments, astrophysicists from all over Europe have gathered in Barcelona to run a single application that can compute the evolution of large scale structures in the universe.

JPG - 683.9 kb
A filament in light at z=2.4
See also the galaxies (to be compared to observed galaxies) and groups porfolio; or the physical properties A poster of a few galaxies is available here while a complete view of the fields at z=1.55 is available here

The MareNostrum galaxy formation project is a multidisciplinary collaboration between astrophysicists of France, Germany, Spain, Israel and USA, together with computer experts from IDRIS (Institut du Développement et des Ressources en Informatique Scientifique) and BSC (Barcelona Supercomputing Center). The application solves a very complex set of mathematical equations by translating them into sophisticated computational algorithms. These algorithms are based on state-of-the-art adaptive mesh refinement techniques and advanced programming technologies in order to optimize the timely execution of the same application on several thousands of processors in parallel.

JPG - 3 Mb
50 h-1 gas at z=2.5
1024^3 initial grid; 4 levels of refinements (c) 2007 Pichon Teyssier

The simulation is now computing the evolution of a patch of our universe — a cubic box of 150 millions light years on a side — with unprecedented accuracy. It requires roughly 10 billion computational elements to describe the different kinds of matter that are believed to compose each individual galaxy of our universe: dark matter, gas and stars. This requires the combined power of 2048 PowerPC 970 MP processors and up to 3.2 TB of RAM memory. Contrary to other large computational problems in which information can be split into independent tasks, and because of the non-local nature of the physical processes we are dealing with, all 2048 processors have to exchange large amounts of data very frequently. To support this type of processing, the application takes advantage of the high bandwidth and low latency Myrinet interconnect installed on the MareNostrum computer. A personal computer, provided it had enough memory to store all the data, would need around 114 years to do the same task.

JPG - 4 Mb
50 h-1 temperature at z=2.4
1024^3 initial grid; 4 levels of refinements (c) 2007 Pichon Teyssier

In about four months, using almost one million CPU hours, several billions years of the history of the universe will be simulated. The simulation makes intensive use of the I/O sub-system. To allow such a huge simulation to run smoothly during weeks of computation and to get an optimal performance of the system, application tuning was required: the simulation package provides a restart mechanism that allowed for recovery and resumption of the computation. In this way, the application is able to deal with hardware failures without having to restart from the beginning. This mechanism requires around 30 TB of data to be written to save the application state. In order to minimize the Global File System contention, an optimized directory structure has been proposed, supporting a sustained "parallel write" performance of 1.6 Gbps. Other design aspects were taken into account in order to improve a massive "parallel read and broadcast" over the Myrinet network, in order to read and dispatch the initial condition data over all the processors.

JPG - 5 Mb
stars in MN at z=1.55
This is a zoom of the inner region. (c) C. Pichon R. Teyssier. See also the corresponding coloured image which represents all galaxies in the simulation at z=1.55 in a field of view of 50x50x800 Mpc/h at a resolution of 1.5kpc with ilfters in G I K.

As in computer-simulated movies, a large number of snapshots are stored in sequence in order to provide realistic animation. The total amount of scientific data generated will exceed 40 TB. This unique database will constitute a virtual universe that astrophysicists will explore in order to create mock observations and to shed light in the many different processes that gave birth to the galaxies, and in particular, our own Milky Way galaxy.

The first week of computation was performed last September, during which 34 snapshots were generated, producing more than 3 TB of data. During the entire week only two hours were lost because of a hardware failure. This was due to failure of a single compute node — out of the 2100 processors reserved for the run.

The MareNostrum virtual universe was evolved up to the age of 1.5 billion years. Astrophysicists believe that this is precisely the era of the formation for the first Milky Way-like galaxies. Researchers have detected roughly 50 such large objects, with more than 100,000 additional galaxies of smaller size in the simulated universe. They are currently analyzing their physical properties in the virtual catalogue, as well as preparing for the next rounds of computations that will be needed in order to complete the history of the virtual universe.

JPG - 939.8 kb
50 h-1 Mpc Dark Matter
1024^3 AMR simulation (c) Sousbie Pichon Teyssier 2007

One of the most important issues of numerical modelling of complex physical phenomena is the accuracy of the results. Unfortunately, the researchers cannot compare the results from the numerical simulations with laboratory experiments, like in other areas of computational fluid dynamics. Instead, the reliability of the simulations can be assessed by comparing results from different numerical codes starting from the same initial conditions. In this regard, the researchers are also simulating the MareNostrum Universe with a totally different numerical approach, using more than two billion particles to represent the different fluid components. This simulation is part of a long-term project named The MareNostrum Numerical Cosmology Project (MNCP). Its aim is to use the capabilities of MareNostrum supercomputer to perform simulations of the universe with unprecedented resolution.

JPG - 1.3 Mb
Mare Nostrum Multifield at z=1.55
From top to bottom Dark Matter, Gas, Temperature, Metallicity Stars with a factor X4 zoom from left to right. (c) Pichon Rasera Teyssier 2007.

This is the copy of an article written in collaboration between R. Teyssier, G. Yepes and BSC for HPCWire.

PDF - 969.1 kb
Article PLS Mare Nostrum
Cet article, en Francais decrit la simulation MareNostrum. Il a ete publie par Pour La Science
PDF - 904.1 kb
Article Science et Vie
description de la simulation MareNostrum