• For Home
    • Mobile Storage
      • Ultra Dual Drive USB Type-C | SanDisk
      • iXpand Flash Drive for iPhone and iPad | SanDisk
        • iXpand Flash Drive | SanDisk
      • Ultra Dual USB Drive 3.0 | SanDisk
      • Connect Wireless Stick | SanDisk
      • Ultra USB Type-C Flash Drive | SanDisk
      • Ultra Dual Drive m3.0 | SanDisk
      • iXpand Base for iPhone | SanDisk
    • Cards & Readers
      • SD Cards
        • ExtremePro microSD
        • Extreme PRO SDHC/SDXC UHS-I Memory Card | SanDisk
        • Extreme PLUS SDHC/SDXC UHS-I Memory Card | SanDisk
        • Extreme SDHC/SDXC UHS-I Memory Card | SanDisk
        • Ultra PLUS SDHC/SDXC Memory Card | SanDisk
        • Ultra SDHC/SDXC Memory Card 80MB/s read speed | SanDisk
        • SDHC/SDXC Memory Card | SanDisk
      • microSD Cards | SanDisk
        • Extreme PLUS microSD UHS-I Card A1 | SanDisk
        • Extreme microSD Card | SanDisk
        • Extreme microSD for Action Cameras | SanDisk
        • Ultra PLUS microSDXC | SanDisk
        • Ultra microSD 95MB/s read speed | SanDisk
        • Ultra microSD UHS-I Card for Cameras | SanDisk
        • microSD Card | SanDisk
        • microSD High Endurance Video Monitoring Card | SanDisk
        • Ultra microSD UHS-I Card 48MB/s read speed | SanDisk
        • Ultra microSD for Smartphone | SanDisk
        • Ultra PLUS microSD UHS-I Card for Cameras | SanDisk
        • Extreme PRO microSDXC UHS-II Card | SanDisk
        • Ultra PLUS microSDXC | SanDisk
        • Ultra microSDXC | SanDisk
        • Extreme microSDXC | SanDisk
      • Compact Flash | SanDisk
        • Extreme PRO CompactFlash Memory Card | SanDisk
        • Extreme CompactFlash Memory Card | SanDisk
        • Ultra CompactFlash Memory Card | SanDisk
      • CFast | SanDisk
        • Extreme PRO CFast 2D Memory Card | SanDisk
      • Memory Card Readers | SanDisk
        • Extreme PRO CFast 2.0 Reader/Writer | SanDisk
        • ImageMate All-in-One USB 3.0 Reader | SanDisk
        • MobileMate Duo Adapter and Reader | SanDisk
        • MobileMate USB Reader | SanDisk
        • Extreme PRO SD UHS-II Card Reader/Writer | SanDisk
        • Extreme PRO SD UHS-II Card USB-C Reader/Writer | SanDisk
        • ImageMate Pro USB 3.1 Reader | SanDisk
    • USB Flash
      • SanDisk Extreme Go USB 3.1 Flash Drive
      • SanDisk Extreme PRO USB 3.1 Solid State Flash Drive
      • Cruzer Dial USB Flash Drive
      • SanDisk Ultra Flair USB 3.0 Flash Drive
      • SanDisk Ultra USB 3.0 Flash Drive
      • SanDisk Ultra Fit USB 3.0 Flash Drive
      • Cruzer Force USB Flash Drive | SanDisk
      • Cruzer Glide USB Flash Drive
      • Cruzer Orbit USB Flash Drive
      • Cruzer Switch USB Flash Drive
      • Cruzer U USB Flash Drive
      • Cruzer Blade USB Flash Drive
      • Cruzer Edge USB Flash Drive
      • Cruzer Fit USB Flash Drive
      • SanDisk Ultra Fit USB 3.1 Flash Drive
    • SSD
      • SanDisk Extreme 900 Portable SSD
      • SanDisk Extreme 510 Portable SSD
      • SanDisk Extreme 500 Portable SSD
      • SanDisk Ultra II mSATA SSD
      • SanDisk Extreme Pro SSD
      • SanDisk Ultra II SSD
      • SanDisk SSD Plus
      • SanDisk SSD Notebook Upgrade Tool Kit
      • SanDisk Ultra 3D SSD
      • Ultra 3D SSD Redirect
      • SanDisk Extreme Portable SSD
      • SanDisk Extreme PRO M.2 NVMe 3D SSD
    • MP3 Players
      • Clip Sport MP3 Player | SanDisk
      • Clip Jam MP3 Player | SanDisk
      • Clip Sport Plus MP3 Player | SanDisk
    • Extreme Team
      • Steve McCurry
      • Akito Mizutani
      • Walter Iooss
      • Elizabeth Kreutz
      • Tom Bol
      • Lucas Gilman
      • Marcelo Maragni
      • Jeff Lewis
      • Corey Rich
      • Christian Pondella
      • Caio Guatelli
      • Terrell Lloyd
      • Michael Grecco
      • Akash Das
      • Kaustav Saikia
      • Amy Tierney
      • Alex Liu
      • Scott A. Woodward
      • Steve Simon
      • Kike Calvo
      • C.S. Ling
      • Peter Eastway
      • Cliff Mautner
      • Bambi Cantrell
      • Bob and Dawn Davis
      • Zach & Jody Gray
      • Marcus Bell
      • Sam Nicholson
      • Richard Finn Gregory
      • Vincent Laforet
      • Sebastien Devaud
      • Paolo Baccolo
      • Phil Coates
      • David Newton
      • Daniel Fox
      • Fred Pompermayer
      • Christian Nørgaard
      • Kirill Umrikhin
      • Matthew Jordan Smith
      • Richard Walch
      • Marcel Laemmerhirt
      • Dom Daher
      • Joao Carlos
      • Claudia Goetzelmann
      • Patrick Bellair
      • Wendell Phillips
      • Tyler Stableford
      • Jonathan & Angela Scott
      • Ellen Anon
      • Daisy Gilardini
      • Georg Tappeiner
      • George Karbus
      • Renan Ozturk
      • Lars Schwellnus
      • Tim Laman
      • Sean Scott
      • Nick Didlick
      • Dave Black
    • Stories
    • Where to Buy
  • For Business
    • Data Center
      • Products
        • Flash Devices (not a page)
          • PCIe Flash (not a page)
          • SSDs (not a page)
        • Flash Software (not a page)
          • Open Source
      • Solutions
        • Database
        • Virtualization
        • Big Data
        • Cloud Solutions
      • Partners
      • Resource Library
        • Case Studies
        • Data Sheets
        • Overviews
        • Report
        • Articles
        • Video
        • Solution Briefs
          • All-Flash Storage System for Ceph and OpenStack
          • Red Hat Ceph Storage on the InfiniFlash All-flash Storage Platform
          • The Solution for Cassandra at Scale | SanDisk
          • Storage for MongoDB Analytics
          • Optimizing Ceph Deployments Solution Brief
      • Blog
    • Computing
      • X600
      • X400
      • Z400s
      • sn720
      • sn520
    • Professional Photo & Video
  • OEM Design
    • Data Center
    • Computing
    • Mobile
      • iNAND Embedded Flash Drives
    • Commercial
    • Industrial
      • iNAND
      • Industrial Cards
      • OEM Cards
    • Automotive
      • Automotive iNAND
      • Automotive SD Cards
      • Automotive SD Cards
    • Connected Home
    • Blog
  • About SanDisk
    • Trusted Brands
      • History of Innovation
      • Ventures
    • Media Center
      • Press Releases
        • 2018
          • Western Digital Unveils New Solutions to Help People Capture, Preserve, Access and Share Their Ever-Growing Collections of Photos and Videos
          • Western Digital Transforms the Mobile Experience with New High-Performance Solutions Designed to Address Today and Tomorrow’s Consumer Demand for High-Quality Content
          • Western Digital Nvme Solutions Enable Data To Thrive In Intelligent Edge And Mobile Computing Environments
          • Western Digital Enables the Evolution of Modern Workflows at NAB 2018
      • Awards & Accolades
      • Events
        • Flash Memory Summit
        • Intel Developer Forum 2015
        • VMworld US
        • LinuxCon North America
        • SpiceWorld
        • VMworld Europe
        • GTEC
        • Oracle OpenWorld
        • HP Discover London
        • Gartner Data Center Conference
        • Photo Plus Expo 2015
        • SQL PASS Summit 2015
        • TigerDirect Innovation – IT Conference and Expo
        • Mobile World Conference
        • Embedded World
        • Open Compute
        • Linux Open Vault Conference
        • LinuxCon North America 2016
        • Embedded World
        • Mobile World Congress
        • SQL PASS Summit
        • Supercomputing16
        • Western Digital: The Future of Data
        • Flash Memory Summit 2017
        • Mobile World Congress
        • Embedded World
        • Computex 2018
      • Media Assets
        • Product
          • Business
          • OEM Design
        • Corporate
          • Management
      • Insight Report
    • Careers
      • Why SanDisk
        • Compensation and Benefits
        • Equal Employment Opportunity
      • University Hires
      • Career Development
    • Global Citizenship
      • Social
      • Environmental
      • Community
      • Scholars
        • 2014 Cal Berkeley Graduate Engineering School Commencement Speech
      • Supplier Management
    • Investor Relations
    • Contact Us
  • Support
    • Ultra +Cloud USB
  • For Home
  • For Business
  • OEM Design
  • About SanDisk
  • Support
 
  • Data Center
    • Computing
    • Professional Photo & Video
  • Products
  • Solutions
  • Partners
  • Resource Library
  • Blog

Case Studies

SanDisk Helps Accelerate High Energy Physics Research for the Origin of the Universe

Using six 2.6TB Fusion ioMemory™ ioDrive®2 Duo drives, the Universities of Michigan and Victoria are sharing massive volumes of data with the CERN Laboratory and 100 other computing centers. The solution may expedite the potential discovery of new particles and forces that will help explain the nature of the universe.

Solution Focus

  • Scientific Research
  • Higher Education
  • Public Sector
  • Supercomputing

Summary

Through the use of Fusion ioMemory™ PCIe solutions, the University of Michigan and University of Victoria are implementing a multi-site supercomputing project to help them share massive volumes of data among the CERN Laboratory in Geneva, Switzerland and 100 computing centers around the world. The solution has the potential to accelerate access to physics data—allowing teams of physicists to expedite the potential discovery of new particles and forces that will help explain the nature of the Universe.

"My colleagues and I are proud and excited that we have now found a way to accelerate the discoveries made for each of these projects with the help of flash memory solutions from SanDisk."

Randall Sobie,
Institute of Particle Physics Research Scientist and Professor at University of Victoria

Background

Francois Englert and Peter W. Higgs received the Nobel Prize in physics for the theoretical discovery of a mechanism that contributes to our understanding of the origin of mass of subatomic particles. This mechanism was confirmed through the discovery of the predicted fundamental particle by the ATLAS and CMS experiments at CERN’s Large Hadron Collider (LHC).

The Large Hadron Collider (LHC) is the world’s largest and most powerful particle accelerator. Initiated on September 10, 2008, the LHC consists of a 27-kilometre ring of superconducting magnets that contain a number of accelerating structures to boost the energy of the particles. Inside the accelerator, two high-energy particle beams travel at close to the speed of light before they are made to collide. They are guided around the accelerator ring by a strong magnetic field maintained by thousands of superconducting electromagnets, which are built from coils of special electric cable that operates in a superconducting state, efficiently conducting electricity without resistance or loss of energy.

All of the controls for the accelerator, including its services and technical infrastructure, are housed under one roof at the CERN Control Centre in Geneva, Switzerland. From here, the beams inside the LHC are made to collide at four locations around the accelerator ring, corresponding to the positions of four particle detectors—ATLAS, CMS, ALICE and LHCb. The LHC is a machine built to study the “Big Bang” when the Universe was formed.

ATLAS is one of the particle physics experiments at the LHC that is dealing with both trying to discover the fundamental laws of physics and with searching for new discoveries in the head-on collisions of protons of extraordinarily high energy. At approximately half the size of Paris’ Notre Dame Cathedral, ATLAS is about 45 meters long, more than 25 meters high, and weighs about 7,000 tons—the equivalent of one hundred 747 jets.

ATLAS is learning about the basic forces that have both shaped our Universe since the beginning of time and that will determine its fate. As the particles crash together in the center of ATLAS, they produce tiny fireballs of primordial energy, thereby recreating the conditions that existed at the birth of the Universe. Among the possible unknowns that ATLAS is exploring are extra dimensions of space, unification of fundamental forces, and evidence for dark matter candidates in the Universe. Due to the discovery of the last missing pieces of the Higgs boson, further data will allow in-depth investigation of the boson’s properties, and thereby, of the origin of mass.

 

The Challenge

The research at the LHC involves a very large and distributed collaboration among hundreds of organizations and thousands of physicists. The result is a huge amount of data; ten petabytes of data is transmitted each year.

The current challenge is to deploy an infrastructure that can support the data being scanned for new discoveries. The key to enabling these scientific discoveries is getting data from the source to the remote local sites as rapidly as possible. The ATLAS data sample is approximately 170 petabytes and must be transferred from CERN to other centers around the world at 100 gigabits per second (Gbps) to enable scientists to rapidly access and analyze data, and speed up discoveries.

In Spring 2015, the LHC will start colliding elements at the highest energies ever achieved in a particle accelerator. With this in mind, the physicists at the Universities of Michigan and Victoria needed to create a data transfer architecture based upon a single server that is able to transfer data among 100 computing centers around the world at 100 Gbps speeds.

Research sites typically make local copies of data when an interesting data set or anomaly has been identified. As sites begin to upgrade to 100 Gbps network connections, the challenge is to utilize the bandwidth efficiently and deploy a subsystem that can deliver at that level. To date, there has not been a vendor with a 100GB interface card.

Physicists from University of Michigan and University of Victoria are working together to develop a simple grid framework infrastructure that can utilize wide area network 100GB connections to enable data to be scanned by local sites for new physics. Large local sites must have the ability to move data across many servers and therefore need the network cache to source and sync data from many sites to distribute locally, as smaller sites deploy a single server to utilize the data.

“The ATLAS and CMS supercomputing projects are very large international projects, each involving approximately 3,000 researchers and most of the world’s countries. These are long term projects—they started 20 years ago and will continue for another 20+ years,” said Randall Sobie, Institute of Particle Physics Research Scientist and Professor at University of Victoria.

 

The Solution

The Caltech, University of Michigan, and University of Victoria teams at Supercomputing 2014 (SC14) had been aware of SanDisk’s Fusion ioDrive® solutions for some time. NAND-based and PCIe-based storage solutions were intriguing due to the promise of obtaining better performance than RAID storage systems. The team expected that using the SanDisk drives was the only way to get sufficient performance from a single box.

Working with Fusion ioMemory solutions, the University of Michigan and University of Victoria research teams deployed a single server storage environment to fuel their multi-site supercomputing project. It was necessary that the solution deliver massive data transmissions, while eliminating the need for a multi-server configuration and reducing complexity, cost, and points of failure. The chosen flash memory solution also dramatically reduced the server footprint needed to transmit these enormous datasets. By utilizing Fusion ioMemory PCIe solutions, the universities have accelerated access to data, allowing physicists to expedite the potential discovery of new particles and forces that will help explain the nature of the Universe.

The design tested at SC14 was focused around building a single server with the capability of meeting data transfer needs of a site connected at 100GE. One high performance server resided in the University of Michigan booth at SC2014 and a second high performance server resided at the University of Victoria. High performance networks were provided by CANARIE, BCNET, CenturyLink, and SCinet in collaboration with the Caltech Networking Team.

At the University of Victoria, a dedicated 100G circuit was established using a Brocade MLXe-4 and CFP2 module to connect to the SC14 show floor. High-performance data transfer nodes were architected using a Dell R920 server equipped with six 2.6TB Fusion ioMemory ioDrive2 Duo solid state storage (SSD) drives that were connected to the MLXe-4 with a 100G CFP2 module and three 40GE NICs. An OpenDaylight controller employing custom multipath extensions was implemented on the SC14 show floor to control the MLXe-4 at the University of Victoria.

Regarding the support from SanDisk, Ian Gable, physics computing specialist with University of Victoria, said, “We were extremely impressed with the deep technical expertise of the SanDisk solutions architect. The late nights and hard work provided by SanDisk were contributing factors to our success.”

The Result

The researchers were very happy with the initial results and declared the performance of the SanDisk Fusion ioDrive2 Duo cards as nothing short of astonishing. The solution achieved 137 Gbps read-to-disk locally and 113 Gbps write-to-disk locally.

Each Dell server drove the network memory at a flat 99.7 Gbps from the University of Victoria via the Brocade MLXe-4 to the Caltech booth and then to the University of Michigan booth on the SC14 show floor—illustrating a loss-free network path between the two sites. Reading from disk using SanDisk Fusion ioDrive2 Duo drives resulted in 65 millisecond latency between the two sites at 73 Gbps. Writing occurred at 60 Gbps. These results indicate that a single server can handle the needs of a 100G wide area circuit.

“We believe that there is potential to improve on these results with further testing and tuning,” commented Shawn McKee, physics researcher at University of Michigan. “However we are quite satisfied with the progress made during the limited time frame of SC14. Using a single 40GE network card we were able to copy data from disk to disk at 37.5 Gbps with just two SanDisk Fusion ioDrive2 Duo cards. This indicates that we should be able to reach our final goal of 100G read and write using this extremely capable hardware.”

To take advantage of these results in production, it will be necessary to fully explore the interaction among all of the components of the long latency, high-bandwidth path and with the multiple levels of caching and buffering along the end-to-end system—Application, Drive, Filesystem, and Network.

“My colleagues and I are proud and excited that we have now found a way to accelerate the discoveries made for each of these projects with the help of flash memory solutions from SanDisk,” said Sobie.

Each University now has a production 100G WAN network connection and will continue to boost the efficiency of the WAN transfer. They will then deploy this formula at other scientific sites.

“SanDisk is thrilled to be working with the teams at the University of Michigan and University of Victoria to help fuel their success by providing fast, cost-effective, and highly scalable flash solutions to increase data access,” said Sumit Sadana, executive vice president and chief strategy officer, SanDisk. “By utilizing flash technology, the researchers can cost-effectively transfer massive amounts of data over long distances, ultimately enabling them to reach new discoveries faster.”

The Future

The ATLAS accelerators have been in shutdown since 2013 to undergo upgrades and infrastructure changes. Previously configured for 8 to 13 teraelectronvolts (TeV), ATLAS will be newly configured to run at 14 TeV to accommodate physics data that will be generated beginning in May 2015. Teams are in the process of upgrading equipment, software, and frameworks to implement a proof of concept infrastructure that demonstrates what options are capable of enabling their data needs. Individual sites will then determine what infrastructure will be adopted depending on yearly budgets and available equipment.

SanDisk Helps Accelerate High Energy Physics Research for the Origin of the Universe
Download

READY TO FLASH FORWARD?

Whether you’re a Fortune 500 or five person startup, SanDisk has solutions that will help you get the most out of your infrastructure.

ONLINE
REQUEST

Go ahead, ask us some questions and we’ll get back to you with answers.

GLOBAL SALES
CONTACTS

Find contact information for offices all around the world.

CUSTOMER
CARE

Have a question about a product you already own? We can help!

SALES INQUIRIES

Whether you'd like to ask a few initial questions or are ready to discuss a SanDisk solution tailored to your organizations's needs, the SanDisk sales team is standing by to help.

We're happy to answer your questions, so please fill out the form below so we can get started.

Field cannot be empty.
Field cannot be empty.
Enter a valid email address.
Field can only contain numbers.
Field cannot be empty.
Field cannot be empty.
Field cannot be empty.
Field cannot be empty.
Field cannot be empty.
Field cannot be empty.

Please indicate your areas of interest:

You must choose an option.

Questions or comments:

Privacy Policy

Thank you. We have received your request.

Site Menu

  • For Home
    • Mobile Storage
    • Cards & Readers
    • USB
    • SSD
    • MP3 Players
  • For Business
    • Data Center
    • Computing
    • Partners
    • Pro Photo & Video
  • OEM Design
    • Mobile
    • Computing
    • Automotive
    • Connected Home
    • Industrial & Other
    • Data Center
  • Media Center
    • Press Releases
    • Awards and Accolades
    • Media Resources
  • Careers
  • Investor Relations
  • Contact Us

Support

  • Business Support
  • Retail Support

Quick Links

  • Data Blog
  • SanDisk Stories
  • Extreme Team

Data Blog

Post Thumbnail

{{feed.title}} | {{getFormattedDate(feed.publishedDate) | date:''}}

Read More »
  • Legal
  • Terms of Use
  • Trademarks
  • Privacy
  • Cookies
  • Taiwan BSMI RoHS
  • About Ads
  •  
SanDisk
© 2018 Western Digital Corporation or its affiliates. All rights reserved. Western Digital Technologies, Inc. is the seller of record and licensee in the Americas of SanDisk® products.