Storage

NFS

For projects that require access to our shared storage, please follow this 5 steps so your VM(s) can have access via NFS. Before proceeding, please make sure the instance you are trying to connect to the shared storage has been deployed with an interface in the data network (net_provider_XXXX) and this network is the second on your instance’s networks list. The net_provider ID is provided by the Cloud Team after your request. The following examples apply to CentOS only.

1.- Install package dnsmasq if it is not present in the system already. Activate the second interface to automatically get an IP on the net_provider_XXXX network.

1
2
$ yum install dnsmasq
$ dhclient eth1

2.- Create/modify your /etc/dnsmasq.conf file so it looks like this (where XXXX is the number associated to your net_provider_XXXX network):

/etc/dnsmasq.conf
1
2
3
4
5
# Route all shared storage queries to the service IP
server=/em-isi-XXXX.ebi.ac.uk/<Service IP provided by the Cloud Team>

# Route the rest of the queries to the Internet (or your preferred DNS server)
server=8.8.8.8

Note: Please request the connection details to the Cloud Team at EBI (embassycloud@ebi.ac.uk).

3.- Change your /etc/resolv.conf so you use the dnsmasq service from now on:

1
2
/etc/resolv.conf
nameserver 127.0.0.1

Note: Bear in mind that we normally configure the private subnets to assign the Google dns servers in the DHCP configuration, so you may have to overwrite this dns configuration in the network settings within your Openstack tenancy.

4.- Start and enable the dnsmasq service:

1
2
$ systemctl enable dnsmasq.service
$ systemctl restart dnsmasq.service

5.- Check the connectivity and test mount the NFS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
$ ping em-isi-XXXX.ebi.ac.uk
64 bytes from 10.10.10.10 (10.10.10.10): icmp_seq=1 ttl=64 time=0.437 ms
...
(Ctrl-C)

$ ping em-isi-XXXX.ebi.ac.uk
64 bytes from 10.10.10.11 (10.10.10.11): icmp_seq=1 ttl=64 time=0.437 ms
...
(Ctrl-C)

# Note: Make sure the ip returned by the 'ping' command changes in every
# execution and also test external name resolution to ensure that dnsmasq is
# working properly:

$ ping www.google.com
64 bytes from lhr35s01-in-f4.1e100.net (172.217.23.4): icmp_seq=1 ttl=64 time=4.607 ms
...
(Ctrl-C)

$ mount -t nfs em-isi-XXXX.ebi.ac.uk:/ifs/yourShareName /mnt

S3 Object Store

Embassy users can request access to our Object Stores, we have an S3 compatible Object Storage backend. Some of the typical Object store use cases are:

  • No requirement for a POSIX filesystem
  • Large datasets
  • Unstructured data
  • Backups
  • Archiving

If you would like to use our Object Stores or simply explore the technology, please send us an email to embassycloud@ebi.ac.uk and we will create an environment for you.

Note

  • The bucket creation feature is not available to users, if you require additional buckets please let us know.
  • Our s3 compatible Object Store is not backed up. Please make sure you don’t use it as your only backup target.

S3

In order to use our s3 compatible object store, you can download the AWS Command Line Interface (awscli) from https://aws.amazon.com/cli/. Alternatively you can also use https://github.com/s3tools/s3cmd, which is another interface written in python.

Examples of use:

$ export AWS_ACCESS_KEY_ID=yourAccessKeyId
$ export AWS_SECRET_ACCESS_KEY=yourSecretAccessKey

(or)

$ aws configure
#(and follow the steps)

# Create a bucket
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 mb s3://test

# List buckets
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 ls
2021-01-29 15:35:21 test

# Upload file to bucket
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 cp helloworld.txt s3://testbucket/
upload: helloworld.txt

# List files within bucket
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 ls s3://testbucket/
2021-01-29 15:36:51 13 helloworld.txt

# Upload directory including only jpgs and txts
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 cp /tmp/foo/ s3://testbucket/ --recursive --exclude "*" --include "*.jpg" --include "*.txt"

# Generate a temporary url for users to download a given object - default expiration time 3600s
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 presign s3://testbucket/myobject
https://s3.embassy.ebi.ac.uk/testbucket/myobject?AWSAccessKeyId=ozB4pHyzrPUjXo1fw57&Signature=pG3xRpKyTuxQq8xatRUusJ6oE%3D&Expires=1526574462

# Obtain the used space and number of objects (result displayed in bytes)
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3api list-objects --bucket testbucket --output json --query "[sum(Contents[].Size), length(Contents[])]"

# Delete a bucket (use --force if it's not empty)
$ aws --endpoint-url https://s3.embassy.ebi.ac.uk s3 rb --force s3://test

S3 Java SDK

If your Java application needs to interact with our s3 compatible Object Store you have two options:

  • Amazon Java SDK: if your application requires portability to Amazon.
  • IBM Cloud Object Storage Java SDK: If you would like to use all the features from our Object Store, use the SDK directly provided by the vendor. You can find examples and code repositories in the link name.

Example code for Amazon Java SDK:

import com.amazonaws.regions.Region;

import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;

import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectSummary;

import java.util.List;

public class EBIaws {

  public static void main( String[] args ) {

  final AmazonS3 s3 = AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(new EndpointConfiguration("https://s3.embassy.ebi.ac.uk/mybucket", "eu-west-2"))
    .withCredentials(new EnvironmentVariableCredentialsProvider())
    .withPathStyleAccessEnabled(true)
    .build();

  System.out.println("Listing objects");
          ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
                .withBucketName(""));
          for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
            System.out.println(" - " + objectSummary.getKey() + "  " +
                               "(size = " + objectSummary.getSize() + ")");
          }
          System.out.println();
      }
}

Virtual Gluster

What is Gluster?

Gluster is a distributed scale out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature. All of this is accomplished without a centralized metadata server. For more information please visit the official gluster documentation.

Automating a Gluster Cluster Deployment

Openstack uses Heat API for orchestration which automates provisioning. We have created a Heat template which can deploy a gluster cluster for you with one simple command.

If you are interested in using this template then please contact embassycloud@ebi.ac.uk .

Example deployment

This is a distributed two replica Gluster deployment (similar to RAID 10).

Shared Model

The Gluster deployment is composed of ten s1.jumbo nodes (including the master), and we use an openstack flavour to define the total Gluster disk space. The disk will all be local to the compute nodes (Ephemeral).

By simply editing gluster-environment.yaml we can change these parameters -

$ cat gluster-environment.yaml
parameter_defaults:
key_name: cems
instance_count: 9
instance_type: s1.jumbo
local_network: f3935f65-00e7-4787-89ca-eaeb302cfbd6
availability_zone: AZ_2
image_id: '1dbc3a5b-0930-4652-a4f2-c5d21166ce5a'

Execute the following commands for deploying your virtual Gluster filesystem:

$ openstack stack create --template gluster_stack.yaml -e gluster-environment.yaml test_gluster
Shared Model
$ openstack stack list
Shared Model
$ openstack server list
Shared Model