-
Notifications
You must be signed in to change notification settings - Fork 9
Home
You must have the following:
- a working Ceph cluster
- a host to act as a gateway (iSCSI target)
- a host to act as a client (iSCSI initiator)
- lrbd installed on the gateway
For Adding a Gateway and Adding a Target
- another host to act as a second or redundant gateway
The lrbd configuration is a global configuration for all gateways and clients. The gateway sections include the Ceph pools, RBD images, portals (i.e. network interfaces), static targets and authentication. Within these sections are specific access definitions and restrictions for initiators.
For the first configuration with static values, take a look at the simple.json file located in /usr/share/doc/packages/lrbd/samples. Copy this file to another location such as your home directory.
The file is a JSON data structure containing four sections. Update each section as required.
If you wish to use the default pool and RBD image, verify the following:
- the rbd pool exists
# rados lspools - the archive image exists
# rbd -p rbd ls
Otherwise, create your own pool and RBD image. Update the pool and image in the pools section.
Update the host to match the hostname of the gateway. Update the initiator to match the client. For Linux, find the value in /etc/iscsi/initiatorname.iscsi.
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
}
]
Under the targets section, update the host to match the hostname of the gateway. Replace the target with the output of iscsi-name.
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.igw1.x86:sn.db7fc86c644"
}
]
Replace the address with a valid network address of the gateway host.
"portals": [
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
}
]
The authentication is set to tpg. All initiators will use the same credentials to connect to this target. Replace the host with the hostname of the gateway. Set the desired userid and password for CHAP authentication.
"auth": [
{
"host": "igw1",
"authentication": "tpg",
"tpg": {
"userid": "common1",
"password": "pass1"
}
}
]
Add comments if you wish using a '#'. Any comments will be stripped while importing the configuration.
Import the configuration and then query the configuration from a gateway host.
# lrbd -f ~/simple.json
# lrbd -o
Another option to display the configuration is
# lrbd -l
In this case, both outputs will be the same. The -o displays all hosts while -l shows only entries relevant to this host. This is most useful when determining whether a configuration is correct for a gateway.
Inspect the current configuration.
# targetcli ls
Run the lrbd command to apply the saved configuration.
# lrbd
The lrbd command outputs each rbd and targetcli command with its output.
Inspect the current configuration again.
# targetcli ls
The output will look similar to the following:
o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [0 Storage Object] | o- iblock ............................................... [1 Storage Object] | | o- archive .............................. [/dev/rbd/rbd/archive activated] | o- pscsi ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ....................................................... [0 Targets] o- iscsi .......................................................... [1 Target] | o- iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk .......... [1 TPG] | o- tpg1 ........................................................ [enabled] | o- acls ........................................................ [1 ACL] | | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN] | | o- mapped_lun0 ......................................... [lun0 (rw)] | o- luns ........................................................ [1 LUN] | | o- lun0 ...................... [iblock/archive (/dev/rbd/rbd/archive)] | o- portals .................................................. [1 Portal] | o- 172.16.1.16:3260 .............................. [OK, iser disabled] o- loopback ...................................................... [0 Targets] o- qla2xxx ....................................................... [0 Targets] o- tcm_fc ........................................................ [0 Targets] o- vhost ......................................................... [0 Targets]
The import, editing or querying can be done on any Ceph client with lrbd installed. Applying the configuration will only make changes to the current host.
Host specific configurations can be queried from another host. For example, running the following from host igw1
# lrbd -H igw2 -l
will display the configuration of host igw2. Applying the configuration of another host is similar. From a host test1, running
# lrbd -H igw2
applies the configuration from host igw2. This allows investigation from separate hardware or temporary replacement of failed hardware without modifying the configuration.
Growing the configuration to match your requirements is straightforward. Add the necessary entries often by copying an existing entry and making simple changes. Be aware that a valid JSON structure needs a comma between elements but not after the last element in a list.
The following examples reference the simple.json in the previous section.
Create an image within the same pool.
# rbd -p rbd create data --size=2048
Copy the tpg entry containing image, portal and initiator. Change the image in the new entry, save your changes and apply. An initiator can access two RBD images from the same pool on the same gateway over the same network path.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": "portal1",
"image": "data",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
Nearly identical to the adding an image, but change the initiator in the new entry. Two different initiators can access the same RBD image on the same gateway over the same network path.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1991-05.com.microsoft:client.example.com"
}
In the portals section, duplicate the portal entry and change the name and address.
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
},
{
"name": "portal2",
"addresses": [ "172.16.2.16" ]
}
Use the new portal name. In this example, each initiator will use a dedicated network interface.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": "portal2",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f99"
}
In the portals section, provide a second address to an existing portal. Any entry using this portal will provide redundancy across two network paths.
{
"name": "portal1",
"addresses": [ "172.16.1.16", "172.16.2.16" ]
}
Copy an existing gateway entry. Maintain the same host and change any or all of portal, image and initiator. This allows isolated paths from initiator to image.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"host": "igw1",
"tpg": [
{
"portal": "portal2",
"image": "data",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f99"
}
]
}
Similar to adding a tpg, copy an existing gateway entry. Change the host of the new entry to the hostname of your second gateway. Do not use this for redundancy. This configuration is for two gateways with independent images.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"host": "igw2",
"tpg": [
{
"portal": "portal3",
"image": "data",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
Add another portal entry for the address of the second gateway.
{
"name": "portal1",
"addresses": [ "172.16.1.16", "172.16.2.16" ]
},
{
"name": "portal3",
"addresses": [ "172.16.1.17" ]
}
Duplicate a pool entry. Update the pool and image. Each image is accessed via the same network path.
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
},
{
"pool": "car",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "cement",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
}
More than one gateway can advertise the same target providing redundancy. This configuration requires a kernel that supports the tpg_enabled_sendtargets (formerly discoverable_while_disabled) and a targetcli that allows portal creation with non-existent addresses. (Versions not yet determined as Aug 7, 2015.)
Copy an existing gateway entry. Change the host keyword to target. Update the hostname to an iqn value. Consider using iscsi-name as a template. Change the embedded hostname (e.g. igw1 to any). Change the portal.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"target": "iqn.2003-01.org.linux-iscsi.any.x86:sn.db7fc86c644",
"tpg": [
{
"portal": "portal4",
"image": "data",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
Update the targets section. Add the new target with a list of hosts.
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk"
},
{
"hosts": [ "igw1", "igw2" ],
"target": "iqn.2003-01.org.linux-iscsi.any.x86:sn.db7fc86c644"
}
Update the portals section. Add the new portal with the address list from both gateways.
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
}
{
"name": "portal4",
"addresses": [ "172.16.1.16", "172.16.1.17" ]
}
When applied, each gateway will have two TPGs. The first is enabled and contains a portal of the address(es) on the local gateway. The second TPG is disabled and contains a portal of the remaining addresses from remote gateways.
For the simple.json configuration, authentication can be configured twelve different ways excluding no authentication. See the /usr/share/doc/packages/lrbd/samples directory for examples. Pull the authentication section from the file for your strategy and replace the existing one in simple.json. Update accordingly.
For the undecided, use the authentication section from complete.json which contains all authentication possibilities. Experiment by disabling features (e.g. discovery or mutual) or selecting other schemes (e.g tpg vs acls).
The configuration may be updated directly with lrbd -e. The entire configuration can be output with lrbd -o > filename and maintained under your choice of version control.
Alternatively, the global configuration can be re-imported after making changes. Running lrbd -f filename will replace any configuration stored within Ceph. The advantage of this method is the input file may contain comments for complex configurations.
Lastly, configurations may be added cumulatively. For instance, if two groups within an organization each have independent gateways and clients but wish to share the same Ceph cluster, multiple configurations can be maintained independently. Each configuration must be valid by itself, but could be divided into separate files by pool or gateway. One file could contain gateways A, B and C while another contains D and E.
The first configuration is imported as normal with lrbd -f ABC.config. The second configuration is added with lrbd -a DE.config. The pools and authentication sections will be merged. The targets and portals sections are replaced. Depending on the configuration, either remove the targets and portals section from the DE.config or duplicate these sections in both files.
The above example could also be executed with lrbd -W which wipes the configuration from Ceph and then executes lrbd -a for each file.
- Adding an initiator
- Adding a portal
- Adding an address
- Adding a tpg
- Adding a gateway
- Adding a pool
- Adding a target
- Authentication
- No authentication
- TPG authentication
- ACL authentication
- TPG+identified authentication
- Identified authentication
- Discovery and Mutual authentication
- Optional Settings
- Backward Compatibility
- Systemd service
- Managing configurations
- Multipath
- Examples
- Redundant gateways
- Access control
- Redundant gateways with isolated paths