1 =================================================================
5 =================================================================
15 - 4. Open-iSCSI daemon
16 - 5. Open-iSCSI Configuration Utility
19 - 8. Advanced Configuration
20 - 9. iSCSI System Info
26 This file describes the Linux* Open-iSCSI Initiator. The software was
27 tested on AMD Opteron (TM) and Intel Xeon (TM).
29 The latest development release is available at:
30 http://www.open-iscsi.org
32 For questions, comments, contributions send e-mail to:
33 open-iscsi@googlegroups.com
37 - highly optimized and very small-footprint data path;
38 - persistent configuration database;
39 - SendTargets discovery;
47 Open-iSCSI project is a high-performance, transport independent,
48 multi-platform implementation of RFC3720 iSCSI.
50 Open-iSCSI is partitioned into user and kernel parts.
52 The kernel portion of Open-iSCSI is a from-scratch code
53 licensed under GPL. The kernel part implements iSCSI data path
54 (that is, iSCSI Read and iSCSI Write), and consists of three
55 loadable modules: scsi_transport_iscsi.ko, libiscsi.ko and iscsi_tcp.ko.
57 User space contains the entire control plane: configuration
58 manager, iSCSI Discovery, Login and Logout processing,
59 connection-level error processing, Nop-In and Nop-Out handling,
60 and (in the future:) Text processing, iSNS, SLP, Radius, etc.
62 The user space Open-iSCSI consists of a daemon process called
63 iscsid, and a management utility iscsiadm.
69 As of today, the Open-iSCSI Initiator requires a host running the
70 Linux operating system with kernel version 2.6.16, or later. 2.6.14 and
71 2.6.15 are partially supported. Known issues with 2.6.14 - .15 support:
73 - If the device is using a write back cache, during session logout
74 the cache sync command will fail.
75 - iscsiadm's -P 3 option will not print out scsi devices.
76 - iscsid will not automatically online devices.
78 You need to enable "Cryptographic API" under "Cryptographic options" in the
79 kernel config. And you must enable "CRC32c CRC algorithm" even if
80 you do not use header or data digests. They are the kernel options,
81 CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively.
83 By default the kernel source found at
84 /lib/modules/`uname -a`/build
85 will be used to compile the open-iscsi modules. To specify a different
86 kernel to build against use:
88 make KSRC=<kernel-src>
92 make KSRC=<kernel-src> KARCH="ARCH=um"
94 To compile on SUSE Linux you'll have to use
96 make KSRC=/usr/src/linux \
97 KBUILD_OUTPUT=/usr/src/linux-obj/<arch>/<config>
99 where <config> is the kernel configuration to use (eg. 'smp').
101 For Red Hat/Fedora and Debian distributions open-iscsi can be installed by
102 typing "make install". This will copy iscsid and iscsiadm to /usr/sbin, the
103 init script to /etc/init.d, and the kernel modules: iscsi_tcp.ko,
104 libiscsi_tcp.ko, libiscsi.ko and scsi_transport_iscsi to
105 /lib/modules/`uname -r`/kernel/drivers/scsi/ overwriting existing iscsi modules.
107 For Debian, be sure to install the linux-headers package that
108 corresponds to your kernel in order to compile the kernel modules
109 ('aptitude install linux-headers-`uname -r`'). You may also wish to
110 run 'make -C kernel/ dpkg_divert' before installing kernel modules if
111 you run a Debian-provided kernel. This will use dpkg-divert(8) to
112 move the packaged kernel modules out of the way, and ensure that
113 future kernel upgrades will not overwrite them.
115 Also, please be aware that the compatibility patches that enable these
116 iscsi modules to run on kernels older than 2.6.25 will not update the
117 ib_iser module; you may get warnings related to mismatched symbols on
118 this driver, in which case you'll be unable to load ib_iser and
119 open-iscsi simultaneously.
124 The daemon implements control path of iSCSI protocol, plus some management
125 facilities. For example, the daemon could be configured to automatically
126 re-start discovery at startup, based on the contents of persistent
127 iSCSI database (see next section).
133 Usage: iscsid [OPTION]
135 -c, --config=[path] Execute in the config file (/etc/iscsi/iscsid.conf).
136 -f, --foreground run iscsid in the foreground
137 -d, --debug debuglevel print debugging information
138 -u, --uid=uid run as uid, default is current user
139 -g, --gid=gid run as gid, default is current user group
140 -h, --help display this help and exit
141 -v, --version display version and exit
145 5. Open-iSCSI Configuration Utility
146 ===================================
148 Open-iSCSI persistent configuration is implemented as a DBM database
149 available on all Linux installations.
151 The database contains two tables:
153 - Discovery table (/etc/iscsi/send_targets);
154 - Node table (/etc/iscsi/nodes).
156 The regular place for iSCSI database files: /etc/iscsi/nodes
158 The iscsiadm utility is a command-line tool to manage (update, delete,
159 insert, query) the persistent database.
161 The utility presents set of operations that a user can perform
162 on iSCSI nodes, sessions, connections, and discovery records.
164 Open-iscsi does not use the term node as defined by the iSCSI RFC,
165 where a node is a single iSCSI initiator or target. Open-iscsi uses the
166 term node to refer to a portal on a target, so tools like iscsiadm
167 require that --targetname and --portal argument be used when in node mode.
169 For session mode, a session id (sid) is used. The sid of a session can be
170 found by running iscsiadm -m session -i. The session id is not currently
171 persistent and is partially determined by when the session is setup.
173 Note that some of the iSCSI Node and iSCSI Discovery operations
174 do not require iSCSI daemon (iscsid) loaded.
180 Usage: iscsiadm [OPTION]
182 -m, --mode <op> specify operational mode op = <discovery|node>
183 -m discovery --type=[type] --interface=iscsi_ifacename \
184 --portal=[ip:port] --login --print=[N] \
185 --op=[op]=[NEW | UPDATE | DELETE]
186 perform [type] discovery for target portal with
187 ip-address [ip] and port [port].
189 By default this command will remove records
190 for portals no longer returned. And, if a portal is
191 returned by the target, then the discovery command
192 will create a new record or modify an existing one
193 with values from iscsi.conf and the command line.
195 [op] can be passed in multiple times to this
196 command, and it will alter the DB manipulation.
198 If [op] is passed in and the value is
199 "new", iscsiadm will add records for portals that do
200 not yet have records in the db.
202 If [op] is passed in and the value is
203 "update", iscsiadm will update records using info
204 from iscsi.conf and the command line for portals
205 that are returned during discovery and have
208 If [op] is passed in and the value is "delete",
209 iscsiadm will delete records for portals that
210 were not returned during discovery.
212 See the example section for more info.
214 See below for how to setup iscsi ifaces for
215 software iscsi or override the system defaults.
217 Multiple ifaces can be passed in during discovery.
218 -m discovery --print=[N] display all discovery records from internal
219 persistent discovery database.
220 -m discovery --interface --portal=[ip:port] --print=[N] --login
221 perform discovery based on portal in database. See
222 above for info in the interface argument.
224 For the above commands "print" is optional. If
225 used, N can be 0 or 1.
226 0 = The old flat style of output is used.
227 1 = The tree style with the inteface info is used.
229 If print is not used the old flay style is used.
230 -m discovery --portal=[ip:port] --op=[op] [--name=[name] --value=[value]]
231 perform specific DB operation [op] for specific
232 discovery portal. It could be one of:
233 [new], [delete], [update] or [show]. In case of
234 [update], you have to provide [name] and [value]
236 -m node display all discovered nodes from internal
237 persistent discovery database
238 -m node --targetname=[name] --portal=[ip:port] \
239 --interface=iscsi_ifacename] \
240 [--login|--logout|--rescan|--stats]
241 -m node --targetname=[name] --portal=[ip:port]
242 --interface=[driver,HWaddress] \
243 --op=[op] [--name=[name] --value=[value]]
244 -m node --targetname=[name] --portal=[ip:port]
245 --interface=iscsi_ifacename] \
247 perform specific DB operation [op] for specific
248 interface on host that will connect to portal on
249 target. targetname, portal and interface are optional.
250 See below for how to setup iscsi ifaces for
251 software iscsi or override the system defaults.
254 [new], [delete], [update] or [show]. In case of
255 [update], you have to provide [name] and [value]
257 [delete] - Note that if a session is using the
258 node record, the session will be logged out then
259 the record will be deleted.
261 Print level can be 0 to 1.
263 Rescan will perform a SCSI layer scan of the session
266 Stats prints the iSCSI stats for the session.
267 -m node --logoutall=[all|manual|automatic]
268 Logout "all" the running sessions or just the ones
269 with a node startup value manual or automatic.
270 Nodes marked as ONBOOT are skipped.
271 -m node --loginall=[all|manual|automatic]
272 Login "all" the running sessions or just the ones
273 with a node startup value manual or automatic.
274 Nodes marked as ONBOOT are skipped.
275 -m session display all active sessions and connections
276 -m session --sid=[sid] [ --print=level | --rescan | --logout ]
277 --op=[op] [--name=[name] --value=[value]]
278 perform operation for specific session with
279 session id sid. If no sid is given the operation
280 will be performed on all running sessions if possible.
281 --logout and --op work like they do in node mode,
282 but in session mode targetname and portal info is
285 Print level can be 0 to 2.
286 1 = Print basic session info like node we are
287 connected to and whether we are connected.
288 2 = Print iscsi params used.
289 3 = Print SCSI info like LUNs, device state.
290 If no sid and no operation is given print out the
292 -m iface --interface=iscsi_ifacename --op=[op] [--name=[name] --value=[value]]
294 perform operation on fiven interface with name
297 See below for examples.
298 -m host --host=hostno --print=level
299 Display information for a specific host if hostno
300 is passed in. If no hostno is passed in then info
301 for all hosts is printed.
303 Print level can be 0 to 4.
304 1 = Print info for how like its state, MAC, and
306 2 = Print basic session info for nodes the host
308 3 = Print iscsi params used.
309 4 = Print SCSI info like LUNs, device state.
310 -d, --debug debuglevel print debugging information
311 -V, --version display version and exit
312 -h, --help display this help and exit
316 5.1 iSCSI iface setup
317 =====================
319 The next sections describe how to setup iSCSI ifaces so you can bind
320 a session to a NIC port when using software iscsi (section 5.1.1), and
321 it describes how to setup ifaces for use with offload cards from Chelsio
322 and Broadcm (section 5.1.2).
325 5.1.1 How to setup iSCSI interfaces (iface) for binding
326 =======================================================
328 If you wish to allow the network susbsystem to figure out
329 the best path/NIC to use then you can skip this section. For example
330 if you have setup your portals and NICs on different subnets then
331 this the following is not needed for software iscsi.
334 This feature is experimental. The interface may change. When reporting
335 bugs, if you cannot do a "ping -I ethX target_portal", then check your
336 network settings first. If you cannot ping the portal, then you will
337 not be able to bind a session to a NIC.
339 What is a scsi_host and iface for software, hardware and partial
342 Software iscsi, like iscsi_tcp and iser, allocate a scsi_host per session
343 and does a single connection per session. As a result
344 /sys/class_scsi_host and /proc/scsi will report a scsi_host for
345 each connection/session you have logged into. Offload iscsi, like
346 Chelsio cxgb3i, allocates a scsi_host for each PCI device (each
347 port on a HBA will show up as a different PCI device so you get
348 a scsi_host per HBA port).
350 To manage both types of initiator stacks, iscsiadm uses the interface (iface)
351 structure. For each HBA port or for software iscsi for each network
352 device (ethX) or NIC, that you wish to bind sessions to you must create
353 a iface config /etc/iscsi/ifaces.
358 iface0 qla4xxx,00:c0:dd:08:63:e8,20.15.0.7,default,iqn.2005-06.com.redhat:madmax
359 iface1 qla4xxx,00:c0:dd:08:63:ea,20.15.0.9,default,iqn.2005-06.com.redhat:madmax
361 Will report iface configurations that are setup in /etc/iscsi/ifaces.
364 iface_name transport_name,hwaddress,ipaddress,net_ifacename,initiatorname
366 For software iscsi, you can create the iface configs by hand, but it is
367 reccomended that you use iscsiadm's iface mode. There is a iface.example in
368 /etc/iscsi/ifaces which can be used as a template for the daring.
370 For each network object you wish to bind a session to you must create
371 a seperate iface config in /etc/iscsi/ifaces and each iface config file
372 must have a unique name which is less than or equal to 64 characters.
376 If you have NIC1 with MAC address 00:0F:1F:92:6B:BF and NIC2 with
377 MAC address 00:C0:DD:08:63:E7 and you wanted to do software iscsi over
378 TCP/IP. Then in /etc/iscsi/ifaces/iface0 you would enter:
380 iface.transport_name = tcp
381 iface.hwaddress = 00:0F:1F:92:6B:BF
383 and in /etc/iscsi/ifaces/iface1 you would enter:
385 iface.transport_name = tcp
386 iface.hwaddress = 00:C0:DD:08:63:E7
388 Warning: Do not name a iface config file "default" or "iser".
389 They are special value/file that is used by the iscsi tools for
390 backward compatibility. If you name a iface default or iser, then
391 the behavior is not defined.
393 To use iscsiadm to create iface0 above for you run:
395 (This will create a new empty iface config. If there was already a iface
396 with the name "iface0" this command will overwrite it.)
397 # iscsiadm -m iface -I iface0 --op=new
399 (This will set the hwaddress.)
400 # iscsiadm -m iface -I iface0 --op=update -n iface.hwaddress -v 00:0F:1F:92:6B:BF
402 If you had sessions logged in iscsiadm will not update, overwrite
403 a iface. You must log out first. If you have a iface bound to a node/portal
404 but you have not logged in then, iscsiadm will update the config and
405 all existing bindings.
408 You should now skip to 5.1.3 to see how to log in using the iface and for
409 some helpful management commands.
413 5.1.2 Setting up a iface for a iSCSI offload card
414 =================================================
416 This section describes how to setup ifaces for use with Chelsio
419 By default, iscsiadm will create a iface for each Broadcom and Chelsio
420 port. The iface name will be of the form:
422 $transport/driver_name.$MAC_ADDRESS
427 default tcp,<empty>,<empty>,<empty>,<empty>
428 iser iser,<empty>,<empty>,<empty>,<empty>
429 cxgb3i.00:07:43:05:97:07 cxgb3i,00:07:43:05:97:07,<empty>,<empty>,<empty>
432 Will report iface configurations that are setup in /etc/iscsi/ifaces.
435 iface_name transport_name,hwaddress,ipaddress,net_ifacename,initiatorname
437 iface_name: name of iface
438 transport_name: name of driver
439 hwaddress: MAC address
440 ipaddress: IP address to use for this port
441 net_iface_name: Net_ifacename will be <empty> because change between
442 reboots. It is used for software iSCSI's vlan or alias binding.
443 initiatorname: Initiatorname to be used if you want to override the
444 default one in /etc/iscsi/initiatorname.iscsi.
448 To display these values in a more friendly run:
450 iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07
451 # BEGIN RECORD 2.0-871
452 iface.iscsi_ifacename = cxgb3i.00:07:43:05:97:07
453 iface.net_ifacename = <empty>
454 iface.ipaddress = <empty>
455 iface.hwaddress = 00:07:43:05:97:07
456 iface.transport_name = cxgb3i
457 iface.initiatorname = <empty>
461 Before you can use the iface, you must set the IP address for the port
462 with the following command:
464 iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07 -o update -n iface.ipaddress -v 20.15.0.66
468 For the name of the value we want to update we use the name from
469 the "iscsiadm -m iface -I cxgb3i.00:07:43:05:97:07" command which is
473 Now, we can use this iface to login into targets, which is described in the
477 5.1.3 Discoverying iSCSI targets/portals
478 ========================================
480 Be aware that iscsiadm will use the default route to do discovery. It will
481 not use the iface specified. So if you are using a offload card, you will
482 need a seperate network connection to the target for discovery purposes.
483 *This will be fixed in the next version of open-iscsi*
485 When you run iscsiadm to do discovery, it will check for interfaces
486 in /etc/iscsi/ifaces and bind the portals that are discovered so that
487 they will be logged in through each iface. This behavior can also be overriden
488 by passing in the interfaces you want to use. For example if you had defined
489 two interface but only wanted to use one you can use the
490 --interface/-I argument:
492 iscsiadm -m discovery -t st -p ip:port -I iface1 -P 1
494 If you had defined interfaces but wanted the old behavior, where
495 we do not bind a session to a iface, then you can use the special iface
498 iscsiadm -m discovery -t st -p ip:port -I default -P 1
500 And if you did not define any interfaces in /etc/iscsi/ifaces and do
501 not pass anything into iscsiadm, running iscsiadm will do the default
502 behavior, where we allow the network subsystem to decide which
505 If you later want to remove the bindings for a specific target and
506 iface then you can run:
508 iscsiadm -m node -T my_target -I iface0 --op=delete
510 To do this for a specific portal on a target run:
512 iscsiadm -m node -T my_target -p ip:port -I iface0 --op=delete
514 If you wanted to delete all bindinds for iface0 then you can run
516 iscsiadm -m node -I iface0 --op=delete
518 And for equalogic targets it is sometimes useful to remove by just portal
520 iscsiadm -m node -p ip:port -I iface0 --op=delete
523 To now log into targets it is the same as with sofware iscsi. See section
524 7 for how to get started.
529 5.2 iscsiadm examples
530 =====================
531 Usage examples using the one-letter options (see iscsiadm man page
536 - SendTargets iSCSI Discovery using the default driver and interface:
538 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260
540 This will first search /etc/iscsi/ifaces for interfaces
541 using software iscsi. If any are found then nodes found during
542 discovery will be setup so that they can logged in through
545 - SendTargets iSCSI Discovery updating existing records:
547 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 \
550 If there a record for targetX and portalY exists in the DB, and
551 is returned during discovery, it will be updated with the info
552 from the iscsi.conf. No new portals will be added and stale
553 portals will not be removed.
555 - SendTargets iSCSI Discovery deleting existing records:
557 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 \
560 If there a record for targetX and portalY exists in the DB, but
561 is not returned during discovery it will be removed from the DB.
562 No new portals will be added and existing portal records will not
565 Note: If a session is logged into portal we are going to delete
566 a record for, it will be logged out then the record will be
569 - SendTargets iSCSI Discovery adding new records:
571 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 \
574 If there targetX and portalY is returned during discovery and does
575 not have a record, it will be added. Existing records are not
578 - SendTargets iSCSI Discovery using multiple ops:
580 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 \
583 This command will add new portals and delete records for portals
584 no longer returned. It will not change the record information for
587 - SendTargets iSCSI Discovery with a specific interface. If you
588 wish to only use a subset of the interfaces in /etc/iscsi/ifaces
589 then you can pass them in during discovery:
591 ./iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260 \
592 --interface=iface0 --interface=iface1
594 Note that for software iscsi, we let the network layer select
595 which NIC to use for discovery, but for later logins iscsiadm
596 will use the NIC defined in the iface config.
598 qla4xxx support is very basic and experimental. It does not store
599 the record info in the card's FLASH or the node DB, so you must
600 rerun discovery every time the driver is reloaded.
602 Node mode. In node mode you can specify which records you want to log
603 into by specifying the targetname, ip address, port or interface
604 (if specifying the interface it must already be setup in the node db).
605 iscsiadm will search the node db, for records which match the values
606 you pass in, so if you pass in the targetname and interface, iscsiadm
607 will search for records with those values and operate on only them.
608 Passing in none of them will result in all node records being operated on.
610 - iSCSI Login to all portals on every node/starget through each interface
613 ./iscsiadm -m node -l
615 - iSCSI login to all portals on a node/target through each interface set
618 ./iscsiadm -m node -T iqn.2005-03.com.max -l
620 - iSCSI login to a specific portal through each interface set in the db:
622 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -l
624 To specify a IPv6 address the following can be used:
626 ./iscsiadm -m node -T iqn.2005-03.com.max \
627 -p 2001:c90::211:9ff:feb8:a9e9 -l
629 The above command would use the default port, 3260. To specify a
630 port use the following:
632 ./iscsiadm -m node -T iqn.2005-03.com.max \
633 -p [2001:c90::211:9ff:feb8:a9e9]:3260 -l
635 - iSCSI Login to a specific portal through the NIC setup as iface0:
637 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
640 - iSCSI Logout to all portals on every node/starget through each interface
643 ./iscsiadm -m node -u
645 Warning: this does not check startup values like the logout/login all
646 option. Do not use this if you are running iscsi on your root disk.
648 - iSCSI logout to all portals on a node/target through each interface set
651 ./iscsiadm -m node -T iqn.2005-03.com.max -u
653 - iSCSI logout to a specific portal through each interface set in the db:
655 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 -u
657 - iSCSI Logout to a specific portal through the NIC setup as iface0:
659 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
662 - Changing iSCSI parameter:
664 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260 \
665 -o update -n node.cnx[0].iscsi.MaxRecvDataSegmentLength -v 65536
667 You can also change paramaters for multiple records at once, by
668 specifying different combinations of the target, portal and interface
671 - Adding custom iSCSI portal:
673 ./iscsiadm -m node -o new -T iqn.2005-03.com.max \
674 -p 192.168.0.1:3260,2 -I iface4
676 The -I/--interface is optional. If not passed in, "default" is used.
677 For tcp or iser, this would allow the network layer to decide what is
680 Note that for this command the target portal group tag (TPGT) should
681 be passed in. If it is not passed in on the initial creation command
682 then the user must run iscsiadm again to set the value. Also
683 if the TPGT is not initially passed in, the old behavior of not
684 tracking whether the record was statically or dynamically created
687 - Adding custom NIC config to multiple targets:
689 ./iscsiadm -m node -o new -I iface4
691 This command will add a interface config using the iSCSI and SCSI
692 settings from iscsid.conf to every target that is in the node db.
694 - Removing iSCSI portal:
696 ./iscsiadm -m node -o delete -T iqn.2005-03.com.max -p 192.168.0.4:3260
698 You can also delete multiple records at once, by specifying different
699 combinations of the target, portal and interface like above.
701 - Display iSCSI portal onfiguration:
703 ./iscsiadm -m node -T iqn.2005-03.com.max -p 192.168.0.4:3260
707 ./iscsiadm -m node -o show -T iqn.2005-03.com.max -p 192.168.0.4:3260
708 You can also display multiple records at once, by specifying different
709 combinations of the target, portal and interface like above.
711 Note: running "iscsiadm -m node" will only display the records. It
712 will not display the configuration info. You must run,
713 "iscsiadm -m node -o show".
715 - Show all node records:
719 This will print the nodes using the old flat format where the
720 interface and driver are not displayed. To display that info
721 use the -P argument with the arguent "1":
723 ./iscsiadm -m node -P 1
725 - Show all records in discovery database:
727 ./iscsiadm -m discovery
729 - Show all records in discovery database and show the targets that
730 were discovered from each record:
732 ./iscsiadm -m discovery -P 1
734 - Display discovery record setting:
736 ./iscsiadm -m discovery -p 192.168.0.4:3260
738 - Display session statistics:
740 ./iscsiadm -m session -r 1 --stats
742 This function also works in node mode. Instead of the "-r $sid"
743 argument, you would pass in the node info like targetname and/or portal,
746 - Perform a SCSI scan on a session
748 ./iscsiadm -m session -r 1 --rescan
750 This function also works in node mode. Instead of the "-r $sid"
751 argument, you would pass in the node info like targetname and/or portal,
754 Note: Rescanning does not delete old LUNs. It will only pick up new
757 - Display running sessions:
759 ./iscsiadm -m session -P 1
764 The default configuration file is /etc/iscsi/iscsid.conf. This file contains
765 only configuration that could be overwritten by iSCSI Discovery,
766 or manualy updated via iscsiadm utility. Its OK if this file does not
767 exist in which case compiled-in default configuration will take place
768 for newer discovered Target nodes.
770 See the man page and the example file for the current syntax.
771 The manpages for iscsid, iscsiadm are in the doc subdirectory and can be
772 installed in the appropriate man page directories and need to be manually
773 copied into e.g. /usr/local/share/man8.
777 There are three steps needed to set up a system to use iSCSI storage:
778 7.1. iSCSI startup using the init script or manual startup.
779 7.2. Discover targets.
780 7.3. Automate target logins for future system reboots.
782 The init scripts will start the iSCSI daemon and log into any
783 connections or nodes that are set up for automatic login. If your distro
784 does not have a init script, then you will have to start the daemon
785 and log into the targets manually.
787 7.1.1 iSCSI startup using the init script
788 -----------------------------------------------
792 To start open-iscsi in Red Hat/Fedora you can do:
794 service open-iscsi start
796 To get open-iscsi to automatically start at run time you may have to
798 chkconfig --level <levels> open-iscsi on
799 Where <levels> are the run levels.
801 And, to automatically mount a file system during startup
802 you must have the partition entry in /etc/fstab marked with the "_netdev"
803 option. For example this would mount a iscsi disk sdb:
805 /dev/sdb /mnt/iscsi ext3 _netdev 0 0
809 Otherwise, if there is a initd script for your distro in etc/initd that
810 gets installed with "make install"
812 /etc/init.d/open-iscsi start
814 will usually get you started.
816 7.1.2 Manual Startup:
817 ---------------------
819 7.1.2.1 Starting up the iSCSI daemon (iscsid) and loading modules:
820 -----------------------------------------------------------------
821 If there is no initd script, you must start the tools by hand. First load the
824 modprobe -q iscsi_tcp
826 after that start iSCSI daemon process:
830 or alternatively, start it with debug enabled and with output
831 redirected to the current console:
835 7.1.2.2 Logging into Targets:
836 ---------------------------
837 Use the configuration utility, iscsiadm, to add/remove/update Discovery
838 records, iSCSI Node records or monitor active iSCSI sessions (see above or the
839 iscsiadm man files and see section 7.2 below for how to discover targets).
843 will print out the nodes that have been discovered as:
845 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
846 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
850 ip:port,target_portal_group_tag targetname
852 If you are using the iface argument or want to see the driver
853 info use the following:
855 ./iscsiadm -m node -P 1
857 Target: iqn.1992-08.com.netapp:sn.33615311
858 Portal: 10.15.84.19:3260,2
860 Portal: 10.15.85.19:3260,3
866 Portal ip_address:port,tpgt
869 where targetname is the name of the target and ip_address:port is the address
870 and port of the portal. tpgt, is the portal group tag of
871 the portal, and is not used in iscsiadm commands except for static
872 record creation. And iface name is the name of the iscsi interface
873 defined in /etc/iscsi/ifaces. If no interface was defined in
874 /etc/iscsi/ifaces or passed in, the default behavior is used.
875 Default here is iscsi_tcp/tcp to be used over which ever NIC the
876 network layer decides is best.
878 To login, take the ip, port and targetname from above and run:
880 ./iscsiadm -m node -T targetname -p ip:port -l
882 In this example we would run
884 ./iscsiadm -m node -T iqn.1992-08.com.netapp:sn.33615311 -p 10.15.84.19:3260 -l
886 Note: drop the portal group tag from the "iscsiadm -m node" output.
888 7.2. Discover Targets
889 ---------------------
890 Once the iSCSI service is running, you can perform discovery using
893 iscsiadm -m discovery -t sendtargets -p ip:port
895 where "ip" is the address of the portal and port is the port.
897 Or you can you perform discovery using iSNS by setting the address
898 of the iSNS server in iscsid.conf with the "isns.address" value and
901 iscsiadm -m discovery -t isns
903 *** Warning *** iSNS support is experimental in this release. The command
904 line options for it will change in future releases.
906 Both commands will print out the list of all discovered targets and their
909 # iscsiadm -m discovery -t st -p 10.15.85.19:3260
910 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
911 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
913 The format for the output is:
915 ip:port,tpgt targetname
917 In this example, for the first target the ip address is 10.15.85.19.
918 The port is 3260. The target portal group is 3. The target name
919 is iqn.1992-08.com.netapp:sn.33615311.
921 If you would also like to see the iscsi inteface which will be used
922 for each session then use the --print[N] option.
924 iscsiadm -m discovery -t sendtargets -p ip:port -P 1
927 Target: iqn.1992-08.com.netapp:sn.33615311
928 Portal: 10.15.84.19:3260,2
930 Portal: 10.15.85.19:3260,3
933 In this example, The IP address of the first portal is 10.15.84.19.
934 The port is 3260. The target portal group is 3. The target name
935 is iqn.1992-08.com.netapp:sn.33615311. The iface being used is iface2.
937 While discovery targets are kept in the discovery db, they are
938 useful only for re-discovery. The discovered targets (a.k.a. nodes)
939 are stored as records in the node db.
941 The discovered targets are not logged into yet. Rather than logging
942 into the discovered nodes (making LUs from those nodes available as
943 storage), it is better to automate the login to the nodes we need.
945 If you wish to log into a target manually now, see section
946 "7.1.2.2 Logging in targets" above.
948 7.3. Automate Target Logins for Future System Statups
949 -----------------------------------------------------
950 Note: this may only work for distros with init scripts.
952 To automate login to a node, use the following with the record ID
953 (record ID is the targetname and portal) of the node discovered in the
955 iscsiadm -m node -T targetname -p ip:port --op update -n node.startup -v automatic
957 To set the automatic setting to all portals on a target through every
958 interface setup for each protal, the following can be run:
959 iscsiadm -m node -T targetname --op update -n node.startup -v automatic
961 Or to set the "node.startup" attribute to "startup" as default for
962 all sessions add the following to the /etc/iscsi/iscsid.conf:
964 node.startup = automatic
966 Setting this in iscsid.conf, will not affect existing nodes. It will only
967 affect nodes that are discovered after setting the value.
969 To login to all the automated nodes, simply restart the iscsi service:
970 e.g /etc/init.d/open-iscsi restart. On your next startup the nodes will
971 be logged into autmotically.
974 8. Advanced Configuration
975 =========================
977 8.1 iSCSI settings for dm-multipath
978 -----------------------------------
980 When using dm-multipath, the iSCSI timers should be set so that commands
981 are quickly failed to the dm-multipath layer. For dm-multipath you should
982 then set values like queue if no path, so that IO errors are retried and
983 queued if all paths are failed in the multipath layer.
986 8.1.1 iSCSI ping/Nop-Out settings
987 ---------------------------------
988 To quickly detect problems in the network, the iSCSI layer will send iSCSI
989 pings (iSCSI NOP-Out requests) to the target. If a NOP-Out times out the
990 iSCSI layer will respond by failing running commands and asking the SCSI
991 layer to requeue them if possible (SCSI disk commands get 5 retries if not
992 using multipath). If dm-multipath is being used the SCSI layer will fail
993 the command to the multipath layer instead of retrying. The multipath layer
994 will then retry the command on another path.
996 To control how often a NOP-Out is sent the following value can be set:
998 node.conn[0].timeo.noop_out_interval = X
1000 Where X is in seconds and the default is 10 seconds. To control the
1001 timeout for the NOP-Out the noop_out_timeout value can be used:
1003 node.conn[0].timeo.noop_out_timeout = X
1005 Again X is in seconds and the default is 15 seconds.
1007 Normally for these values you can use:
1009 node.conn[0].timeo.noop_out_interval = 5
1010 node.conn[0].timeo.noop_out_timeout = 10
1012 If there are a lot of IO error messages, then the above values may be too
1013 aggressive and you may need to increase the values for your network conditions
1014 and workload, or you may need to check your network for possible problems.
1017 8.1.2 replacement_timeout
1018 -------------------------
1019 The next iSCSI timer that will need to be tweaked is:
1021 node.session.timeo.replacement_timeout = X
1023 Here X is in seconds.
1025 replacement_timeout will control how long to wait for session re-establishment
1026 before failing pending SCSI commands and commands that are being operated on by
1027 the SCSI layer's error handler up to a higher level like multipath or to
1028 an application if multipath is not being used.
1031 8.1.2.1 Running Commands, the SCSI Error Handler, and replacement_timeout
1032 -------------------------------------------------------------------------
1033 Remember, from the Nop-out discussion that if a network problem is detected,
1034 the running commands are failed immediately. There is one exception to this
1035 and that is when the SCSI layer's error handler is running. To check if
1036 the SCSI error handler is running iscsiadm can be run as:
1038 iscsiadm -m session -P 3
1042 Host Number: X State: Recovery
1044 When the SCSI EH is running, commands will not be failed until
1045 node.session.timeo.replacement_timeout seconds.
1047 To modify the timer that starts the SCSI EH, you can either write
1048 directly to the device's sysfs file:
1050 echo X > /sys/block/sdX/device/timeout
1052 where X is in seconds or on most distros you can modify the udev rule.
1054 To modify the udev rule open /etc/udev/rules.d/50-udev.rules, and find the
1057 ACTION=="add", SUBSYSTEM=="scsi" , SYSFS{type}=="0|7|14", \
1058 RUN+="/bin/sh -c 'echo 60 > /sys$$DEVPATH/timeout'"
1060 And change the echo 60 part of the line to the value that you want.
1062 The default timeout for normal File System commands is 30 seconds when udev
1063 is not being used. If udev is used the default is the above value which
1064 is normally 60 seconds.
1067 8.1.2.2 Pending Commands and replacement_timeout
1068 ------------------------------------------------
1069 Commonly, the SCSI/BLOCK layer will queue 256 commands, but the path can
1070 only take 32. When a network problem is detected, the 32 commands
1071 in flight will be sent back to the SCSI layer immediately and because
1072 multipath is being used this will cause the commands to be sent to the multipath
1073 layer for execution on another path. However the other 96 commands that were
1074 still in the SCSI/BLOCK queue, will remain here until the session is
1075 re-established or until node.session.timeo.replacement_timeout seconds has
1076 gone by. After replacement_timeout seconds, the pending commands will be
1077 failed to the multipath layer, and all new incoming commands will be
1078 immediately failed back to the multipath layer. If a session is later
1079 re-established, then new commands will be queued and executed. Normally,
1080 multipathd's path tester mechanism will detect that the session has been
1081 re-established and the path is accessible again, and it will inform
1085 8.1.3 Optimal replacement_timeout Value
1086 ---------------------------------------
1088 The default value for replacement_timeout is 120 seconds, but because
1089 multipath's queue_if_no_path and no_path_retry setting can prevent IO errors
1090 from being propagated to the application, replacement_timeout can be set to a
1091 shorter value like 5 to 15 seconds. By setting it lower pending IO is quickly
1092 sent to a new path and executed while the iSCSI layer attempts
1093 re-establishment of the session. If all paths end up being failed, then the
1094 multipath and device mapper layer will internally queue IO based on the
1095 multipath.conf settings, instead of the iSCSI layer.
1098 8.2 iSCSI settings for iSCSI root
1099 ---------------------------------
1101 When accessing the root partition directly through a iSCSI disk, the
1102 iSCSI timers should be set so that iSCSI layer has several chances to try to
1103 re-establish a session and so that commands are not quickly requeued to
1104 the SCSI layer. Basically you want the opposite of when using dm-multipath.
1106 For this setup, you can turn off iSCSI pings by setting:
1108 node.conn[0].timeo.noop_out_interval = 0
1109 node.conn[0].timeo.noop_out_timeout = 0
1111 And you can turn the replacement_timer to a very long value:
1113 node.session.timeo.replacement_timeout = 86400
1116 9. iSCSI System Info
1117 ====================
1119 To get information about the running sessions: including the session and
1120 device state, session ids (sid) for session mode, and some of the
1121 negotiated parameters, run:
1123 iscsiadm -m session -P 2
1125 If you are looking for something shorter like just the sid to node mapping
1128 iscsiadm -m session -P 0
1132 This will print the list of running sessions with the format:
1134 driver [sid] ip:port,target_portal_group_tag targetname
1136 # iscsiadm -m session
1137 tcp [2] 10.15.84.19:3260,2 iqn.1992-08.com.netapp:sn.33615311
1138 tcp [3] 10.15.85.19:3260,3 iqn.1992-08.com.netapp:sn.33615311
1140 To print the hw address info use the -P option with "1":
1142 iscsiadm -m session -P 1
1144 This will print the sessions with the following format:
1146 Current Portal: portal currently logged into
1147 Persistent Portal: portal we would fall back to if we had got redirected during login
1148 Iface Transport: driver/transport_name
1149 Iface IPaddress: IP address of iface being used
1150 Iface HWaddress: HW address used to bind session
1151 Iface Netdev: netdev value used to bind session
1152 SID: iscsi sysfs session id
1153 iSCSI Connection State: iscsi state
1155 Note: if a older kernel is being used or if the session is not bound
1156 then the keyword "default" is print to indicate that the default
1157 network behavior is being used.
1160 #iscsiadm -m session -P 1
1161 Target: iqn.1992-08.com.netapp:sn.33615311
1162 Current Portal: 10.15.85.19:3260,3
1163 Persistent Portal: 10.15.85.19:3260,3
1164 Iface Transport: tcp
1165 Iface IPaddress: 10.11.14.37
1166 Iface HWaddress: default
1167 Iface Netdev: default
1169 iSCSI Connection State: LOGGED IN
1170 Internal iscsid Session State: NO CHANGE
1172 The connection state is currently not available for qla4xxx.
1175 To get a HBA/Host view of the session there is the host mode.
1180 cxgb3i: [7] 10.10.15.51,[00:07:43:05:97:07],eth3 <empty>
1182 This prints the list of iSCSI hosts in the system with the format:
1183 driver [hostno] ipaddress,[hwaddress],net_ifacename,initiatorname
1186 To print this info in a more user friendly way the -P argument can be used:
1188 iscsiadm -m host -P 1
1192 Initiatorname: <empty>
1193 IPaddress: 10.10.15.51
1194 HWaddress: 00:07:43:05:97:07
1197 Here, you can also see the sate of the host.
1199 You can also pass in any value from 1 - 4 to print more info like the
1200 sessions running through the host, what ifaces are being used and what
1201 devices are accessed through it.
1203 To print the info for a specific host then you can pass in the -H argument
1204 with the host number:
1205 iscsiadm -m host -P 1 -H 7