[2020-02-23 05:12:01]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-02-23 05:12:02]

 | 1-11|   OOOOOOOO | 1-21|   OOOOOOOO | 1-31|   OOOOO... | 1-41|   OOOOOOOO | 1-51|   OOOOOOOO
 | 1-12|   OOOOOOOO | 1-22|   OOOOOOOO | 1-32|   OOOOO... | 1-42|   OOOOOOOO | 1-52|   OOOOOOOO
 | 1-13|   OOOOOOOO | 1-23|   OOOOOOOO | 1-33|   OOOOO... | 1-43|   OOOOOOOO | 1-53|   OOOOOOOO
 | 1-14|   OOOOOOOO | 1-24|   OOOOOOOO | 1-34|   OOOO.... | 1-44|   OOOOOOOO | 1-54|   OOOOOOO.
 | 1-15|   OOOOOOOO | 1-25|   OOOOOOOO | 1-35|   OOOOOOOO | 1-45|   OOOOOOOO | 1-55|   OOOOOOOO
 | 1-16|   OOOOOOOO | 1-26|   OOOOO... | 1-36|   OOOOOOOO | 1-46|   OOOOOOOO | 1-56|   OOOOOOOO
 | 1-17|   OOOOOOOO | 1-27|   OOOOO... | 1-37|   OOOOOOOO | 1-47|   OOOOOOOO | 1-57|   OOOOOOOO
 | 1-18|   OOOOOOOO | 1-28|   OOOOO... | 1-38|   OOOOOOOO | 1-48|   OOOOOOO. | 1-58|   OOOOOOO.


 | 3-11|   OOOOOOO. | 3-21|   OOOOOO.. | 3-31|   OOOOO... | 3-41|   OOOOOOOO | 3-51|   OOOOOOOO | 3-61|   OOOOOOOO | 3-71|   OOOOOOOO
 | 3-12|   OOOOOOO. | 3-22|   OOOOOOO. | 3-32|   OOOOO... | 3-42|   ........ | 3-52|   OOOOOOOO | 3-62|   OOOOOOOO | 3-72|   OOOOOOOO
 | 3-13|   OOOOOOO. | 3-23|   OOOOOOO. | 3-33|   OOOOO... | 3-43|   OOOOOOO. | 3-53|   OOOOOOOO | 3-63|   OOOOOOOO | 3-73|   OOOOOOOO
 | 3-14|   OOOOOOO. | 3-24|   OOOOOOO. | 3-34|   OOOOO... | 3-44|   OOOOOO.. | 3-54|   OOOOOOOO | 3-64|   OOOOOOOO | 3-74|   O.......
 | 3-15|   OOOOO... | 3-25|   OOOOOOO. | 3-35|   OOOOO... | 3-45|   OOOOOOOO | 3-55|   O....... | 3-65|   OOOOOO.. | 3-75|   OOOOOOOO
 | 3-16|   OOOOO... | 3-26|   OOOOO... | 3-36|   OOOOO... | 3-46|   OOOOOOOO | 3-56|   OOOOOOOO | 3-66|   OOOOOOOO | 3-76|   OOOOOOOO
 | 3-17|   OOOOOOOO | 3-27|   OOOOO... | 3-37|   OOOOO... | 3-47|   OOOOOOOO | 3-57|   OOOOOOOO | 3-67|   OOOOOOOO | 3-77|   O.......
 | 3-18|     DOWN   | 3-28|   OOOOO... | 3-38|   OOOOO... | 3-48|   OOOOOOOO | 3-58|   OOOOOOOO | 3-68|   OOOOOOOO | 3-78|   ........


 | 4-11|   OOOOOOOO | 4-21|   OOOOOOO. | 4-31|   OOOOOOOO | 4-41|   OOOOOOOO | 4-51|   OOOOOOOO | 4-61|   OOOOOOOO | 4-71|   OOOOOOOO | 4-81|   ........
 | 4-12|   OOOOOOOO | 4-22|   OOOOOOOO | 4-32|   OOOOOOOO | 4-42|   OOOOOOOO | 4-52|   OOOOOOOO | 4-62|   OOOOOOOO | 4-72|   OOOOOOOO | 4-82|   OOOO....
 | 4-13|   OOOOOOOO | 4-23|   OOO..... | 4-33|   OOOOOOOO | 4-43|   OOOOOOOO | 4-53|   OOOOOOOO | 4-63|   OOOOOOOO | 4-73|   OOOOOOOO | 4-83|   ........
 | 4-14|   OOOOOOOO | 4-24|   OOOOOOOO | 4-34|   OOOOOOOO | 4-44|   OOOOOOOO | 4-54|   OOOOOOOO | 4-64|   OOOOOOOO | 4-74|   OOOOOOOO                   


 | 5- 1|   OOOOOOOO | 5-11|   OOOOOOOO | 5-21|   OOOOOOO.
 | 5- 2|   OOOOOOOO | 5-12|   OOOOOOOO | 5-22|   OOOOOOO.
 | 5- 3|   OOOOOOOO | 5-13|   OOOOOOOO | 5-23|   OOOOOOOO
 | 5- 4|   OOOOOOOO | 5-14|   OOOOOOOO | 5-24|   OOOOOOOO
 | 5- 5|   OOOOOOOO | 5-15| * O....... | 5-25|   OOOOOOO.
 | 5- 6|   OOOOOOOO | 5-16|   OOOOOOOO | 5-26|   OOOOOOOO
 | 5- 7|   OOOOOOOO | 5-17|   OOOOOOO. | 5-27|   OOOOOOOO
 | 5- 8|   OOOOOOOO | 5-18|   OOOOOOO. | 5-28|   OOOOOOO.
 | 5- 9|   OOOOOOOO | 5-19|   OOOOOOOO | 5-29|   OOOOOOO.
 | 5-10|   OOOOOOOO | 5-20|   OOOOOOO. | 5-30|   OOOOOOO.


 | 9- 1|   ........
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   ........
 |10- 2|   ........


Totals:
Total cores: 3504 
Total memory: 34.75 TiB 

Down:
Down cores: 48 (  1.37%) 
Down memory: 0.47 TiB (  1.35%) 

Used:
Used cores: 2968 ( 84.70%) 
Used memory: 7.94 TiB ( 22.85%) 

Utilization excluding down
CPU utilization:  85.88% (global)  85.08% (average per node)
Memory utilization:  23.17% (global)  27.64% (average per node)

Average per node utilization (excluding down):  85.80% 
[2020-02-23 05:12:02]
s03n18 Not responding 
s05n15 SlurmdSpoolDir is full 

[2020-02-23 05:10:15]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-02-23 05:00:04]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       38595.2GB       41412.8GB    48% [##########          ]      1490.3M      1488.4M  100%
      9501       3002       normal       80008.1GB       40965.2GB       39042.9GB    51% [##########          ]      1490.3M      1488.4M  100%
      9501       3003       normal       80008.1GB       40468.2GB       39539.9GB    51% [##########          ]      1490.3M      1488.4M  100%
      9501       3004       normal       80008.1GB       40728.2GB       39279.8GB    51% [##########          ]      1490.3M      1488.4M  100%
      9501       3005       normal       80008.1GB       42600.5GB       37407.5GB    53% [###########         ]      1490.3M      1488.3M  100%
      9501       3006       normal       80008.1GB       40622.3GB       39385.8GB    51% [##########          ]      1490.3M      1488.3M  100%
      9501       3007       normal       80008.1GB       38619.5GB       41388.6GB    48% [##########          ]      1490.3M      1488.5M  100%
      9501       3008       normal       80008.1GB       46477.9GB       33530.2GB    58% [############        ]      1490.3M      1488.3M  100%
      9501       3009       normal       80008.1GB       42490.0GB       37518.1GB    53% [###########         ]      1490.3M      1488.3M  100%
      9501       3010       normal       80008.1GB       41639.5GB       38368.5GB    52% [##########          ]      1490.3M      1488.3M  100%
      9501       3011       normal       80008.1GB       42824.4GB       37183.7GB    54% [###########         ]      1490.3M      1488.3M  100%
      9501       3012       normal       80008.1GB       42606.6GB       37401.5GB    53% [###########         ]      1490.3M      1488.3M  100%
      9501       3013       normal       80008.1GB       42773.6GB       37234.5GB    53% [###########         ]      1490.3M      1488.3M  100%
      9502       3014       normal       80008.1GB       43290.3GB       36717.8GB    54% [###########         ]      1490.3M      1488.3M  100%
      9502       3015       normal       80008.1GB       44790.0GB       35218.0GB    56% [###########         ]      1490.3M      1488.3M  100%
      9502       3016    emergency       80008.1GB        3615.5GB       76392.6GB     5% [#                   ]      1490.3M      1490.2M  100%
      9502       3017       normal       80008.1GB       43122.4GB       36885.6GB    54% [###########         ]      1490.3M      1488.3M  100%
      9502       3018       normal       80008.1GB       43993.8GB       36014.3GB    55% [###########         ]      1490.3M      1488.3M  100%
      9502       3019       normal       80008.1GB       42855.2GB       37152.9GB    54% [###########         ]      1490.3M      1488.3M  100%
      9502       3020       normal       80008.1GB       43054.3GB       36953.8GB    54% [###########         ]      1490.3M      1488.3M  100%
      9601       3021       normal       80008.1GB       40518.5GB       39489.6GB    51% [##########          ]      1490.3M      1488.3M  100%
      9601       3022       normal       80008.1GB       43993.9GB       36014.1GB    55% [###########         ]      1490.3M      1488.3M  100%
      9601       3023       normal       80008.1GB       41590.0GB       38418.1GB    52% [##########          ]      1490.3M      1488.3M  100%
      9601       3024       normal       80008.1GB       41570.2GB       38437.8GB    52% [##########          ]      1490.3M      1488.3M  100%
      9601       3025       normal       80008.1GB       42146.6GB       37861.4GB    53% [###########         ]      1490.3M      1488.3M  100%
      9601       3026       normal       80008.1GB       41143.7GB       38864.4GB    51% [##########          ]      1490.3M      1488.3M  100%
      9601       3027       normal       80008.1GB       39582.2GB       40425.9GB    49% [##########          ]      1490.3M      1488.3M  100%
      9601       3028       normal       80008.1GB       41940.5GB       38067.6GB    52% [##########          ]      1490.3M      1488.3M  100%
      9601       3029       normal       80008.1GB       41080.5GB       38927.6GB    51% [##########          ]      1490.3M      1488.3M  100%
      9601       3030       normal       80008.1GB       41171.0GB       38837.1GB    51% [##########          ]      1490.3M      1488.4M  100%
      9601       3031       normal       80008.1GB       41154.3GB       38853.8GB    51% [##########          ]      1490.3M      1488.4M  100%
      9601       3032       normal       80008.1GB       39668.6GB       40339.5GB    50% [##########          ]      1490.3M      1488.3M  100%
      9601       3033       normal       80008.1GB       38235.5GB       41772.6GB    48% [##########          ]      1490.3M      1488.4M  100%
      9601       3034       normal       80008.1GB       42332.9GB       37675.1GB    53% [###########         ]      1490.3M      1488.4M  100%
      9602       3035       normal       80008.1GB       43403.2GB       36604.9GB    54% [###########         ]      1490.3M      1488.4M  100%
      9602       3036       normal       80008.1GB       40562.1GB       39446.0GB    51% [##########          ]      1490.3M      1488.4M  100%
      9602       3037       normal       80008.1GB       40968.0GB       39040.1GB    51% [##########          ]      1490.3M      1488.4M  100%
      9602       3038       normal       80008.1GB       40723.2GB       39284.9GB    51% [##########          ]      1490.3M      1488.5M  100%
      9602       3039       normal       80008.1GB       42107.1GB       37901.0GB    53% [###########         ]      1490.3M      1488.4M  100%
      9602       3040       normal       80008.1GB       39453.2GB       40554.9GB    49% [##########          ]      1490.3M      1488.4M  100%
      9602       3041       normal       80008.1GB       38561.4GB       41446.7GB    48% [##########          ]      1490.3M      1488.4M  100%
      9701       3042          low      119987.9GB       88709.8GB       31278.0GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3043          low      119987.9GB       86714.1GB       33273.8GB    72% [##############      ]      2235.0M      2226.1M  100%
      9701       3044          low      119987.9GB       91443.4GB       28544.5GB    76% [###############     ]      2235.0M      2226.1M  100%
      9701       3045          low      119987.9GB       88928.1GB       31059.8GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3046          low      119987.9GB       89629.4GB       30358.4GB    75% [###############     ]      2235.0M      2226.1M  100%
      9701       3047          low      119987.9GB       87336.9GB       32651.0GB    73% [###############     ]      2235.0M      2226.1M  100%
      9701       3048          low      119987.9GB       89161.6GB       30826.2GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3049          low      119987.9GB       87777.6GB       32210.3GB    73% [###############     ]      2235.0M      2226.1M  100%
      9701       3050          low      119987.9GB       89529.3GB       30458.5GB    75% [###############     ]      2235.0M      2226.1M  100%
      9701       3051          low      119987.9GB       89302.1GB       30685.8GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3052          low      119987.9GB       89709.3GB       30278.5GB    75% [###############     ]      2235.0M      2226.1M  100%
      9701       3053          low      119987.9GB       88309.6GB       31678.3GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3054          low      119987.9GB       89171.0GB       30816.9GB    74% [###############     ]      2235.0M      2226.1M  100%
      9701       3055          low      119987.9GB       89986.6GB       30001.2GB    75% [###############     ]      2235.0M      2226.1M  100%
      9702       3056          low      119987.9GB       88884.3GB       31103.5GB    74% [###############     ]      2235.0M      2226.1M  100%
      9702       3057          low      119987.9GB       87962.0GB       32025.8GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3058          low      119987.9GB       89280.3GB       30707.6GB    74% [###############     ]      2235.0M      2226.1M  100%
      9702       3059          low      119987.9GB       89087.6GB       30900.3GB    74% [###############     ]      2235.0M      2226.1M  100%
      9702       3060          low      119987.9GB       87889.3GB       32098.5GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3061          low      119987.9GB       89829.5GB       30158.4GB    75% [###############     ]      2235.0M      2226.1M  100%
      9702       3062          low      119987.9GB       87529.5GB       32458.4GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3063          low      119987.9GB       90063.9GB       29924.0GB    75% [###############     ]      2235.0M      2226.1M  100%
      9702       3064          low      119987.9GB       87750.5GB       32237.4GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3065          low      119987.9GB       87409.5GB       32578.4GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3066          low      119987.9GB       91897.7GB       28090.1GB    77% [###############     ]      2235.0M      2226.1M  100%
      9702       3067          low      119987.9GB       89552.5GB       30435.3GB    75% [###############     ]      2235.0M      2226.1M  100%
      9702       3068          low      119987.9GB       88141.8GB       31846.1GB    73% [###############     ]      2235.0M      2226.1M  100%
      9702       3069          low      119987.9GB       87219.1GB       32768.7GB    73% [###############     ]      2235.0M      2226.1M  100%
      9801       3070          low      119987.9GB       87360.6GB       32627.2GB    73% [###############     ]      2235.0M      2226.1M  100%
      9801       3071          low      119987.9GB       71694.7GB       48293.2GB    60% [############        ]      2235.0M      2229.1M  100%
      9801       3072          low      119987.9GB       88823.7GB       31164.2GB    74% [###############     ]      2235.0M      2226.1M  100%
      9801       3073          low      119987.9GB       89124.6GB       30863.2GB    74% [###############     ]      2235.0M      2226.1M  100%
      9801       3074          low      119987.9GB       89831.1GB       30156.7GB    75% [###############     ]      2235.0M      2226.1M  100%
      9801       3075          low      119987.9GB       87836.6GB       32151.2GB    73% [###############     ]      2235.0M      2226.1M  100%
      9801       3076          low      119987.9GB       86775.4GB       33212.5GB    72% [##############      ]      2235.0M      2226.1M  100%
      9801       3077          low      119987.9GB       89842.1GB       30145.7GB    75% [###############     ]      2235.0M      2226.1M  100%
      9801       3078          low      119987.9GB       88773.4GB       31214.5GB    74% [###############     ]      2235.0M      2226.1M  100%
      9801       3079          low      119987.9GB       87385.6GB       32602.3GB    73% [###############     ]      2235.0M      2226.1M  100%
      9801       3080          low      119987.9GB       72547.7GB       47440.2GB    60% [############        ]      2235.0M      2229.0M  100%
      9801       3081          low      119987.9GB       89345.8GB       30642.1GB    74% [###############     ]      2235.0M      2226.1M  100%
      9801       3082          low      119987.9GB       87980.9GB       32007.0GB    73% [###############     ]      2235.0M      2226.1M  100%
      9801       3083          low      119987.9GB       90314.5GB       29673.3GB    75% [###############     ]      2235.0M      2226.1M  100%
      9802       3084          low      119987.9GB       88604.5GB       31383.3GB    74% [###############     ]      2235.0M      2226.1M  100%
      9802       3085          low      119987.9GB       90141.7GB       29846.2GB    75% [###############     ]      2235.0M      2226.1M  100%
      9802       3086          low      119987.9GB       89504.9GB       30483.0GB    75% [###############     ]      2235.0M      2226.1M  100%
      9802       3087          low      119987.9GB       87407.6GB       32580.3GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3088          low      119987.9GB       89281.0GB       30706.9GB    74% [###############     ]      2235.0M      2226.1M  100%
      9802       3089          low      119987.9GB       89983.3GB       30004.5GB    75% [###############     ]      2235.0M      2226.1M  100%
      9802       3090          low      119987.9GB       87163.0GB       32824.8GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3091          low      119987.9GB       87516.8GB       32471.1GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3092          low      119987.9GB       88148.3GB       31839.6GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3093          low      119987.9GB       87628.5GB       32359.4GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3094          low      119987.9GB       89053.8GB       30934.0GB    74% [###############     ]      2235.0M      2226.1M  100%
      9802       3095          low      119987.9GB       88177.5GB       31810.4GB    73% [###############     ]      2235.0M      2226.1M  100%
      9802       3096          low      119987.9GB       89434.2GB       30553.6GB    75% [###############     ]      2235.0M      2226.1M  100%
      9802       3097          low      119987.9GB       89045.9GB       30942.0GB    74% [###############     ]      2235.0M      2226.1M  100%
      9502       3098       normal      119993.9GB       41160.4GB       78833.5GB    34% [#######             ]      2235.1M      2233.2M  100%
      9502       3099       normal      119993.9GB       40997.0GB       78996.8GB    34% [#######             ]      2235.1M      2233.2M  100%
      9502       3100       normal      119993.9GB       41676.7GB       78317.2GB    35% [#######             ]      2235.1M      2233.2M  100%
      9502       3101       normal      119993.9GB       40636.5GB       79357.4GB    34% [#######             ]      2235.1M      2233.2M  100%
      9502       3102       normal      119993.9GB       39380.1GB       80613.8GB    33% [#######             ]      2235.1M      2233.2M  100%
      9502       3103       normal      119993.9GB       41132.3GB       78861.6GB    34% [#######             ]      2235.1M      2233.2M  100%
      9502       3104       normal      119993.9GB       42316.2GB       77677.7GB    35% [#######             ]      2235.1M      2233.2M  100%
      9602       3105       normal      119993.9GB       41884.1GB       78109.8GB    35% [#######             ]      2235.1M      2233.2M  100%
      9602       3106       normal      119993.9GB       40196.1GB       79797.8GB    33% [#######             ]      2235.1M      2233.2M  100%
      9602       3107       normal      119993.9GB       40818.1GB       79175.8GB    34% [#######             ]      2235.1M      2233.2M  100%
      9602       3108       normal      119993.9GB       42242.2GB       77751.6GB    35% [#######             ]      2235.1M      2233.2M  100%
      9602       3109       normal      119993.9GB       42199.2GB       77794.7GB    35% [#######             ]      2235.1M      2233.2M  100%
      9602       3110       normal      119993.9GB       40718.3GB       79275.6GB    34% [#######             ]      2235.1M      2233.2M  100%
      9602       3111       normal      119993.9GB       39656.5GB       80337.4GB    33% [#######             ]      2235.1M      2233.2M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7179986.7 GB
Total free space on /faststorage        4499578.8 GB

                                 Use%   61.47%

2020-02-23 05:13:07.453387110 +01:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-02-22 04:18:02] NOTICE !!! fs5 responding again via IB     
[2020-02-22 04:16:02] WARNING !!! fs5 not responding via IB     
[2020-02-21 16:42:02] s05n15 now marked for drain: SlurmdSpoolDir is full 
[2020-02-20 04:50:01] s01n14 resumed
[2020-02-20 04:46:01] s01n14 now marked for drain: sanity-check: ls for faststorage took 18.7167 sec 
[2020-02-20 03:06:02] s01n52 resumed
[2020-02-20 03:04:02] s01n36 resumed
[2020-02-20 02:56:02] s01n47 resumed
[2020-02-20 02:52:02] s01n48 resumed
[2020-02-20 02:50:02] s01n15 resumed
[2020-02-20 02:48:02] s01n58 resumed
[2020-02-20 02:46:01] s01n24 resumed
[2020-02-20 02:44:02] s01n11 resumed
[2020-02-20 02:42:02] s01n46 resumed
[2020-02-20 02:42:02] s01n42 resumed
[2020-02-20 02:40:02] s01n37 resumed
[2020-02-20 02:38:01] s01n47 now marked for drain: sanity-check: ls for faststorage took 286.26 sec 
[2020-02-20 02:38:01] s01n44 resumed
[2020-02-20 02:38:01] s01n41 resumed
[2020-02-20 02:36:02] s01n48 now marked for drain: sanity-check: ls for faststorage took 273.445 sec 
[2020-02-20 02:36:02] s01n44 now marked for drain: sanity-check: ls for faststorage took 142.848 sec 
[2020-02-20 02:36:02] s01n21 resumed
[2020-02-20 02:36:02] s01n18 resumed
[2020-02-20 02:30:02] s01n58 now marked for drain: sanity-check: ls for faststorage took 247.798 sec 
[2020-02-20 02:30:02] s01n17 resumed
[2020-02-20 02:30:02] s01n12 resumed
[2020-02-20 02:28:02] s01n37 now marked for drain: sanity-check: ls for faststorage took 227.177 sec 
[2020-02-20 02:28:02] s01n26 resumed
[2020-02-20 02:28:02] s01n14 resumed
[2020-02-20 02:28:02] s01n13 resumed
[2020-02-20 02:26:02] s01n16 resumed
[2020-02-20 02:24:02] s01n52 now marked for drain: sanity-check: ls for faststorage took 238.323 sec 
[2020-02-20 02:24:02] s01n46 now marked for drain: sanity-check: ls for faststorage took 218.453 sec 
[2020-02-20 02:24:02] s01n11 now marked for drain: sanity-check: ls for faststorage took 324.395 sec 
[2020-02-20 02:22:01] s01n36 now marked for drain: sanity-check: ls for faststorage took 332.798 sec 
[2020-02-20 02:20:02] s01n21 now marked for drain: sanity-check: ls for faststorage took 197.292 sec 
[2020-02-20 02:18:02] s01n45 resumed
[2020-02-20 02:18:02] s01n41 now marked for drain: sanity-check: ls for faststorage took 319.05 sec 
[2020-02-20 02:18:02] s01n24 now marked for drain: sanity-check: ls for faststorage took 328.053 sec 
[2020-02-20 02:16:02] s01n42 now marked for drain: sanity-check: ls for faststorage took 303.687 sec 
[2020-02-20 02:16:02] s01n32 resumed
[2020-02-20 02:14:01] s01n12 now marked for drain: sanity-check: ls for faststorage took 259.063 sec 
[2020-02-20 02:12:02] s01n16 now marked for drain: sanity-check: ls for faststorage took 252.828 sec 
[2020-02-20 02:12:02] s01n14 now marked for drain: sanity-check: ls for faststorage took 169.487 sec 
[2020-02-20 02:08:02] s01n17 now marked for drain: sanity-check: ls for faststorage took 245.957 sec 
[2020-02-20 02:02:02] s01n26 now marked for drain: sanity-check: ls for faststorage took 275.688 sec 
[2020-02-20 01:58:02] s01n13 now marked for drain: sanity-check: ls for faststorage took 231.372 sec 
[2020-02-20 01:56:01] s01n15 now marked for drain: sanity-check: ls for faststorage took 251.188 sec 
[2020-02-20 01:52:02] s01n32 now marked for drain: sanity-check: ls for faststorage took 212.745 sec 
[2020-02-20 01:44:02] s01n58 resumed
[2020-02-20 01:42:02] s01n18 now marked for drain: sanity-check: ls for faststorage took 185.39 sec 
[2020-02-23 05:00:03]
Backup on    fs1 initiated 0-06:30:00 ago
Backup on    fs2 initiated 3-11:15:01 ago
Backup on    fs3 initiated 14-03:00:01 ago
Backup on    fs5 initiated 10-07:00:01 ago
Backup on    fs6  finished 1-00:46:07 ago
Backup on s95n01  finished 0-06:52:42 ago
[2020-02-23 05:12:03]
################
# RUNNING JOBS #
################
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT
-----  ----      --------  ----------  ---------
1      agomez    1         4G          1-00:00:00
1      cactus    12        2G          8-08:00:00
1      clara     24        256G        5-00:00:00
1      clara     36        384G        5-00:00:00
1      iveera    1         8G          5-00:00:00
1      kalle     1         128G        4-04:00:00
1      leandro   8         32G         5-00:00:00
1      lucie     1         6G          4-04:00:00
1      michaelk  1         4G          12:00:00
1      michaelk  1         4G          2:00:00
1      michaelk  1         8G          2:00:00
1      michaelk  2         16G         1:00:00
1      michaelk  2         8G          12:00:00
1      michaelk  4         8G          12:00:00
1      michaelk  8         16G         4:00:00
1      rohde     1         25G         8-08:00:00
1      vang      1         16G         2-00:00:00
1      vang      8         64G         2-00:00:00
2      agomez    16        32G         8:00:00
2      michaelk  16        64G         2-00:00:00
2      michaelk  1         8G          1:00:00
2      michaelk  1         8G          4:00:00
2      vang      1         8G          12:00:00
2      vang      8         64G         2:00:00
3      lefa      4         64G         20:00:00
3      lucie     1         110G        12-12:00:00
4      michaelk  36        128G        12-00:00:00
4      vang      1         8G          2:00:00
6      michaelk  1         1G          1:00:00
7      vang      16        64G         2:00:00
8      michaelk  16        16G         12:00:00
9      dreamlab  1         2G          10-00:00:00
9      igrove    1         14G         16-16:00:00
9      michaelk  1         2G          4:00:00
10     vang      4         32G         12:00:00
12     michaelk  2         32G         12:00:00
17     palle     4         32G         2-00:00:00
20     plamy     8         32G         10-00:00:00
24     michaelk  4         8G          2:00:00
32     vang      1         16G         2:00:00
73     maretty   10        16G         12:00:00
1183   besen     1         1G          1-00:00:00
###############
# QUEUED JOBS #
###############
Count  USER     MIN_CPUS  MIN_MEMORY  TIMELIMIT
-----  ----     --------  ----------  ---------
1      lefa     24        124G        1-12:00:00
2      lefa     4         64G         20:00:00
20     palle    4         32G         2-00:00:00
1970   agomez   16        32G         8:00:00
2374   maretty  10        16G         12:00:00
8857   besen    1         1G          1-00:00:00
 
#################
# CPUs per User #
#################
NCPUs  USER      QOS
-----  ----      ---
1      iveera    qos1
1      kalle     qos1
1      rohde     basic
4      lucie     qos1
8      leandro   qos1
9      dreamlab  basic
9      igrove    qos2
12     cactus    basic
12     lefa      bumped
20     maretty   bumped
24     clara     bumped
25     michaelk  bumped
33     agomez    bumped
36     clara     qos1
68     palle     qos1
160    plamy     basic
176    michaelk  gold
215    vang      gold
261    michaelk  qos1
710    maretty   qos1
1183   besen     qos1
########################
# QUEUED CPUs per User #
########################
NCPUs  USER     QOS
-----  ----     ---
8      lefa     basic
24     lefa     bumped
80     palle    qos1
8857   besen    qos1
23740  maretty  qos1
31520  agomez   qos1
[2020-02-11 12:05:41]
### Node diagnostics (experimental) ###