[2020-06-05 17:24:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-06-05 17:24:03]

 | 1-11|   OOOOOO.. | 1-21|   OOOOOOOO | 1-31|   OOOOOO.. | 1-41|   ........ | 1-51|   OOOOOOOO | 1-61|   OOOOOOOO
 | 1-12|   OOOOOOOO | 1-22|   OOOOOOOO | 1-32|   OOOO.... | 1-42|   OOOOOOOO | 1-52|   OOOOOOOO | 1-62|   OOOO....
 | 1-13|   OOOOOOOO | 1-23|   OOOOOOOO | 1-33|   OOOOOOOO | 1-43|   OOOOOOOO | 1-53|   OOOOOOOO | 1-63|   OOOOOOOO
 | 1-14|   OOOO.... | 1-24|   OOO..... | 1-34|   OOOOOOOO | 1-44|   OOOOOOOO | 1-54|   OOOOOOOO | 1-64|   OOOOOOOO
 | 1-15|   OOOO.... | 1-25|   OOOOOO.. | 1-35|   OOOOOOO. | 1-45|   OOOOOOOO | 1-55|   OOOOOOOO | 1-65|   OOOO....
 | 1-16|   OO...... | 1-26|   OOOOOOOO | 1-36|   OOOOOOOO | 1-46|   OOOOOOOO | 1-56|   OOOOOOOO | 1-66|   OOOOOOOO
 | 1-17|   O....... | 1-27|   OOOOOOOO | 1-37|   OOOOOO.. | 1-47|   OOOOOOOO | 1-57|   OOOOOOOO | 1-67|   OOOOOOOO
 | 1-18|   O....... | 1-28|   OO...... | 1-38|   O....... | 1-48|   OOOOOOOO | 1-58|   OOOOOOOO                   


 | 3-11|   O....... | 3-21|   OOOOOOOO | 3-31|   OOOOOOOO | 3-41|   OOOO.... | 3-51|   OOOOOOOO | 3-61|   OOOOOOOO | 3-71|   OOOOOOOO
 | 3-12|   OO...... | 3-22|   OOOOOOOO | 3-32|   OOOOOOOO | 3-42|   OOOO.... | 3-52|   OOOOOOOO | 3-62|   OOOOOOOO | 3-72|   OOOOO...
 | 3-13|     DOWN   | 3-23|   OOOOOOOO | 3-33|   OOOOOOOO | 3-43|   OOOOOOOO | 3-53|   OOOOOOOO | 3-63|   OOOOOOOO | 3-73|   OOOOOOOO
 | 3-14|   OOOOOOOO | 3-24|   OOOOOOOO | 3-34|   OOOOOOOO | 3-44|   OOOOOOOO | 3-54|   OOOOOOOO | 3-64|   OOOOOOOO | 3-74|   OOOOO...
 | 3-15|   OOOOOOOO | 3-25|   ........ | 3-35|   OOOOOOOO | 3-45|   OOOOOOO. | 3-55|   OOOOO... | 3-65|   O....... | 3-75|   OOOO....
 | 3-16|   OOOOOOOO | 3-26|   OOOOOOOO | 3-36|   OOOOOOOO | 3-46|   OOOOOOOO | 3-56|   OOOOO... | 3-66|   OOOO.... | 3-76|   OOOO....
 | 3-17|   OOOOOOOO | 3-27|   OOOOOOOO | 3-37|   OOOOOOOO | 3-47|   OOOOOOOO | 3-57|   OOOO.... | 3-67|   OOOOO... | 3-77|   OOOOOOOO
 | 3-18|   OOOOOOOO | 3-28|   OOOOOOOO | 3-38|   OOOOOOOO | 3-48|   OOOOOOO. | 3-58|   OOOOOOOO | 3-68|   OOOO.... | 3-78|   ........


 | 4-11|   OOOOOOOO | 4-21|   OOOOOOOO | 4-31|   OOOOO... | 4-41|   OOOOOOOO | 4-51|   OOOOOOOO | 4-61|   OOOOOOOO | 4-71|   OOOOOOOO | 4-81|   OOOOOOOO
 | 4-12|   OOOOOOOO | 4-22|   OOOOOOOO | 4-32|   OOOOOO.. | 4-42|   OOOOOOOO | 4-52|   OOOOOOOO | 4-62|   OOOOOOOO | 4-72|   OOOOOOO. | 4-82|   ........
 | 4-13|   OOOOOOOO | 4-23|   OOOOOOOO | 4-33|   OOOOOOO. | 4-43|   OOOOOOOO | 4-53|   OOOOOOOO | 4-63|   OOOOOO.. | 4-73|   OOOOOOOO | 4-83|   OO......
 | 4-14|   OOOOOOOO | 4-24|   OOOOOOOO | 4-34|   OOOOOOOO | 4-44|     DOWN   | 4-54|   OOOOOOOO | 4-64|   OOOOOOOO | 4-74|   OOOOOOO.                   


 | 5- 1|   OOOOO... | 5-11|   OOOOOOOO | 5-21|   OOOOOOO.
 | 5- 2|   OOOOOOO. | 5-12|     DOWN   | 5-22|   OOOOOOO.
 | 5- 3|   OOOOOOOO | 5-13|   OOOOOOO. | 5-23|   OOOOOOO.
 | 5- 4|   OOOOO... | 5-14|   OOOOOOO. | 5-24|   OOOOOOO.
 | 5- 5|   OOOOOOOO | 5-15|   OOOOOOOO | 5-25|     DOWN  
 | 5- 6|   OOOOOOO. | 5-16|   OOOOO... | 5-26|   OOOOOOO.
 | 5- 7|   OOOOOOO. | 5-17|   OOOOOO.. | 5-27|   OOOOO...
 | 5- 8|   OOOOOOO. | 5-18|   OOOOOOO. | 5-28|   OOOOOO..
 | 5- 9|   OOOOOOO. | 5-19|   OOOOOO.. | 5-29|   OOOOOOO.
 | 5-10|   OOOOOOO. | 5-20|     DOWN   | 5-30|   OOOOO...


 | 9- 1|   OOOOOOOO
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   OOOOO...
 |10- 2|   OOOOO...


 |91- 1|   OOOOOOOO
 |91- 2|   OOOOOOOO
 |91- 3|     DOWN  
 |91- 4|   OOOOOOOO
 |91- 5|   OOOOOOOO
 |91- 6|   OOOOOOOO


Totals:
Total cores: 3736 
Total memory: 36.38 TiB 

Down:
Down cores: 168 (  4.50%) 
Down memory: 1.62 TiB (  4.47%) 

Used:
Used cores: 2755 ( 73.74%) 
Used memory: 25.42 TiB ( 69.87%) 

Utilization excluding down
CPU utilization:  77.21% (global)  78.46% (average per node)
Memory utilization:  73.14% (global)  75.21% (average per node)

Average per node utilization (excluding down):  83.55% 
[2020-06-05 17:24:03]
s03n13 bw test 
s04n44 df on /faststorage hangs 
s05n12 testing 
s05n20 Node unexpectedly rebooted 
s05n25 Seems dead ; talks via ipmi but reports power off and doesn't turn on 
s91n03 Kill task failed 

[2020-06-05 17:20:15]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-06-05 17:15:05]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       61891.0GB       18117.1GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3002       normal       80008.1GB       61873.2GB       18134.9GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3003       normal       80008.1GB       61887.6GB       18120.5GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3004       normal       80008.1GB       61735.7GB       18272.4GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3005       normal       80008.1GB       60438.7GB       19569.4GB    76% [###############     ]      1490.3M      1487.8M  100%
      9501       3006       normal       80008.1GB       61764.3GB       18243.8GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3007       normal       80008.1GB       62917.6GB       17090.4GB    79% [################    ]      1490.3M      1487.9M  100%
      9501       3008       normal       80008.1GB       61321.6GB       18686.5GB    77% [###############     ]      1490.3M      1487.8M  100%
      9501       3009       normal       80008.1GB       61676.7GB       18331.4GB    77% [###############     ]      1490.3M      1487.7M  100%
      9501       3010       normal       80008.1GB       60789.5GB       19218.6GB    76% [###############     ]      1490.3M      1487.7M  100%
      9501       3011       normal       80008.1GB       61208.6GB       18799.5GB    77% [###############     ]      1490.3M      1487.7M  100%
      9501       3012       normal       80008.1GB       60686.4GB       19321.7GB    76% [###############     ]      1490.3M      1487.7M  100%
      9501       3013       normal       80008.1GB       61143.8GB       18864.3GB    76% [###############     ]      1490.3M      1487.7M  100%
      9502       3014       normal       80008.1GB       59177.5GB       20830.6GB    74% [###############     ]      1490.3M      1487.7M  100%
      9502       3015       normal       80008.1GB       60423.9GB       19584.2GB    76% [###############     ]      1490.3M      1487.7M  100%
      9502       3016       normal       80008.1GB       69220.9GB       10787.2GB    87% [#################   ]      1490.3M      1489.6M  100%
      9502       3017       normal       80008.1GB       61014.4GB       18993.7GB    76% [###############     ]      1490.3M      1487.7M  100%
      9502       3018       normal       80008.1GB       60366.5GB       19641.5GB    75% [###############     ]      1490.3M      1487.7M  100%
      9502       3019       normal       80008.1GB       60349.8GB       19658.3GB    75% [###############     ]      1490.3M      1487.7M  100%
      9502       3020       normal       80008.1GB       61213.9GB       18794.1GB    77% [###############     ]      1490.3M      1487.7M  100%
      9601       3021       normal       80008.1GB       61219.9GB       18788.2GB    77% [###############     ]      1490.3M      1487.7M  100%
      9601       3022       normal       80008.1GB       61099.3GB       18908.8GB    76% [###############     ]      1490.3M      1487.7M  100%
      9601       3023       normal       80008.1GB       60351.0GB       19657.1GB    75% [###############     ]      1490.3M      1487.8M  100%
      9601       3024       normal       80008.1GB       62063.3GB       17944.8GB    78% [################    ]      1490.3M      1487.8M  100%
      9601       3025       normal       80008.1GB       60606.6GB       19401.5GB    76% [###############     ]      1490.3M      1487.8M  100%
      9601       3026       normal       80008.1GB       61122.0GB       18886.0GB    76% [###############     ]      1490.3M      1487.8M  100%
      9601       3027       normal       80008.1GB       62407.5GB       17600.5GB    78% [################    ]      1490.3M      1487.8M  100%
      9601       3028       normal       80008.1GB       60009.1GB       19998.9GB    75% [###############     ]      1490.3M      1487.8M  100%
      9601       3029       normal       80008.1GB       60994.5GB       19013.5GB    76% [###############     ]      1490.3M      1487.8M  100%
      9601       3030       normal       80008.1GB       62015.8GB       17992.3GB    78% [################    ]      1490.3M      1487.8M  100%
      9601       3031       normal       80008.1GB       61128.0GB       18880.1GB    76% [###############     ]      1490.3M      1487.8M  100%
      9601       3032       normal       80008.1GB       61046.1GB       18961.9GB    76% [###############     ]      1490.3M      1487.8M  100%
      9601       3033       normal       80008.1GB       62760.5GB       17247.6GB    78% [################    ]      1490.3M      1487.8M  100%
      9601       3034       normal       80008.1GB       59889.7GB       20118.4GB    75% [###############     ]      1490.3M      1487.8M  100%
      9602       3035       normal       80008.1GB       60411.4GB       19596.7GB    76% [###############     ]      1490.3M      1487.8M  100%
      9602       3036       normal       80008.1GB       61200.6GB       18807.5GB    76% [###############     ]      1490.3M      1487.8M  100%
      9602       3037       normal       80008.1GB       60853.6GB       19154.5GB    76% [###############     ]      1490.3M      1487.8M  100%
      9602       3038       normal       80008.1GB       62211.2GB       17796.9GB    78% [################    ]      1490.3M      1488.0M  100%
      9602       3039       normal       80008.1GB       61047.1GB       18961.0GB    76% [###############     ]      1490.3M      1487.8M  100%
      9602       3040       normal       80008.1GB       60499.3GB       19508.7GB    76% [###############     ]      1490.3M      1487.9M  100%
      9602       3041       normal       80008.1GB       61376.2GB       18631.9GB    77% [###############     ]      1490.3M      1487.8M  100%
      9701       3042       normal      119987.9GB       68327.8GB       51660.1GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3043       normal      119987.9GB       68894.9GB       51093.0GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3044       normal      119987.9GB       68758.4GB       51229.4GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3045       normal      119987.9GB       68703.0GB       51284.9GB    57% [###########         ]      2235.0M      2225.7M  100%
      9701       3046       normal      119987.9GB       68404.6GB       51583.2GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3047       normal      119987.9GB       68943.4GB       51044.4GB    57% [###########         ]      2235.0M      2225.7M  100%
      9701       3048       normal      119987.9GB       68762.7GB       51225.1GB    57% [###########         ]      2235.0M      2225.7M  100%
      9701       3049       normal      119987.9GB       68630.7GB       51357.1GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3050       normal      119987.9GB       68278.1GB       51709.7GB    57% [###########         ]      2235.0M      2225.7M  100%
      9701       3051       normal      119987.9GB       68688.5GB       51299.4GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3052       normal      119987.9GB       68850.6GB       51137.2GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3053       normal      119987.9GB       68707.2GB       51280.7GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3054       normal      119987.9GB       68389.1GB       51598.8GB    57% [###########         ]      2235.0M      2225.8M  100%
      9701       3055       normal      119987.9GB       68067.4GB       51920.4GB    57% [###########         ]      2235.0M      2225.7M  100%
      9702       3056       normal      119987.9GB       68512.6GB       51475.3GB    57% [###########         ]      2235.0M      2225.7M  100%
      9702       3057       normal      119987.9GB       68703.4GB       51284.5GB    57% [###########         ]      2235.0M      2225.7M  100%
      9702       3058       normal      119987.9GB       68628.3GB       51359.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3059       normal      119987.9GB       69596.8GB       50391.1GB    58% [############        ]      2235.0M      2225.7M  100%
      9702       3060       normal      119987.9GB       68726.4GB       51261.4GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3061       normal      119987.9GB       68869.0GB       51118.9GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3062       normal      119987.9GB       68382.8GB       51605.1GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3063       normal      119987.9GB       68983.6GB       51004.3GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3064       normal      119987.9GB       68273.4GB       51714.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3065       normal      119987.9GB       68376.8GB       51611.1GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3066       normal      119987.9GB       69220.9GB       50767.0GB    58% [############        ]      2235.0M      2225.8M  100%
      9702       3067       normal      119987.9GB       68459.3GB       51528.6GB    57% [###########         ]      2235.0M      2225.8M  100%
      9702       3068       normal      119987.9GB       68292.6GB       51695.3GB    57% [###########         ]      2235.0M      2225.7M  100%
      9702       3069       normal      119987.9GB       68820.1GB       51167.8GB    57% [###########         ]      2235.0M      2225.7M  100%
      9801       3070       normal      119987.9GB       68124.0GB       51863.8GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3071       normal      119987.9GB       68920.2GB       51067.7GB    57% [###########         ]      2235.0M      2228.7M  100%
      9801       3072       normal      119987.9GB       69166.7GB       50821.1GB    58% [############        ]      2235.0M      2225.8M  100%
      9801       3073       normal      119987.9GB       69042.3GB       50945.6GB    58% [############        ]      2235.0M      2225.7M  100%
      9801       3074       normal      119987.9GB       68523.4GB       51464.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3075       normal      119987.9GB       68139.2GB       51848.6GB    57% [###########         ]      2235.0M      2225.7M  100%
      9801       3076       normal      119987.9GB       68874.4GB       51113.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3077       normal      119987.9GB       68796.8GB       51191.1GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3078       normal      119987.9GB       68874.9GB       51112.9GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3079       normal      119987.9GB       68840.1GB       51147.8GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3080       normal      119987.9GB       68152.6GB       51835.2GB    57% [###########         ]      2235.0M      2228.6M  100%
      9801       3081       normal      119987.9GB       68515.8GB       51472.1GB    57% [###########         ]      2235.0M      2225.7M  100%
      9801       3082       normal      119987.9GB       68066.9GB       51921.0GB    57% [###########         ]      2235.0M      2225.8M  100%
      9801       3083       normal      119987.9GB       68447.0GB       51540.9GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3084       normal      119987.9GB       68152.4GB       51835.5GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3085       normal      119987.9GB       68162.5GB       51825.3GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3086       normal      119987.9GB       68687.4GB       51300.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3087       normal      119987.9GB       68803.3GB       51184.5GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3088       normal      119987.9GB       68688.9GB       51299.0GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3089       normal      119987.9GB       68644.4GB       51343.5GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3090       normal      119987.9GB       68835.8GB       51152.1GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3091       normal      119987.9GB       68458.7GB       51529.2GB    57% [###########         ]      2235.0M      2225.7M  100%
      9802       3092       normal      119987.9GB       68999.6GB       50988.3GB    58% [############        ]      2235.0M      2225.8M  100%
      9802       3093       normal      119987.9GB       68593.8GB       51394.0GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3094       normal      119987.9GB       68357.6GB       51630.2GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3095       normal      119987.9GB       68477.6GB       51510.3GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3096       normal      119987.9GB       68328.0GB       51659.9GB    57% [###########         ]      2235.0M      2225.8M  100%
      9802       3097       normal      119987.9GB       68756.6GB       51231.3GB    57% [###########         ]      2235.0M      2225.7M  100%
      9502       3098       normal      119993.9GB       60742.3GB       59251.6GB    51% [##########          ]      2235.1M      2232.6M  100%
      9502       3099       normal      119993.9GB       61099.5GB       58894.4GB    51% [##########          ]      2235.1M      2232.6M  100%
      9502       3100       normal      119993.9GB       61298.2GB       58695.7GB    51% [##########          ]      2235.1M      2232.6M  100%
      9502       3101       normal      119993.9GB       60824.4GB       59169.5GB    51% [##########          ]      2235.1M      2232.6M  100%
      9502       3102       normal      119993.9GB       60406.1GB       59587.8GB    50% [##########          ]      2235.1M      2232.6M  100%
      9502       3103       normal      119993.9GB       61004.0GB       58989.9GB    51% [##########          ]      2235.1M      2232.6M  100%
      9502       3104       normal      119993.9GB       61307.4GB       58686.5GB    51% [##########          ]      2235.1M      2232.6M  100%
      9602       3105       normal      119993.9GB       59355.4GB       60638.5GB    49% [##########          ]      2235.1M      2232.6M  100%
      9602       3106       normal      119993.9GB       60255.6GB       59738.3GB    50% [##########          ]      2235.1M      2232.6M  100%
      9602       3107       normal      119993.9GB       61800.2GB       58193.6GB    52% [##########          ]      2235.1M      2232.6M  100%
      9602       3108       normal      119993.9GB       61195.7GB       58798.2GB    51% [##########          ]      2235.1M      2232.6M  100%
      9602       3109       normal      119993.9GB       61214.8GB       58779.1GB    51% [##########          ]      2235.1M      2232.6M  100%
      9602       3110       normal      119993.9GB       61298.3GB       58695.6GB    51% [##########          ]      2235.1M      2232.6M  100%
      9602       3111       normal      119993.9GB       61166.3GB       58827.6GB    51% [##########          ]      2235.1M      2232.6M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7211065.3 GB
Total free space on /faststorage        4468500.2 GB

                                 Use%   61.74%

2020-06-05 17:25:08.417009853 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-06-05 14:40:02] s03n33 resumed
[2020-06-05 14:38:02] s03n33 now marked for drain: sanity-check: can't ping gfs6 
[2020-06-05 07:20:02] s91n02 resumed
[2020-06-05 07:18:02] s91n02 now marked for drain: sanity-check: can't ping gfs6 
[2020-06-04 21:02:02] s05n30 resumed
[2020-06-04 20:48:02] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-04 14:02:02] s05n18 resumed
[2020-06-04 10:50:02] s05n18 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-06-03 23:02:02] s05n30 resumed
[2020-06-03 22:10:02] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-03 22:08:02] s05n30 resumed
[2020-06-03 22:06:03] s05n30 now marked for drain: sanity-check: can't ping gfs5 
[2020-06-03 22:02:02] s05n30 resumed
[2020-06-03 21:24:01] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-03 17:02:02] s05n30 resumed
[2020-06-03 16:40:02] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-03 16:18:02] s05n22 resumed
[2020-06-03 16:18:02] s05n19 resumed
[2020-06-03 16:02:02] s05n30 resumed
[2020-06-03 15:50:02] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-03 15:18:02] s05n18 resumed
[2020-06-03 13:44:02] s91n03 now marked for drain: Kill task failed 
[2020-06-03 10:28:02] s05n18 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-06-03 10:02:01] s91n03 resumed
[2020-06-03 10:02:01] s05n30 resumed
[2020-06-03 09:36:02] s05n22 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-06-03 09:36:02] s05n19 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-06-03 09:18:02] s05n30 now marked for drain: Sanity check timer not running 
[2020-06-03 09:12:02] s91n03 now marked for drain: Sanity check timer not running 
[2020-06-03 09:00:02] s91n06 resumed
[2020-06-03 09:00:02] s91n05 resumed
[2020-06-03 09:00:02] s91n03 resumed
[2020-06-03 09:00:02] s91n01 resumed
[2020-06-03 09:00:02] s10n02 resumed
[2020-06-03 09:00:02] s09n02 resumed
[2020-06-03 09:00:02] s05n26 resumed
[2020-06-03 09:00:02] s05n24 resumed
[2020-06-03 09:00:02] s05n23 resumed
[2020-06-03 09:00:02] s05n22 resumed
[2020-06-03 09:00:02] s05n19 resumed
[2020-06-03 09:00:02] s05n15 resumed
[2020-06-03 09:00:02] s05n13 resumed
[2020-06-03 09:00:02] s05n10 resumed
[2020-06-03 09:00:02] s05n08 resumed
[2020-06-03 09:00:02] s05n03 resumed
[2020-06-03 09:00:02] s05n01 resumed
[2020-06-03 09:00:02] s04n83 resumed
[2020-06-03 09:00:02] s04n72 resumed
[2020-06-03 09:00:02] s04n64 resumed
[2020-06-03 09:00:02] s04n61 resumed
[2020-06-03 09:00:02] s04n54 resumed
[2020-06-05 17:00:02]
Backup on    fs1  finished 0-14:26:03 ago
Backup on    fs2 initiated 1-23:15:01 ago
Backup on    fs3  finished 0-09:12:23 ago
Backup on    fs5  finished 0-06:07:03 ago
Backup on    fs6  finished 1-11:34:01 ago
Backup on s95n01  finished 1-17:48:05 ago
[2020-06-05 17:24:03]
################
# RUNNING JOBS #
################
Count  USER       MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----       --------  ----------  ---------    ---
1      andros     8         32G         9:55:00      qos1
1      anneaa     1         16G         4:00:00      bumped
1      besen      1         16G         2-00:00:00   qos1
1      besen      16        64G         2-00:00:00   qos1
1      cks        10        200G        3-00:00:00   bumped
1      cks        16        50G         1-12:00:00   basic
1      clara      24        256G        3-00:00:00   qos1
1      elivee     20        10G         10-00:00:00  basic
1      gabrielr   3         15G         11:00:00     qos1
1      iyorgos    1         16G         4:00:00      bumped
1      janneae    1         8G          8:00:00      basic
1      julienlin  2         50G         8:00:00      qos1
1      kleppe     1         4G          5:00:00      bumped
1      mtv        15        32G         2-00:00:00   bumped
1      mtxellrb   1         100G        4:00:00      bumped
1      mtxellrb   4         200G        23:00:00     bumped
1      noor       1         25G         10:00:00     basic
1      owyn       24        1T          83-12:00:00  basic
1      palle      24        120G        2-08:00:00   bumped
1      zexi       1         10G         30-12:00:00  basic
1      zexi       8         40G         30-12:00:00  basic
2      besen      16        8G          2-00:00:00   qos1
3      doug       6         25G         12:00:00     bumped
3      juraj      2         128G        1-00:00:00   bumped
4      doug       6         25G         15:00:00     bumped
4      janneae    1         40G         14-00:00:00  basic
4      noor       1         140G        2-00:00:00   basic
5      michaelk   1         4G          12:00:00     qos1
6      juraj      2         128G        1-00:00:00   basic
6      mtxellrb   4         250G        12:00:00     qos1
10     plamy      16        128G        4-00:00:00   basic
11     gabrielr   10        15G         23:00:00     qos1
13     zexi       8         20G         30-12:00:00  basic
23     mica16     1         6G          2:00:00      bumped
31     lucie      24        256G        1-00:00:00   qos1
119    mica16     1         6G          2:00:00      basic
148    fprive     8         64G         12:00:00     qos1
###############
# QUEUED JOBS #
###############
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT   QOS
-----  ----      --------  ----------  ---------   ---
1      kmt       1         8G          8:00:00     basic
1      noor      1         10G         7:00:00     basic
1      noor      1         140G        2-00:00:00  basic
12     lucie     24        256G        1-00:00:00  qos1
31     mtxellrb  4         250G        12:00:00    qos1
175    juraj     2         128G        1-00:00:00  basic
468    gabrielr  10        15G         23:00:00    qos1
1254   fprive    8         64G         12:00:00    qos1
 
#################
# CPUs per User #
#################
NCPUs  USER       QOS
-----  ----       ---
1      anneaa     bumped
1      iyorgos    bumped
1      kleppe     bumped
2      julienlin  qos1
5      janneae    basic
5      michaelk   qos1
5      mtxellrb   bumped
5      noor       basic
6      juraj      bumped
8      andros     qos1
10     cks        bumped
12     juraj      basic
15     mtv        bumped
16     cks        basic
20     elivee     basic
23     mica16     bumped
24     clara      qos1
24     mtxellrb   qos1
24     owyn       basic
24     palle      bumped
42     doug       bumped
49     besen      qos1
113    gabrielr   qos1
113    zexi       basic
119    mica16     basic
160    plamy      basic
744    lucie      qos1
1184   fprive     qos1
########################
# QUEUED CPUs per User #
########################
NCPUs  USER      QOS
-----  ----      ---
1      kmt       basic
2      noor      basic
124    mtxellrb  qos1
288    lucie     qos1
350    juraj     basic
4680   gabrielr  qos1
10032  fprive    qos1