[2020-07-16 18:24:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-07-16 18:24:02]

 | 3-11|   ........ | 3-21|   OOOOOOOO | 3-31|   OOOOOOOO | 3-41|   OOOO.... | 3-51|   OOOOOOO. | 3-61|   OOOOOOO. | 3-71|   OOOOOOO.
 | 3-12|   ........ | 3-22|   OOOOOOOO | 3-32|   OOOOOOOO | 3-42|   ........ | 3-52|   OOOOOOO. | 3-62|   OOOOOOO. | 3-72|     DOWN  
 | 3-13|   ........ | 3-23|   OOOOOOOO | 3-33|   OOOOOOOO | 3-43|   O....... | 3-53|   OOOOOOOO | 3-63|   OOOOOOO. | 3-73|   OOO.....
 | 3-14|   O....... | 3-24|   OOOOOOOO | 3-34|   OOOO.... | 3-44|   OOOOOOO. | 3-54|   OOOOOOO. | 3-64|   OOOOOOO. | 3-74|   OOOOOOO.
 | 3-15|   ........ | 3-25|     DOWN   | 3-35|   OOOOOOOO | 3-45|   OOOOOOO. | 3-55|   OOOOOOO. | 3-65|   OOOOOOOO | 3-75|   OOOOOOO.
 | 3-16|   OOOO.... | 3-26|   OOOOOOOO | 3-36|   OOOOOOOO | 3-46|   OOOOOOO. | 3-56|   OOOOOOO. | 3-66|     DOWN   | 3-76|   OOOOOOO.
 | 3-17|   OOOOOO.. | 3-27|   ........ | 3-37|   OOOOOOOO | 3-47|   OOOOOOO. | 3-57|   OOOOOOO. | 3-67|     DOWN   | 3-77|   OOOOOOO.
 | 3-18|   OOOOOOOO | 3-28|   ........ | 3-38|   ........ | 3-48|     DOWN   | 3-58|   OOOOOOOO | 3-68|   OOOOOOO. | 3-78|   ........


 | 4-11|   OOOOOO.. | 4-21|   OOOOOOO. | 4-31|   OOOOOOO. | 4-41|   OOOOOOO. | 4-51|   OOOOOOO. | 4-61|   OOOOOOOO | 4-71|   OOOOOOO. | 4-81|   ........
 | 4-12|   OOOOOOO. | 4-22|   OOOOOOO. | 4-32|   OOOOOOO. | 4-42|   OOOOOOO. | 4-52|   OOOOOOOO | 4-62|   OOOOOOO. | 4-72|   OOOOOOO. | 4-82|   ........
 | 4-13|   OOOOOOO. | 4-23|   OOOOOOO. | 4-33|   OOOOOOO. | 4-43|   OOOOOOO. | 4-53|   OOOOOOO. | 4-63|   OOOOOOOO | 4-73|   OOOOOOO. | 4-83|   ........
 | 4-14|   OOOOOOO. | 4-24|   OOOOOOO. | 4-34|   OOOOOOO. | 4-44|   OOOOOOO. | 4-54|   OOOOOOO. | 4-64|   OOOOOOO. | 4-74|   OOOOOOO.                   


 | 5- 1|   OOOOOOO. | 5-11|   OOOOOO.. | 5-21|   OOOOOOO.
 | 5- 2|   OOOOOOO. | 5-12|   OOOOOOOO | 5-22|   OOOOOOOO
 | 5- 3|   OOOOOOOO | 5-13|   OOOOOO.. | 5-23|   OOOOOO..
 | 5- 4|   OOOOOOO. | 5-14|   OOOOOOO. | 5-24|   OOOOO...
 | 5- 5|   OOOOOOO. | 5-15|   OOOOOOO. | 5-25|     DOWN  
 | 5- 6|   OOOOOOO. | 5-16|   OOOOOOOO | 5-26|     DOWN  
 | 5- 7|   OOOOOOO. | 5-17|   OOOOOOOO | 5-27|   OOOOOOOO
 | 5- 8|   OOOOOOOO | 5-18|   OOOOOOOO | 5-28|   OOOOOOOO
 | 5- 9|   OOOOOOO. | 5-19|   OOOOOOOO | 5-29|   OOOOOOOO
 | 5-10|   OOOOOOO. | 5-20|     DOWN   | 5-30|   OOOOOOO.


 | 9- 1|   OOOOOOOO
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   ........
 |10- 2|   ........


 |21-11|   ........ |21-21|   ........ |21-31|   ........ |21-41|   ........ |21-51|   ........ |21-61|   OOOOO... |21-71|   OOOOOO.. |21-81|   ........ |21-91|   ........
 |21-12|   ........ |21-22|   ........ |21-32|   ........ |21-42|   ........ |21-52|     DOWN   |21-62|   OOOO.... |21-72|   OO...... |21-82|   ........ |21-92|   ........
 |21-13|   ........ |21-23|   ........ |21-33|   ........ |21-43|   ........ |21-53|   OO...... |21-63|   OOOO.... |21-73|   ........ |21-83|   ........ |21-93|   ........
 |21-14|   ........ |21-24|   ........ |21-34|   ........ |21-44|   ........ |21-54|   OOOOOO.. |21-64|   OOO..... |21-74|   ........ |21-84|   ........ |21-94|   ........


 |22-11|   OOOOOOO. |22-21|   OOOOOOOO |22-31|   OOOOOOO. |22-41|   OOOOOOO. |22-51|   OOOOOO..
 |22-12|   OOOOOOOO |22-22|   OOOOOOOO |22-32|   OOOOOOO. |22-42|   OOOOOO.. |22-52|   OOOOOOO.
 |22-13|   OOOOOOO. |22-23|   OOOOOOOO |22-33|   OOOOOOO. |22-43|   OOOOOOO. |22-53|   OOOOOOO.
 |22-14|   OOOOOOOO |22-24|   OOOOOOOO |22-34|   OOOOOOO. |22-44|   OOOOOOO. |22-54|   OOOOOOO.


 |91- 1|   OOOOOO..
 |91- 2|   OOOOOO..
 |91- 3|   OOOOOO..
 |91- 4|   OOOO....
 |91- 5|   OOO.....
 |91- 6|   OOO.....


Totals:
Total cores: 6568 
Total memory: 58.03 TiB 

Down:
Down cores: 252 (  3.84%) 
Down memory: 2.24 TiB (  3.86%) 

Used:
Used cores: 2431 ( 37.01%) 
Used memory: 32.40 TiB ( 55.83%) 

Utilization excluding down
CPU utilization:  38.49% (global)  46.54% (average per node)
Memory utilization:  58.07% (global)  62.75% (average per node)

Average per node utilization (excluding down):  67.53% 
[2020-07-16 18:24:02]
s03n25 Node unexpectedly rebooted 
s03n48 Not responding 
s03n66 Not responding 
s03n67 sanity-check: s03n67 30-Jun-2020 11:01 /faststorage not available 
s03n72 Not responding 
s05n20 Node unexpectedly rebooted 
s05n25 Node unexpectedly rebooted 
s05n26 batch job complete failure 
s21n52 Not responding 

[2020-07-16 18:20:29]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-07-16 18:15:09]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       66210.3GB       13797.8GB    83% [#################   ]      1490.3M      1487.7M  100%
      9501       3002       normal       80008.1GB       66289.3GB       13718.8GB    83% [#################   ]      1490.3M      1487.7M  100%
      9501       3003       normal       80008.1GB       65754.0GB       14254.1GB    82% [################    ]      1490.3M      1487.7M  100%
      9501       3004       normal       80008.1GB       65608.8GB       14399.3GB    82% [################    ]      1490.3M      1487.7M  100%
      9501       3005       normal       80008.1GB       64677.6GB       15330.5GB    81% [################    ]      1490.3M      1487.7M  100%
      9501       3006       normal       80008.1GB       66209.1GB       13798.9GB    83% [#################   ]      1490.3M      1487.7M  100%
      9501       3007       normal       80008.1GB       66790.8GB       13217.3GB    83% [#################   ]      1490.3M      1487.8M  100%
      9501       3008       normal       80008.1GB       65229.1GB       14778.9GB    82% [################    ]      1490.3M      1487.6M  100%
      9501       3009       normal       80008.1GB       66153.3GB       13854.8GB    83% [#################   ]      1490.3M      1487.6M  100%
      9501       3010       normal       80008.1GB       64619.0GB       15389.0GB    81% [################    ]      1490.3M      1487.6M  100%
      9501       3011       normal       80008.1GB       62937.2GB       17070.9GB    79% [################    ]      1490.3M      1487.7M  100%
      9501       3012       normal       80008.1GB       64790.7GB       15217.4GB    81% [################    ]      1490.3M      1487.6M  100%
      9501       3013       normal       80008.1GB       65128.1GB       14880.0GB    81% [################    ]      1490.3M      1487.6M  100%
      9502       3014       normal       80008.1GB       62674.7GB       17333.4GB    78% [################    ]      1490.3M      1487.6M  100%
      9502       3015       normal       80008.1GB       63562.8GB       16445.3GB    79% [################    ]      1490.3M      1487.6M  100%
      9502       3016          low       80008.1GB       71216.7GB        8791.4GB    89% [##################  ]      1490.3M      1489.6M  100%
      9502       3017       normal       80008.1GB       65101.5GB       14906.5GB    81% [################    ]      1490.3M      1487.6M  100%
      9502       3018       normal       80008.1GB       65049.3GB       14958.7GB    81% [################    ]      1490.3M      1487.6M  100%
      9502       3019       normal       80008.1GB       64792.2GB       15215.9GB    81% [################    ]      1490.3M      1487.6M  100%
      9502       3020       normal       80008.1GB       65375.6GB       14632.5GB    82% [################    ]      1490.3M      1487.6M  100%
      9601       3021       normal       80008.1GB       65280.6GB       14727.5GB    82% [################    ]      1490.3M      1487.6M  100%
      9601       3022       normal       80008.1GB       64649.8GB       15358.3GB    81% [################    ]      1490.3M      1487.6M  100%
      9601       3023       normal       80008.1GB       64740.9GB       15267.2GB    81% [################    ]      1490.3M      1487.7M  100%
      9601       3024       normal       80008.1GB       66599.9GB       13408.2GB    83% [#################   ]      1490.3M      1487.6M  100%
      9601       3025       normal       80008.1GB       64570.1GB       15438.0GB    81% [################    ]      1490.3M      1487.6M  100%
      9601       3026       normal       80008.1GB       64891.3GB       15116.8GB    81% [################    ]      1490.3M      1487.6M  100%
      9601       3027       normal       80008.1GB       66739.5GB       13268.5GB    83% [#################   ]      1490.3M      1487.7M  100%
      9601       3028       normal       80008.1GB       64523.3GB       15484.8GB    81% [################    ]      1490.3M      1487.7M  100%
      9601       3029       normal       80008.1GB       65239.2GB       14768.9GB    82% [################    ]      1490.3M      1487.7M  100%
      9601       3030       normal       80008.1GB       66010.1GB       13998.0GB    83% [#################   ]      1490.3M      1487.7M  100%
      9601       3031       normal       80008.1GB       65359.7GB       14648.4GB    82% [################    ]      1490.3M      1487.7M  100%
      9601       3032       normal       80008.1GB       65077.5GB       14930.6GB    81% [################    ]      1490.3M      1487.7M  100%
      9601       3033       normal       80008.1GB       66675.5GB       13332.5GB    83% [#################   ]      1490.3M      1487.7M  100%
      9601       3034       normal       80008.1GB       64275.1GB       15733.0GB    80% [################    ]      1490.3M      1487.7M  100%
      9602       3035       normal       80008.1GB       64466.7GB       15541.3GB    81% [################    ]      1490.3M      1487.7M  100%
      9602       3036       normal       80008.1GB       65162.2GB       14845.9GB    81% [################    ]      1490.3M      1487.7M  100%
      9602       3037       normal       80008.1GB       65344.0GB       14664.1GB    82% [################    ]      1490.3M      1487.7M  100%
      9602       3038       normal       80008.1GB       66867.0GB       13141.1GB    84% [#################   ]      1490.3M      1487.8M  100%
      9602       3039       normal       80008.1GB       65544.3GB       14463.7GB    82% [################    ]      1490.3M      1487.7M  100%
      9602       3040       normal       80008.1GB       65461.8GB       14546.2GB    82% [################    ]      1490.3M      1487.7M  100%
      9602       3041       normal       80008.1GB       65382.2GB       14625.9GB    82% [################    ]      1490.3M      1487.7M  100%
      9701       3042       normal      119987.9GB       71586.3GB       48401.6GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3043       normal      119987.9GB       72126.0GB       47861.9GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3044       normal      119987.9GB       72937.0GB       47050.9GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3045       normal      119987.9GB       73072.4GB       46915.5GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3046       normal      119987.9GB       71888.0GB       48099.8GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3047       normal      119987.9GB       72836.7GB       47151.2GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3048       normal      119987.9GB       72151.0GB       47836.8GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3049       normal      119987.9GB       73223.4GB       46764.5GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3050       normal      119987.9GB       71874.7GB       48113.2GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3051       normal      119987.9GB       72831.7GB       47156.1GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3052       normal      119987.9GB       72466.8GB       47521.1GB    60% [############        ]      2235.0M      2225.6M  100%
      9701       3053       normal      119987.9GB       73149.8GB       46838.0GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3054       normal      119987.9GB       73420.4GB       46567.4GB    61% [############        ]      2235.0M      2225.6M  100%
      9701       3055       normal      119987.9GB       72563.2GB       47424.7GB    60% [############        ]      2235.0M      2225.6M  100%
      9702       3056       normal      119987.9GB       72497.9GB       47490.0GB    60% [############        ]      2235.0M      2225.6M  100%
      9702       3057       normal      119987.9GB       72799.6GB       47188.2GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3058       normal      119987.9GB       72283.3GB       47704.5GB    60% [############        ]      2235.0M      2225.6M  100%
      9702       3059       normal      119987.9GB       73168.6GB       46819.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3060       normal      119987.9GB       72834.8GB       47153.1GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3061       normal      119987.9GB       73020.2GB       46967.6GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3062       normal      119987.9GB       72366.7GB       47621.2GB    60% [############        ]      2235.0M      2225.6M  100%
      9702       3063       normal      119987.9GB       73729.8GB       46258.1GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3064       normal      119987.9GB       72906.6GB       47081.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3065       normal      119987.9GB       72423.4GB       47564.5GB    60% [############        ]      2235.0M      2225.6M  100%
      9702       3066       normal      119987.9GB       73290.2GB       46697.6GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3067       normal      119987.9GB       72604.1GB       47383.8GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3068       normal      119987.9GB       72644.9GB       47343.0GB    61% [############        ]      2235.0M      2225.6M  100%
      9702       3069       normal      119987.9GB       72729.6GB       47258.2GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3070       normal      119987.9GB       72264.5GB       47723.4GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3071       normal      119987.9GB       72983.2GB       47004.7GB    61% [############        ]      2235.0M      2228.5M  100%
      9801       3072       normal      119987.9GB       73177.3GB       46810.6GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3073       normal      119987.9GB       72462.6GB       47525.3GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3074       normal      119987.9GB       72948.6GB       47039.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3075       normal      119987.9GB       71759.7GB       48228.1GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3076       normal      119987.9GB       72258.9GB       47728.9GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3077       normal      119987.9GB       73256.7GB       46731.1GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3078       normal      119987.9GB       72867.0GB       47120.8GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3079       normal      119987.9GB       72536.6GB       47451.3GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3080       normal      119987.9GB       71964.6GB       48023.3GB    60% [############        ]      2235.0M      2228.4M  100%
      9801       3081       normal      119987.9GB       72257.0GB       47730.9GB    60% [############        ]      2235.0M      2225.6M  100%
      9801       3082       normal      119987.9GB       72797.2GB       47190.7GB    61% [############        ]      2235.0M      2225.6M  100%
      9801       3083       normal      119987.9GB       72385.1GB       47602.8GB    60% [############        ]      2235.0M      2225.6M  100%
      9802       3084       normal      119987.9GB       71864.7GB       48123.2GB    60% [############        ]      2235.0M      2225.6M  100%
      9802       3085       normal      119987.9GB       72109.1GB       47878.8GB    60% [############        ]      2235.0M      2225.6M  100%
      9802       3086       normal      119987.9GB       73234.6GB       46753.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3087       normal      119987.9GB       73314.3GB       46673.6GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3088       normal      119987.9GB       72629.6GB       47358.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3089       normal      119987.9GB       72485.3GB       47502.6GB    60% [############        ]      2235.0M      2225.6M  100%
      9802       3090       normal      119987.9GB       73599.4GB       46388.4GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3091       normal      119987.9GB       72860.9GB       47127.0GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3092       normal      119987.9GB       72927.6GB       47060.2GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3093       normal      119987.9GB       72944.5GB       47043.4GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3094       normal      119987.9GB       73481.5GB       46506.3GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3095       normal      119987.9GB       73254.4GB       46733.5GB    61% [############        ]      2235.0M      2225.6M  100%
      9802       3096       normal      119987.9GB       72135.0GB       47852.9GB    60% [############        ]      2235.0M      2225.6M  100%
      9802       3097       normal      119987.9GB       73162.6GB       46825.2GB    61% [############        ]      2235.0M      2225.6M  100%
      9502       3098       normal      119993.9GB       64547.2GB       55446.7GB    54% [###########         ]      2235.1M      2232.5M  100%
      9502       3099       normal      119993.9GB       65641.6GB       54352.2GB    55% [###########         ]      2235.1M      2232.5M  100%
      9502       3100       normal      119993.9GB       64995.6GB       54998.2GB    54% [###########         ]      2235.1M      2232.5M  100%
      9502       3101       normal      119993.9GB       64444.1GB       55549.8GB    54% [###########         ]      2235.1M      2232.5M  100%
      9502       3102       normal      119993.9GB       64858.7GB       55135.1GB    54% [###########         ]      2235.1M      2232.5M  100%
      9502       3103       normal      119993.9GB       64792.6GB       55201.2GB    54% [###########         ]      2235.1M      2232.5M  100%
      9502       3104       normal      119993.9GB       65327.1GB       54666.8GB    54% [###########         ]      2235.1M      2232.5M  100%
      9602       3105       normal      119993.9GB       63454.4GB       56539.5GB    53% [###########         ]      2235.1M      2232.5M  100%
      9602       3106       normal      119993.9GB       64413.9GB       55580.0GB    54% [###########         ]      2235.1M      2232.5M  100%
      9602       3107       normal      119993.9GB       65836.1GB       54157.7GB    55% [###########         ]      2235.1M      2232.5M  100%
      9602       3108       normal      119993.9GB       66705.4GB       53288.5GB    56% [###########         ]      2235.1M      2232.5M  100%
      9602       3109       normal      119993.9GB       65044.8GB       54949.1GB    54% [###########         ]      2235.1M      2232.5M  100%
      9602       3110       normal      119993.9GB       65517.9GB       54476.0GB    55% [###########         ]      2235.1M      2232.5M  100%
      9602       3111       normal      119993.9GB       65457.1GB       54536.7GB    55% [###########         ]      2235.1M      2232.5M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7663416.4 GB
Total free space on /faststorage        4016149.1 GB

                                 Use%   65.61%

2020-07-16 18:25:08.238074632 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-07-16 17:30:02] s21n41 resumed
[2020-07-16 17:28:02] s21n41 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 16:52:02] s21n41 resumed
[2020-07-16 16:50:01] s21n41 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 13:18:01] s21n14 resumed
[2020-07-16 13:16:02] s21n14 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 13:02:01] s21n53 resumed
[2020-07-16 13:00:02] s21n53 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 11:36:02] s22n51 resumed
[2020-07-16 11:34:01] s22n51 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 11:18:02] s21n22 resumed
[2020-07-16 11:16:02] s21n22 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 11:04:02] s21n22 resumed
[2020-07-16 11:00:02] s21n22 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 10:48:02] s22n13 resumed
[2020-07-16 10:46:02] s22n13 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 09:58:01] s22n12 resumed
[2020-07-16 09:56:02] s22n12 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 09:16:02] s21n92 resumed
[2020-07-16 09:14:01] s21n92 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 08:40:02] s22n21 resumed
[2020-07-16 08:38:02] s22n21 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 08:08:02] s22n14 resumed
[2020-07-16 08:06:02] s22n14 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 07:44:02] s05n27 resumed
[2020-07-16 07:42:02] s05n27 now marked for drain: health-check: /proc hangs 
[2020-07-16 07:10:02] s22n51 resumed
[2020-07-16 07:08:01] s22n51 now marked for drain: health-check: can't ping gfs5 
[2020-07-16 07:06:02] s03n75 resumed
[2020-07-16 07:04:02] s03n75 now marked for drain: health-check: can't ping gfs5 
[2020-07-16 06:30:02] s21n22 resumed
[2020-07-16 06:28:01] s21n22 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 06:10:02] s05n27 resumed
[2020-07-16 06:08:02] s05n27 now marked for drain: health-check: /proc hangs 
[2020-07-16 06:00:02] s21n91 resumed
[2020-07-16 05:58:02] s21n91 now marked for drain: health-check: /proc hangs 
[2020-07-16 05:54:02] s21n22 resumed
[2020-07-16 05:52:02] s21n22 now marked for drain: health-check: can't ping gfs6 
[2020-07-16 05:40:02] s05n24 resumed
[2020-07-16 05:38:02] s05n24 now marked for drain: health-check: /proc hangs 
[2020-07-16 05:18:02] s21n64 resumed
[2020-07-16 05:16:02] s21n64 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 05:16:02] s04n34 resumed
[2020-07-16 05:14:02] s04n34 now marked for drain: health-check: can't ping gfs3 
[2020-07-16 03:48:02] s05n17 resumed
[2020-07-16 03:46:02] s05n17 now marked for drain: health-check: /proc hangs 
[2020-07-16 03:24:02] s21n24 resumed
[2020-07-16 03:24:02] s21n23 resumed
[2020-07-16 03:22:02] s21n24 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 03:22:02] s21n23 now marked for drain: health-check: df /faststorage hangs 
[2020-07-16 03:16:02] s21n33 resumed
[2020-07-16 18:00:05]
Backup on    fs1  finished 0-15:31:12 ago
Backup on    fs2 initiated 26-00:15:03 ago
Backup on    fs3 initiated -18--7:-59:-57 ago
Backup on    fs5 initiated 0-20:00:03 ago
Backup on    fs6  finished 0-12:54:50 ago
Backup on s95n01  finished 0-18:36:52 ago
[2020-07-16 18:24:02]
################
# RUNNING JOBS #
################
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----      --------  ----------  ---------    ---
1      au342309  16        128G        12:00:00     basic
1      b960499   1         32G         8-00:00:00   basic
1      dreamlab  3         8G          1-00:00:00   basic
1      dreamlab  6         20G         5-00:00:00   basic
1      dreamlab  6         20G         5-00:00:00   bumped
1      emp       10        250G        12:00:00     qos1
1      iastar    1         1T          2-00:00:00   qos1
1      iastar    1         503G        3-18:00:00   qos1
1      ijjduan   64        500G        54-00:17:00  qos1
1      iprada    4         32G         6:00:00      qos1
1      jusren    1         20G         8-08:00:00   basic
1      leh       1         4G          5:00:00      basic
1      lucie     1         20G         22:00:00     qos1
1      lucie     3         50G         12:00:00     qos1
1      michaelk  1         16G         12:00:00     qos1
1      michaelk  1         4G          12:00:00     bumped
1      mtxellrb  4         100G        2-00:00:00   basic
1      qianqian  10        50G         7-22:00:00   qos1
1      simong    1         16G         1-16:00:00   bumped
1      vang      1         16G         2-00:00:00   gold
1      vang      16        64G         2-00:00:00   gold
1      vang      2         64G         2-00:00:00   gold
1      zexi      8         100G        10-12:00:00  basic
2      dario     1         50G         11:59:00     qos1
2      ijjduan   64        500G        INVALID      qos1
2      tbhmbi    1         24G         5-00:00:00   basic
2      vang      1         64G         2-00:00:00   gold
3      vang      1         32G         2-00:00:00   gold
4      dreamlab  6         15G         3-00:00:00   basic
4      michaelk  1         32G         12:00:00     qos1
4      tbhmbi    1         6G          5-00:00:00   qos1
6      lucie     16        128G        1-06:00:00   qos1
12     yanciaux  8         2G          4:00:00      basic
20     vang      2         8G          2-00:00:00   gold
25     zexi      8         80G         10-12:00:00  basic
30     vang      1         128G        2-00:00:00   gold
38     lucie     2         64G         20:00:00     qos1
42     lucie     1         100G        2-12:00:00   qos1
84     vang      1         4G          12:00:00     gold
215    lucie     2         10G         3-00:00:00   qos1
359    mroussel  1         6G          20:00        qos1
647    mroussel  1         15G         20:00:00     qos1
###############
# QUEUED JOBS #
###############
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT   QOS
-----  ----      --------  ----------  ---------   ---
2      conlon    18        24G         1-00:00:00  qos1
16     lucie     2         64G         20:00:00    qos1
23     mroussel  1         6G          20:00       qos1
44     dreamlab  3         8G          1-00:00:00  basic
64     mroussel  1         15G         20:00:00    qos1
175    lucie     1         100G        2-12:00:00  qos1
 
#################
# CPUs per User #
#################
NCPUs  USER      QOS
-----  ----      ---
1      b960499   basic
1      jusren    basic
1      leh       basic
1      michaelk  bumped
1      simong    bumped
2      dario     qos1
2      iastar    qos1
2      tbhmbi    basic
4      iprada    qos1
4      mtxellrb  basic
4      tbhmbi    qos1
5      michaelk  qos1
6      dreamlab  bumped
10     emp       qos1
10     qianqian  qos1
16     au342309  basic
33     dreamlab  basic
96     yanciaux  basic
178    vang      gold
192    ijjduan   qos1
208    zexi      basic
648    lucie     qos1
1006   mroussel  qos1
########################
# QUEUED CPUs per User #
########################
NCPUs  USER      QOS
-----  ----      ---
36     conlon    qos1
87     mroussel  qos1
132    dreamlab  basic
207    lucie     qos1