[2020-08-13 13:08:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-08-13 13:08:02]

 | 3-11|   OOOOOOOO | 3-21|     DOWN   | 3-31|   OOOOOOO. | 3-41|   ........ | 3-51|     DOWN   | 3-61|     DOWN   | 3-71|   OO......
 | 3-12|   OOOOOOO. | 3-22|   ........ | 3-32|   OOO..... | 3-42|     DOWN   | 3-52|   OOOOOOOO | 3-62|   OOOO.... | 3-72|     DOWN  
 | 3-13|     DOWN   | 3-23|   OOOO.... | 3-33|   ........ | 3-43|   OOOOOO.. | 3-53|   ........ | 3-63|   OOOOOO.. | 3-73|   ........
 | 3-14|     DOWN   | 3-24|   OOOOOO.. | 3-34|   ........ | 3-44|   OOOOOO.. | 3-54|   ........ | 3-64|   OOOOO... | 3-74|   ........
 | 3-15|     DOWN   | 3-25|     DOWN   | 3-35|   ........ | 3-45|   ........ | 3-55|   ........ | 3-65|   OOOOOOO. | 3-75|   ........
 | 3-16|   OOOO.... | 3-26|     DOWN   | 3-36|   OOOO.... | 3-46|   ........ | 3-56|   ........ | 3-66|     DOWN   | 3-76|     DOWN  
 | 3-17|   ........ | 3-27|   OOOOOO.. | 3-37|     DOWN   | 3-47|   ........ | 3-57|   ........ | 3-67|     DOWN   | 3-77|   OOOOOO..
 | 3-18|   OOOOOOOO | 3-28|     DOWN   | 3-38|   OOOOOOOO | 3-48|     DOWN   | 3-58|   ........ | 3-68|   OOOOOO.. | 3-78|   ........


 | 4-11|   OOOO.... | 4-21|   ........ | 4-31|   ........ | 4-41|     DOWN   | 4-51|   ........ | 4-61|   ........ | 4-71|   ........ | 4-81|   ........
 | 4-12|   ........ | 4-22|   ........ | 4-32|   ........ | 4-42|   O....... | 4-52|   ........ | 4-62|   ........ | 4-72|   O....... | 4-82|   ........
 | 4-13|   ........ | 4-23|   ........ | 4-33|     DOWN   | 4-43|     DOWN   | 4-53|   OO...... | 4-63|   ........ | 4-73|   ........ | 4-83|   ........
 | 4-14|   ........ | 4-24|   OOOO.... | 4-34|     DOWN   | 4-44|   ........ | 4-54|   ........ | 4-64|   ........ | 4-74|   ........                   


 | 5- 1|     DOWN   | 5-11|   ........ | 5-21|   OOO.....
 | 5- 2|   O....... | 5-12|   ........ | 5-22|   ........
 | 5- 3|     DOWN   | 5-13|   ........ | 5-23|   ........
 | 5- 4|   OOO..... | 5-14|   ........ | 5-24|     DOWN  
 | 5- 5|   OOOOOOOO | 5-15|   ........ | 5-25|     DOWN  
 | 5- 6| * O....... | 5-16|   OO...... | 5-26|   ........
 | 5- 7|   ........ | 5-17|   ........ | 5-27|   ........
 | 5- 8|   ........ | 5-18|   ........ | 5-28|   ........
 | 5- 9|   ........ | 5-19|     DOWN   | 5-29|   ........
 | 5-10|     DOWN   | 5-20|     DOWN   | 5-30|   ........


 | 9- 1|   ........
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   OOO.....
 |10- 2|   ........


 |21-11|   O....... |21-21|   ........ |21-31|   ........ |21-41|   ........ |21-51|   ........ |21-61|   OOOOOOOO |21-71|   ........ |21-81|   OOOOOOOO |21-91|     DOWN  
 |21-12|   ........ |21-22|   ........ |21-32|   ........ |21-42|   ........ |21-52|     DOWN   |21-62|   OOOOOOOO |21-72|     DOWN   |21-82|   OOOOOOOO |21-92|     DOWN  
 |21-13|   ........ |21-23|   ........ |21-33|   ........ |21-43|   ........ |21-53|   ........ |21-63|   OOOOO... |21-73|   OOOOOOOO |21-83|   ........ |21-93|     DOWN  
 |21-14|   ........ |21-24|   ........ |21-34|   ........ |21-44|   ........ |21-54|   OOOOOOOO |21-64|     DOWN   |21-74|   OOOOOOO. |21-84|   ........ |21-94|     DOWN  


 |22-11|     DOWN   |22-21|   ........ |22-31|   OOOO.... |22-41|   OO...... |22-51|     DOWN  
 |22-12|   O....... |22-22|   ........ |22-32|   OOOOOOOO |22-42|   ........ |22-52|     DOWN  
 |22-13|     DOWN   |22-23|   ........ |22-33|     DOWN   |22-43|   OO...... |22-53|   ........
 |22-14|     DOWN   |22-24|   ........ |22-34|   OOOO.... |22-44|     DOWN   |22-54|   ........


 |91- 1|     DOWN  
 |91- 2|     DOWN  
 |91- 3|     DOWN  
 |91- 4|     DOWN  
 |91- 5|     DOWN  
 |91- 6|     DOWN  


Totals:
Total cores: 6568 
Total memory: 58.03 TiB 

Down:
Down cores: 1655 ( 25.20%) 
Down memory: 13.62 TiB ( 23.48%) 

Used:
Used cores: 908 ( 13.82%) 
Used memory: 7.87 TiB ( 13.56%) 

Utilization excluding down
CPU utilization:  18.48% (global)  19.63% (average per node)
Memory utilization:  17.73% (global)  20.53% (average per node)

Average per node utilization (excluding down):  22.83% 
[2020-08-13 13:08:02]
s03n13 health-check: s03n13 13-Aug-2020 13:01 Lost part of memory 
s03n14 Not responding 
s03n15 Not responding 
s03n21 Not responding 
s03n25 Node unexpectedly rebooted 
s03n26 Not responding 
s03n28 Not responding 
s03n37 Not responding 
s03n42 Node unexpectedly rebooted 
s03n48 Not responding 
s03n51 health-check: s03n51 13-Aug-2020 13:01 beegfs client Error storage targetID 
s03n61 health-check: s03n61 13-Aug-2020 13:01 /faststorage not available 
s03n66 Not responding 
s03n67 sanity-check: s03n67 30-Jun-2020 11:01 /faststorage not available 
s03n72 Not responding 
s03n76 Not responding 
s04n33 Kill task failed 
s04n34 Node unexpectedly rebooted 
s04n41 Node unexpectedly rebooted 
s04n43 health-check: s04n43 13-Aug-2020 13:01 beegfs client Error storage targetID 
s05n01 Node unexpectedly rebooted 
s05n03 Node unexpectedly rebooted 
s05n06 health-check: s05n06 13-Aug-2020 13:01 beegfs client Error storage targetID 
s05n10 Node unexpectedly rebooted 
s05n19 Not responding 
s05n20 Node unexpectedly rebooted 
s05n24 Not responding 
s05n25 Not responding 
s21n52 Not responding 
s21n64 Not responding 
s21n72 Not responding 
s21n91 fan malfunction 
s21n92 fan malfunction 
s21n93 fan malfunction 
s21n94 fan malfunction 
s22n11 Create /scratch folder failed 
s22n13 Create /scratch folder failed 
s22n14 Node unexpectedly rebooted 
s22n33 Node unexpectedly rebooted 
s22n44 Create /scratch folder failed 
s22n51 has issues 
s22n52 potential memory errors 
s91n01 Not responding 
s91n02 Not responding 
s91n03 Not responding 
s91n04 Not responding 
s91n05 Not responding 
s91n06 Node unexpectedly rebooted 

[2020-08-13 13:05:14]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-08-13 13:00:04]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       67659.8GB       12348.3GB    85% [#################   ]      1490.3M      1487.6M  100%
      9501       3002       normal       80008.1GB       67915.2GB       12092.8GB    85% [#################   ]      1490.3M      1487.6M  100%
      9501       3003       normal       80008.1GB       67597.6GB       12410.5GB    84% [#################   ]      1490.3M      1487.6M  100%
      9501       3004       normal       80008.1GB       67996.3GB       12011.7GB    85% [#################   ]      1490.3M      1487.6M  100%
      9501       3005       normal       80008.1GB       66154.8GB       13853.3GB    83% [#################   ]      1490.3M      1487.5M  100%
      9501       3006       normal       80008.1GB       67825.6GB       12182.5GB    85% [#################   ]      1490.3M      1487.5M  100%
      9501       3007       normal       80008.1GB       68794.9GB       11213.2GB    86% [#################   ]      1490.3M      1487.6M  100%
      9501       3008       normal       80008.1GB       67326.6GB       12681.5GB    84% [#################   ]      1490.3M      1487.5M  100%
      9501       3009       normal       80008.1GB       67997.2GB       12010.9GB    85% [#################   ]      1490.3M      1487.5M  100%
      9501       3010       normal       80008.1GB       66579.4GB       13428.7GB    83% [#################   ]      1490.3M      1487.5M  100%
      9501       3011       normal       80008.1GB       64623.4GB       15384.7GB    81% [################    ]      1490.3M      1487.5M  100%
      9501       3012    emergency       80008.1GB       67073.0GB       12935.0GB    84% [#################   ]      1490.3M      1487.5M  100%
      9501       3013       normal       80008.1GB       66353.7GB       13654.4GB    83% [#################   ]      1490.3M      1487.5M  100%
      9502       3014       normal       80008.1GB       64499.2GB       15508.8GB    81% [################    ]      1490.3M      1487.5M  100%
      9502       3015       normal       80008.1GB       66367.7GB       13640.3GB    83% [#################   ]      1490.3M      1487.5M  100%
      9502       3016          low       80008.1GB       71273.6GB        8734.5GB    89% [##################  ]      1490.3M      1489.6M  100%
      9502       3017       normal       80008.1GB       66624.2GB       13383.8GB    83% [#################   ]      1490.3M      1487.5M  100%
      9502       3018       normal       80008.1GB       68165.0GB       11843.1GB    85% [#################   ]      1490.3M      1487.5M  100%
      9502       3019       normal       80008.1GB       66338.7GB       13669.4GB    83% [#################   ]      1490.3M      1487.5M  100%
      9502       3020       normal       80008.1GB       67440.8GB       12567.3GB    84% [#################   ]      1490.3M      1487.5M  100%
      9601       3021       normal       80008.1GB       66978.7GB       13029.3GB    84% [#################   ]      1490.3M      1487.5M  100%
      9601       3022       normal       80008.1GB       66228.2GB       13779.8GB    83% [#################   ]      1490.3M      1487.5M  100%
      9601       3023       normal       80008.1GB       66772.2GB       13235.9GB    83% [#################   ]      1490.3M      1487.5M  100%
      9601       3024       normal       80008.1GB       68239.6GB       11768.4GB    85% [#################   ]      1490.3M      1487.5M  100%
      9601       3025       normal       80008.1GB       66388.9GB       13619.2GB    83% [#################   ]      1490.3M      1487.5M  100%
      9601       3026    emergency       80008.1GB       65323.0GB       14685.1GB    82% [################    ]      1490.3M      1487.6M  100%
      9601       3027       normal       80008.1GB       68078.3GB       11929.8GB    85% [#################   ]      1490.3M      1487.6M  100%
      9601       3028       normal       80008.1GB       66477.0GB       13531.1GB    83% [#################   ]      1490.3M      1487.5M  100%
      9601       3029       normal       80008.1GB       67027.2GB       12980.9GB    84% [#################   ]      1490.3M      1487.5M  100%
      9601       3030       normal       80008.1GB       67241.4GB       12766.7GB    84% [#################   ]      1490.3M      1487.6M  100%
      9601       3031       normal       80008.1GB       67975.5GB       12032.5GB    85% [#################   ]      1490.3M      1487.6M  100%
      9601       3032       normal       80008.1GB       67127.7GB       12880.4GB    84% [#################   ]      1490.3M      1487.6M  100%
      9601       3033       normal       80008.1GB       68284.8GB       11723.3GB    85% [#################   ]      1490.3M      1487.6M  100%
      9601       3034       normal       80008.1GB       65881.0GB       14127.0GB    82% [################    ]      1490.3M      1487.6M  100%
      9602       3035       normal       80008.1GB       66307.6GB       13700.5GB    83% [#################   ]      1490.3M      1487.6M  100%
      9602       3036       normal       80008.1GB       67035.8GB       12972.3GB    84% [#################   ]      1490.3M      1487.6M  100%
      9602       3037       normal       80008.1GB       67119.1GB       12888.9GB    84% [#################   ]      1490.3M      1487.6M  100%
      9602       3038       normal       80008.1GB       68149.3GB       11858.7GB    85% [#################   ]      1490.3M      1487.7M  100%
      9602       3039       normal       80008.1GB       67226.7GB       12781.4GB    84% [#################   ]      1490.3M      1487.6M  100%
      9602       3040       normal       80008.1GB       67240.7GB       12767.4GB    84% [#################   ]      1490.3M      1487.6M  100%
      9602       3041       normal       80008.1GB       67438.0GB       12570.0GB    84% [#################   ]      1490.3M      1487.6M  100%
      9701       3042       normal      119987.9GB       73158.7GB       46829.2GB    61% [############        ]      2235.0M      2225.5M  100%
      9701       3043       normal      119987.9GB       74719.9GB       45268.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3044       normal      119987.9GB       74923.8GB       45064.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3045       normal      119987.9GB       75159.7GB       44828.1GB    63% [#############       ]      2235.0M      2225.5M  100%
      9701       3046       normal      119987.9GB       73587.9GB       46400.0GB    61% [############        ]      2235.0M      2225.5M  100%
      9701       3047       normal      119987.9GB       74403.9GB       45584.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3048       normal      119987.9GB       73702.5GB       46285.4GB    61% [############        ]      2235.0M      2225.5M  100%
      9701       3049       normal      119987.9GB       75034.2GB       44953.7GB    63% [#############       ]      2235.0M      2225.5M  100%
      9701       3050       normal      119987.9GB       73955.3GB       46032.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3051       normal      119987.9GB       74825.5GB       45162.3GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3052       normal      119987.9GB       74049.2GB       45938.7GB    62% [############        ]      2235.0M      2225.5M  100%
      9701       3053       normal      119987.9GB       75417.5GB       44570.3GB    63% [#############       ]      2235.0M      2225.5M  100%
      9701       3054       normal      119987.9GB       75335.7GB       44652.1GB    63% [#############       ]      2235.0M      2225.5M  100%
      9701       3055       normal      119987.9GB       74647.3GB       45340.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3056       normal      119987.9GB       74261.1GB       45726.8GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3057       normal      119987.9GB       75223.9GB       44764.0GB    63% [#############       ]      2235.0M      2225.5M  100%
      9702       3058       normal      119987.9GB       74054.3GB       45933.6GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3059       normal      119987.9GB       74799.4GB       45188.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3060       normal      119987.9GB       74726.3GB       45261.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3061       normal      119987.9GB       74738.3GB       45249.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3062       normal      119987.9GB       74476.9GB       45511.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3063       normal      119987.9GB       75422.6GB       44565.3GB    63% [#############       ]      2235.0M      2225.5M  100%
      9702       3064       normal      119987.9GB       74464.4GB       45523.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3065       normal      119987.9GB       74158.0GB       45829.8GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3066       normal      119987.9GB       74917.2GB       45070.7GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3067       normal      119987.9GB       74734.1GB       45253.8GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3068       normal      119987.9GB       74731.8GB       45256.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9702       3069       normal      119987.9GB       74209.4GB       45778.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3070       normal      119987.9GB       74299.0GB       45688.9GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3071       normal      119987.9GB       74407.1GB       45580.8GB    62% [############        ]      2235.0M      2228.4M  100%
      9801       3072       normal      119987.9GB       75151.9GB       44836.0GB    63% [#############       ]      2235.0M      2225.5M  100%
      9801       3073       normal      119987.9GB       73959.6GB       46028.3GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3074       normal      119987.9GB       74767.7GB       45220.1GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3075       normal      119987.9GB       73574.2GB       46413.7GB    61% [############        ]      2235.0M      2225.5M  100%
      9801       3076       normal      119987.9GB       74471.4GB       45516.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3077       normal      119987.9GB       74999.5GB       44988.3GB    63% [#############       ]      2235.0M      2225.5M  100%
      9801       3078       normal      119987.9GB       74512.5GB       45475.4GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3079       normal      119987.9GB       74886.2GB       45101.7GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3080       normal      119987.9GB       73676.1GB       46311.8GB    61% [############        ]      2235.0M      2228.3M  100%
      9801       3081       normal      119987.9GB       74038.3GB       45949.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3082       normal      119987.9GB       74771.3GB       45216.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9801       3083       normal      119987.9GB       74441.6GB       45546.2GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3084       normal      119987.9GB       73598.7GB       46389.1GB    61% [############        ]      2235.0M      2225.5M  100%
      9802       3085       normal      119987.9GB       73753.9GB       46234.0GB    61% [############        ]      2235.0M      2225.5M  100%
      9802       3086       normal      119987.9GB       75370.1GB       44617.7GB    63% [#############       ]      2235.0M      2225.5M  100%
      9802       3087       normal      119987.9GB       74772.3GB       45215.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3088       normal      119987.9GB       74406.2GB       45581.7GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3089       normal      119987.9GB       74294.9GB       45692.9GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3090       normal      119987.9GB       75357.2GB       44630.6GB    63% [#############       ]      2235.0M      2225.5M  100%
      9802       3091       normal      119987.9GB       74390.0GB       45597.9GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3092       normal      119987.9GB       75065.0GB       44922.8GB    63% [#############       ]      2235.0M      2225.5M  100%
      9802       3093       normal      119987.9GB       74197.9GB       45790.0GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3094       normal      119987.9GB       75442.0GB       44545.9GB    63% [#############       ]      2235.0M      2225.5M  100%
      9802       3095       normal      119987.9GB       74911.3GB       45076.5GB    62% [############        ]      2235.0M      2225.5M  100%
      9802       3096       normal      119987.9GB       73589.0GB       46398.8GB    61% [############        ]      2235.0M      2225.5M  100%
      9802       3097       normal      119987.9GB       75033.8GB       44954.0GB    63% [#############       ]      2235.0M      2225.5M  100%
      9502       3098       normal      119993.9GB       66736.5GB       53257.4GB    56% [###########         ]      2235.1M      2232.4M  100%
      9502       3099       normal      119993.9GB       67834.8GB       52159.1GB    57% [###########         ]      2235.1M      2232.4M  100%
      9502       3100       normal      119993.9GB       66901.7GB       53092.2GB    56% [###########         ]      2235.1M      2232.4M  100%
      9502       3101       normal      119993.9GB       66278.4GB       53715.5GB    55% [###########         ]      2235.1M      2232.4M  100%
      9502       3102       normal      119993.9GB       66366.3GB       53627.6GB    55% [###########         ]      2235.1M      2232.4M  100%
      9502       3103       normal      119993.9GB       67305.0GB       52688.9GB    56% [###########         ]      2235.1M      2232.4M  100%
      9502       3104       normal      119993.9GB       67320.2GB       52673.7GB    56% [###########         ]      2235.1M      2232.4M  100%
      9602       3105       normal      119993.9GB       65416.5GB       54577.3GB    55% [###########         ]      2235.1M      2232.4M  100%
      9602       3106       normal      119993.9GB       66064.1GB       53929.8GB    55% [###########         ]      2235.1M      2232.4M  100%
      9602       3107       normal      119993.9GB       67614.0GB       52379.8GB    56% [###########         ]      2235.1M      2232.4M  100%
      9602       3108       normal      119993.9GB       68033.0GB       51960.8GB    57% [###########         ]      2235.1M      2232.4M  100%
      9602       3109       normal      119993.9GB       67293.0GB       52700.9GB    56% [###########         ]      2235.1M      2232.4M  100%
      9602       3110       normal      119993.9GB       66901.0GB       53092.8GB    56% [###########         ]      2235.1M      2232.4M  100%
      9602       3111       normal      119993.9GB       67533.0GB       52460.8GB    56% [###########         ]      2235.1M      2232.4M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7864722.7 GB
Total free space on /faststorage        3814842.8 GB

                                 Use%   67.34%

2020-08-13 13:09:10.929598233 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-08-12 15:02:01] s21n94 now marked for drain: fan malfunction 
[2020-08-12 15:02:01] s21n93 now marked for drain: fan malfunction 
[2020-08-12 15:02:01] s21n92 now marked for drain: fan malfunction 
[2020-08-12 15:02:01] s21n91 now marked for drain: fan malfunction 
[2020-08-12 13:58:02] s21n43 resumed
[2020-08-12 13:56:01] s21n43 now marked for drain: health-check: can't ping gfs6 
[2020-08-12 13:32:07] s21n61 resumed
[2020-08-12 13:30:02] s21n61 now marked for drain: health-check: can't ping gfs5 
[2020-08-12 11:46:01] s05n03 now marked for drain: Node unexpectedly rebooted 
[2020-08-12 11:34:02] s21n31 resumed
[2020-08-12 11:32:02] s21n31 now marked for drain: health-check: can't ping gfs5 
[2020-08-12 09:20:02] s04n41 now marked for drain: Node unexpectedly rebooted 
[2020-08-12 09:20:02] s03n42 now marked for drain: Node unexpectedly rebooted 
[2020-08-12 08:48:02] s91n06 now marked for drain: Node unexpectedly rebooted 
[2020-08-12 01:14:01] s22n21 resumed
[2020-08-12 01:12:01] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 23:12:01] s22n21 resumed
[2020-08-11 23:10:02] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 22:56:02] s22n21 resumed
[2020-08-11 22:52:02] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 22:42:02] s22n21 resumed
[2020-08-11 22:40:02] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 22:28:02] s22n21 resumed
[2020-08-11 22:24:01] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 22:18:02] s22n21 resumed
[2020-08-11 22:16:02] s22n21 now marked for drain: health-check: /faststorage not available 
[2020-08-11 22:00:02] s22n21 resumed
[2020-08-11 21:56:02] s22n21 now marked for drain: health-check: /proc hangs 
[2020-08-11 13:20:03] s05n25 now marked for drain: Not responding 
[2020-08-11 12:48:02] s05n10 now marked for drain: Node unexpectedly rebooted 
[2020-08-11 12:28:02] s22n14 now marked for drain: Not responding 
[2020-08-11 12:26:02] s22n33 now marked for drain: Not responding 
[2020-08-11 12:26:02] s05n01 now marked for drain: Node unexpectedly rebooted 
[2020-08-11 12:22:02] s04n34 now marked for drain: Node unexpectedly rebooted 
[2020-08-10 21:52:01] s91n06 resumed
[2020-08-10 21:50:02] s91n06 now marked for drain: health-check: can't ping gfs3 
[2020-08-10 13:12:02] s22n22 resumed
[2020-08-10 13:10:02] s22n22 now marked for drain: health-check: df /faststorage hangs 
[2020-08-10 13:06:02] s22n22 resumed
[2020-08-10 13:02:01] s22n22 now marked for drain: health-check: df /faststorage hangs 
[2020-08-10 12:38:01] s21n51 resumed
[2020-08-10 12:36:02] s21n51 now marked for drain: health-check: df /faststorage hangs 
[2020-08-10 12:28:01] s22n24 resumed
[2020-08-10 11:24:02] s22n34 resumed
[2020-08-10 11:22:02] s22n34 now marked for drain: health-check: can't ping gfs6 
[2020-08-10 10:30:02] s21n21 resumed
[2020-08-10 10:28:03] s21n21 now marked for drain: health-check: can't ping gfs3 
[2020-08-10 09:52:01] s22n23 resumed
[2020-08-10 09:48:02] s22n33 resumed
[2020-08-10 09:46:02] s22n33 now marked for drain: health-check: can't ping gfs3 
[2020-08-10 09:38:01] s22n22 resumed
[2020-08-13 13:00:03]
Backup on    fs1  finished 0-11:01:47 ago
Backup on    fs2 initiated 13-04:31:43 ago
Backup on    fs3  finished -22--18:-32:-4 ago
Backup on    fs5 initiated 0-15:00:01 ago
Backup on    fs6  finished 0-05:57:00 ago
Backup on s95n01  finished 0-13:17:03 ago
[2020-08-13 13:08:02]
################
# RUNNING JOBS #
################
Count  USER       MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----       --------  ----------  ---------    ---
1      anksi      1         30G         7-04:00:00   basic
1      besen      1         32G         4-04:00:00   basic
1      clara      16        128G        12:00:00     qos1
1      clara      36        284G        1-01:00:00   qos1
1      dreamlab   3         15G         3-00:00:00   basic
1      fprive     16        128G        12:00:00     qos1
1      gockert    1         4G          4:00:00      basic
1      jbbr       12        50G         12:00:00     qos1
1      jbbr       12        99G         12:00:00     qos1
1      jklassen   8         32G         4:00:00      basic
1      jrouv      12        128G        3-00:00:00   basic
1      julienlin  2         100G        3:00:00      qos1
1      kleppe     1         11G         4:00:00      basic
1      kmt        10        60G         3-00:00:00   basic
1      michaelk   36        128G        12-00:00:00  gold
1      mroussel   5         18G         20:00:00     qos1
1      mtxellrb   1         100G        1-00:00:00   bumped
1      mtxellrb   1         1G          3-00:00:00   qos1
1      mtxellrb   1         200G        4:00:00      qos1
1      noor       1         180G        4-04:00:00   basic
1      randiip    1         64G         12:00:00     basic
1      siavl      16        128G        12:00:00     basic
1      sla        1         256G        8:00:00      basic
1      sla        64        503G        12:00:00     basic
1      sschmidt   16        120G        4-04:00:00   bumped
1      targe      1         8G          12:00:00     qos1
1      tas        1         128G        6-06:00:00   bumped
1      zexi       8         20G         30-12:00:00  basic
2      iprada     4         32G         4:00:00      qos1
2      mica16     8         20G         20:00:00     qos1
2      pso        8         50G         2-02:00:00   basic
2      tas        1         32G         3-00:00:00   qos1
3      iker       4         32G         20:00:00     qos1
3      jbbr       12        80G         12:00:00     qos1
3      kmt        1         8G          1-00:00:00   qos1
3      zexi       1         10G         10-12:00:00  basic
4      juraj      2         40G         2-00:00:00   bumped
5      tas        1         128G        6-06:00:00   qos1
6      plamy      64        503G        12:00:00     basic
9      igrove     1         8G          8:00:00      qos2
9      jbbr       12        40G         12:00:00     qos1
14     igrove     1         12G         8:00:00      qos2
###############
# QUEUED JOBS #
###############
Count  USER  MIN_CPUS  MIN_MEMORY  TIMELIMIT  QOS
-----  ----  --------  ----------  ---------  ---
 
#################
# CPUs per User #
#################
NCPUs  USER       QOS
-----  ----       ---
1      anksi      basic
1      besen      basic
1      gockert    basic
1      kleppe     basic
1      mtxellrb   bumped
1      noor       basic
1      randiip    basic
1      targe      qos1
1      tas        bumped
2      julienlin  qos1
2      mtxellrb   qos1
3      dreamlab   basic
3      kmt        qos1
5      mroussel   qos1
7      tas        qos1
8      iprada     qos1
8      jklassen   basic
8      juraj      bumped
10     kmt        basic
11     zexi       basic
12     iker       qos1
12     jrouv      basic
16     fprive     qos1
16     mica16     qos1
16     pso        basic
16     siavl      basic
16     sschmidt   bumped
23     igrove     qos2
36     michaelk   gold
52     clara      qos1
65     sla        basic
168    jbbr       qos1
384    plamy      basic
########################
# QUEUED CPUs per User #
########################
NCPUs  USER  QOS
-----  ----  ---