____ _ | __ )| | ___ __ _ | _ \| |/ _ \ / _` | | |_) | | (_) | (_| | |____/|_|\___/ \__, | |___/
_ _ _/_| |__ ___ _ __| |_ _ _/_ / _ \ '_ \ / _ \ '__| | | | |/ _ \ __/ |_) | __/ | | | |_| | __/ \___|_.__/ \___|_| |_|\__,_|\___|

[2026-09-04][Daily]

First steps using Perf

For my next paper, I want to study the impact (and pontential advantages) of using Arenas. I will need to retrieve many low level counters to compute metrics such as instructions/cycle, FLOPS, Cache Miss/Hit ratio...

This post documents usage of perf tool and shows selected counters relevant for my work. Here is the command I've come up so far:

perf stat \
    -e branch-instructions,branch-misses,cycles,instructions,ref-cycles \
    -e context-switches,cpu-migrations,faults,major-faults,minor-faults \
    -e l1-dcache-loads,l1-dcache-load-misses,l1-dcache-stores,
    -e l1-icache-loads,l1-icache-load-misses \
    -e llc-load-misses,llc-loads,llc-store-misses,llc-stores \
    -e branch-load-misses,branch-loads \
    -x , -o perf-output.csv \
    -p pid

Optionnaly enable -a and -A to have counters per cpu core. Flag -x , is there to format in a csv like format.

Selected counters

Hardware

branch-instructions
[Retired branch instructions [This event is an alias of branches]. Unit: cpu_atom]
branch-misses
[Mispredicted branch instructions. Unit: cpu_atom]
cycles
[Total cycles. Be wary of what happens during CPU frequency scaling [This event is an alias of cpu-cycles]. Unit: cpu_atom]
instructions
[Retired instructions. Be careful,these can be affected by various issues,most notably hardware interrupt counts. Unit: cpu_atom]
ref-cycles
[Total cycles; not affected by CPU frequency scaling. Unit: cpu_atom]

Software

context-switches
[Number of context switches [This event is an alias of cs]. Unit: software]
cpu-migrations
[Number of times a process has migrated to a new CPU [This event is an alias of migrations]. Unit: software]
faults
[Number of page faults [This event is an alias of page-faults]. Unit: software]
major-faults
[Number of major page faults. Major faults require I/O to handle. Unit: software]
minor-faults
[Number of minor page faults. Minor faults don't require I/O to handle. Unit: software]

Cache

l1-dcache-loads
[Level 1 data cache read accesses. Unit: cpu_atom]
l1-dcache-load-misses
[Level 1 data cache read misses. Unit: cpu_core]
l1-dcache-stores
[Level 1 data cache write accesses. Unit: cpu_atom]
l1-icache-load-misses
[Level 1 instruction cache read misses. Unit: cpu_atom]
l1-icache-loads
[Level 1 instruction cache read accesses. Unit: cpu_atom]
llc-load-misses
[Last level cache read misses. Unit: cpu_atom]
llc-loads
[Last level cache read accesses. Unit: cpu_atom]
llc-store-misses
[Last level cache write misses. Unit: cpu_atom]
llc-stores
[Last level cache write accesses. Unit: cpu_atom]
branch-load-misses
[Branch prediction unit read misses. Unit: cpu_atom]
branch-loads
[Branch prediction unit read accesses. Unit: cpu_atom]

Floating point operations

fp_flops_retired.fp64
[Counts the number of floating point operations that produce 64 bit double precision results [This event is alias to FP_FLOPS_RETIRED.DP]. Unit: cpu_atom]
fp_flops_retired.fp32
[Counts the number of floating point operations that produce 32 bit single precision results [This event is alias to FP_FLOPS_RETIRED.SP]. Unit: cpu_atom]

Sadly those counters are not available on my servers, only on my laptop.

Output example

Performance counter stats for './basic_cnn fashion-mnist ../../datasets/fashion-mnist 512 64 5':

   26 212 766 105      cpu_atom/branch-instructions/                                           (28,57%)
   27 584 767 272      cpu_core/branch-instructions/                                           (8,64%)
       93 194 748      cpu_atom/branch-misses/                                                 (33,41%)
      134 741 314      cpu_core/branch-misses/                                                 (10,44%)
  122 768 836 200      cpu_atom/cycles/                                                        (38,20%)
  172 871 719 251      cpu_core/cycles/                                                        (12,19%)
  160 269 866 844      cpu_atom/instructions/                                                  (42,94%)
  166 388 277 075      cpu_core/instructions/                                                  (13,92%)
  115 056 226 759      cpu_atom/ref-cycles/                                                    (47,64%)
  111 458 554 838      cpu_core/ref-cycles/                                                    (15,61%)
        2 302 867      context-switches                                                      
           25 174      cpu-migrations                                                        
            8 460      faults                                                                
                0      major-faults                                                          
            8 460      minor-faults                                                          
   52 198 032 375      cpu_atom/l1-dcache-loads/                                               (47,67%)
   53 160 336 862      cpu_core/l1-dcache-loads/                                               (17,26%)
   23 999 580 499      cpu_atom/l1-dcache-stores/                                              (47,71%)
   24 633 512 246      cpu_core/l1-dcache-stores/                                              (17,25%)
    4 295 354 172      cpu_atom/l1-icache-load-misses/                                         (47,74%)
    2 052 490 356      cpu_core/l1-icache-load-misses/                                         (17,23%)
   52 388 909 323      cpu_atom/l1-icache-loads/                                               (47,69%)
   not supported       cpu_core/l1-icache-loads/                                             
       32 125 124      cpu_atom/llc-load-misses/                                               (47,74%)
       11 827 940      cpu_core/llc-load-misses/                                               (17,22%)
      177 384 183      cpu_atom/llc-loads/                                                     (47,63%)
      194 899 053      cpu_core/llc-loads/                                                     (6,80%)
       10 043 198      cpu_atom/llc-store-misses/                                              (9,47%)
        2 295 868      cpu_core/llc-store-misses/                                              (3,41%)
       93 344 359      cpu_atom/llc-stores/                                                    (9,59%)
      107 279 772      cpu_core/llc-stores/                                                    (3,42%)
       92 980 212      cpu_atom/branch-load-misses/                                            (14,33%)
      133 434 049      cpu_core/branch-load-misses/                                            (5,17%)
   26 221 176 502      cpu_atom/branch-loads/                                                  (19,04%)
   27 335 669 391      cpu_core/branch-loads/                                                  (6,92%)
      349 684 052      cpu_atom/fp_flops_retired.fp64/                                         (23,74%)

      9,957278848 seconds time elapsed

     29,375730000 seconds user
     21,445520000 seconds sys

Note that cpu_atom = E-cores / LPE-cores (in my experience) and cpu_core = P-cores.

Example of csv-like output:

25144764209;;cpu_atom/branch-instructions/;17765790212;31;00;;
24817012457;;cpu_core/branch-instructions/;5090736703;9;00;;
92593065;;cpu_atom/branch-misses/;20266793177;35;00;;
126143049;;cpu_core/branch-misses/;6069319907;10;00;;
117201124971;;cpu_atom/cycles/;20299371372;35;00;;
151183640355;;cpu_core/cycles/;7070060511;12;00;;
151342931683;;cpu_atom/instructions/;22863262037;40;00;;
148726762763;;cpu_core/instructions/;8062846654;14;00;;
126165326727;;cpu_atom/ref-cycles/;25424751137;44;00;;
122729020009;;cpu_core/ref-cycles/;9059653042;16;00;;
2233660;;context-switches;56942186418;100;00;;
23957;;cpu-migrations;56942186418;100;00;;
8522;;faults;56942186418;100;00;;
0;;major-faults;56942186418;100;00;;
8522;;minor-faults;56942186418;100;00;;
48876394913;;cpu_atom/l1-dcache-loads/;25418226106;44;00;;
47451821747;;cpu_core/l1-dcache-loads/;10076498451;17;00;;
22484823735;;cpu_atom/l1-dcache-stores/;25347635096;44;00;;
22102337888;;cpu_core/l1-dcache-stores/;10004627221;17;00;;
4328437094;;cpu_atom/l1-icache-load-misses/;25270445505;44;00;;
1871586921;;cpu_core/l1-icache-load-misses/;10024281163;17;00;;
49661278480;;cpu_atom/l1-icache-loads/;25187728645;44;00;;
 not supported ;;cpu_core/l1-icache-loads/;0;100;00;;
32010133;;cpu_atom/llc-load-misses/;25210259523;44;00;;
10511476;;cpu_core/llc-load-misses/;9992310912;17;00;;
176444977;;cpu_atom/llc-loads/;25123817829;44;00;;
185671164;;cpu_core/llc-loads/;3976079926;7;00;;
9627984;;cpu_atom/llc-store-misses/;4998528642;8;00;;
1516057;;cpu_core/llc-store-misses/;2054000003;3;00;;
91444482;;cpu_atom/llc-stores/;5027798667;8;00;;
106497905;;cpu_core/llc-stores/;2022424815;3;00;;
91974699;;cpu_atom/branch-load-misses/;7600657050;13;00;;
124853032;;cpu_core/branch-load-misses/;3054537094;5;00;;
25290746070;;cpu_atom/branch-loads/;10140195981;17;00;;
24744463103;;cpu_core/branch-loads/;4054059728;7;00;;
371742466;;cpu_atom/fp_flops_retired.fp64/;12690258831;22;00;;
370677000;;cpu_atom/fp_flops_retired.all/;15198414678;26;00;;

With fields being: counter-value, unit, event-name, runtime, percentage-running, variance, metric-value, metric-unit

I also found out json produces cleaner output! Use -j for that.

{"counter-value" : "25309645598,000000", "unit" : "", "event" : "cpu_atom/branch-instructions/", "event-runtime" : 17614646985, "pcnt-running" : 31,00}
{"counter-value" : "25775678634,000000", "unit" : "", "event" : "cpu_core/branch-instructions/", "event-runtime" : 4870016625, "pcnt-running" : 8,00}
{"counter-value" : "96552586,000000", "unit" : "", "event" : "cpu_atom/branch-misses/", "event-runtime" : 20078467953, "pcnt-running" : 35,00}
{"counter-value" : "131529814,000000", "unit" : "", "event" : "cpu_core/branch-misses/", "event-runtime" : 5832624953, "pcnt-running" : 10,00}
{"counter-value" : "117434117061,000000", "unit" : "", "event" : "cpu_atom/cycles/", "event-runtime" : 20175070378, "pcnt-running" : 35,00}
{"counter-value" : "151845750042,000000", "unit" : "", "event" : "cpu_core/cycles/", "event-runtime" : 6810299082, "pcnt-running" : 12,00}
{"counter-value" : "153555722653,000000", "unit" : "", "event" : "cpu_atom/instructions/", "event-runtime" : 22613825275, "pcnt-running" : 40,00}
{"counter-value" : "154044207744,000000", "unit" : "", "event" : "cpu_core/instructions/", "event-runtime" : 7773398601, "pcnt-running" : 13,00}
{"counter-value" : "125791794806,000000", "unit" : "", "event" : "cpu_atom/ref-cycles/", "event-runtime" : 25096339295, "pcnt-running" : 44,00}
{"counter-value" : "121804030596,000000", "unit" : "", "event" : "cpu_core/ref-cycles/", "event-runtime" : 8711246809, "pcnt-running" : 15,00}
{"counter-value" : "2151804,000000", "unit" : "", "event" : "context-switches", "event-runtime" : 56231494594, "pcnt-running" : 100,00}
{"counter-value" : "24213,000000", "unit" : "", "event" : "cpu-migrations", "event-runtime" : 56231494594, "pcnt-running" : 100,00}
{"counter-value" : "8728,000000", "unit" : "", "event" : "faults", "event-runtime" : 56231494594, "pcnt-running" : 100,00}
{"counter-value" : "0,000000", "unit" : "", "event" : "major-faults", "event-runtime" : 56231494594, "pcnt-running" : 100,00}
{"counter-value" : "8728,000000", "unit" : "", "event" : "minor-faults", "event-runtime" : 56231494594, "pcnt-running" : 100,00}
{"counter-value" : "49664025808,000000", "unit" : "", "event" : "cpu_atom/l1-dcache-loads/", "event-runtime" : 25150592582, "pcnt-running" : 44,00}
{"counter-value" : "49182847131,000000", "unit" : "", "event" : "cpu_core/l1-dcache-loads/", "event-runtime" : 9669158885, "pcnt-running" : 17,00}
{"counter-value" : "22802501173,000000", "unit" : "", "event" : "cpu_atom/l1-dcache-stores/", "event-runtime" : 25098176644, "pcnt-running" : 44,00}
{"counter-value" : "22771016084,000000", "unit" : "", "event" : "cpu_core/l1-dcache-stores/", "event-runtime" : 9732370633, "pcnt-running" : 17,00}
{"counter-value" : "4225350234,000000", "unit" : "", "event" : "cpu_atom/l1-icache-load-misses/", "event-runtime" : 25089327139, "pcnt-running" : 44,00}
{"counter-value" : "1911354100,000000", "unit" : "", "event" : "cpu_core/l1-icache-load-misses/", "event-runtime" : 9791464789, "pcnt-running" : 17,00}
{"counter-value" : "50271648554,000000", "unit" : "", "event" : "cpu_atom/l1-icache-loads/", "event-runtime" : 25020945650, "pcnt-running" : 44,00}
{"counter-value" : " not supported ", "unit" : "", "event" : "cpu_core/l1-icache-loads/", "event-runtime" : 0, "pcnt-running" : 100,00}
{"counter-value" : "32541704,000000", "unit" : "", "event" : "cpu_atom/llc-load-misses/", "event-runtime" : 25011286060, "pcnt-running" : 44,00}
{"counter-value" : "11689129,000000", "unit" : "", "event" : "cpu_core/llc-load-misses/", "event-runtime" : 9776456073, "pcnt-running" : 17,00}
{"counter-value" : "175281364,000000", "unit" : "", "event" : "cpu_atom/llc-loads/", "event-runtime" : 25024061942, "pcnt-running" : 44,00}
{"counter-value" : "186864602,000000", "unit" : "", "event" : "cpu_core/llc-loads/", "event-runtime" : 3923626782, "pcnt-running" : 7,00}
{"counter-value" : "9736319,000000", "unit" : "", "event" : "cpu_atom/llc-store-misses/", "event-runtime" : 4957895053, "pcnt-running" : 8,00}
{"counter-value" : "4806391,000000", "unit" : "", "event" : "cpu_core/llc-store-misses/", "event-runtime" : 1872408757, "pcnt-running" : 3,00}
{"counter-value" : "91106149,000000", "unit" : "", "event" : "cpu_atom/llc-stores/", "event-runtime" : 4974864171, "pcnt-running" : 8,00}
{"counter-value" : "106020891,000000", "unit" : "", "event" : "cpu_core/llc-stores/", "event-runtime" : 1881482530, "pcnt-running" : 3,00}
{"counter-value" : "96263574,000000", "unit" : "", "event" : "cpu_atom/branch-load-misses/", "event-runtime" : 7517631810, "pcnt-running" : 13,00}
{"counter-value" : "129585878,000000", "unit" : "", "event" : "cpu_core/branch-load-misses/", "event-runtime" : 2866737517, "pcnt-running" : 5,00}
{"counter-value" : "25332721932,000000", "unit" : "", "event" : "cpu_atom/branch-loads/", "event-runtime" : 10024926471, "pcnt-running" : 17,00}
{"counter-value" : "25785828036,000000", "unit" : "", "event" : "cpu_core/branch-loads/", "event-runtime" : 3840641827, "pcnt-running" : 6,00}
{"counter-value" : "363921595,000000", "unit" : "", "event" : "cpu_atom/fp_flops_retired.fp64/", "event-runtime" : 12577970369, "pcnt-running" : 22,00}
{"counter-value" : "363098653,000000", "unit" : "", "event" : "cpu_atom/fp_flops_retired.all/","event-runtime" : 15141919442, "pcnt-running" : 26,00}

Percentage running metric

Right now, I am observing a lot of counters, however this makes the %running (pcnt-running in previous json output) very low: all of my measurements are bellow 50%, and most of them are bellow 25%.

Increasing the runtime does not improve %running. Here we notice that even with a low amount of counters, we are still around 30%, and even worse with a longer runtime:

andrew@rammus ~/d/d/build (main) [nix] > perf stat \
                                                 -e llc-load-misses,llc-loads,llc-store-misses,llc-stores \
                                                 -o perf-output.csv \
                                                 -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062

andrew@rammus ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 11:31:19 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

            77,435      cpu_atom/llc-load-misses/                                               (30.99%)
        10,604,089      cpu_core/llc-load-misses/                                               (35.22%)
        67,754,093      cpu_atom/llc-loads/                                                     (30.96%)
        32,364,585      cpu_core/llc-loads/                                                     (35.25%)
            76,527      cpu_atom/llc-store-misses/                                              (30.92%)
        18,752,097      cpu_core/llc-store-misses/                                              (35.27%)
        74,639,980      cpu_atom/llc-stores/                                                    (30.90%)
        31,887,623      cpu_core/llc-stores/                                                    (35.23%)

       3.238076080 seconds time elapsed

      69.854707000 seconds user
       1.305123000 seconds sys


andrew@rammus ~/d/d/build (main) [nix] > perf stat \
                                                 -e llc-load-misses,llc-loads,llc-store-misses,llc-stores \
                                                 -o perf-output.csv \
                                                 -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 10
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 10
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
Epoch: 2, Loss: 1.046023, Accuracy: 0.691406
Epoch: 3, Loss: 0.849502, Accuracy: 0.736328
Epoch: 4, Loss: 0.718872, Accuracy: 0.754883
Epoch: 5, Loss: 0.632481, Accuracy: 0.775391
Epoch: 6, Loss: 0.582991, Accuracy: 0.794922
Epoch: 7, Loss: 0.549984, Accuracy: 0.815430
Epoch: 8, Loss: 0.525650, Accuracy: 0.823242
Epoch: 9, Loss: 0.504946, Accuracy: 0.834961
Epoch: 10, Loss: 0.486858, Accuracy: 0.844727
andrew@rammus ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 11:31:34 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 10':

         1,252,115      cpu_atom/llc-load-misses/                                               (22.55%)
       115,992,537      cpu_core/llc-load-misses/                                               (29.31%)
       937,299,440      cpu_atom/llc-loads/                                                     (22.55%)
       358,330,039      cpu_core/llc-loads/                                                     (29.31%)
         1,211,776      cpu_atom/llc-store-misses/                                              (22.54%)
        37,874,183      cpu_core/llc-store-misses/                                              (29.32%)
       957,699,065      cpu_atom/llc-stores/                                                    (22.55%)
       184,955,998      cpu_core/llc-stores/                                                    (29.32%)

      27.160037746 seconds time elapsed

     682.447501000 seconds user
       4.728311000 seconds sys

Reducing the number of counters per run effectively improves %running, thus accuracy:

andrew@rammus ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 11:35:54 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

            78,454      cpu_atom/llc-load-misses/                                               (63.30%)
         9,574,620      cpu_core/llc-load-misses/                                               (76.99%)
        66,243,744      cpu_atom/llc-loads/                                                     (63.30%)
        30,441,333      cpu_core/llc-loads/                                                     (76.99%)

       3.293131663 seconds time elapsed

      70.015803000 seconds user
       1.501149000 seconds sys

Grouping counters

On way to maximize %running is to group the relevant counters together:

llc-load-misses,llc-loads
~60% running
llc-store-misses,llc-stores
~50% running, doing them one by one doesn't improve it.
l1-dcache-loads,l1-dcache-load-misses,l1-dcache-stores
~55% running
l1-icache-loads,l1-icache-load-misses
~55% running
branch-instructions,branch-misses,branch-load-misses,branch-loads,cycles,instructions,ref-cycles
~55% running, seems like it's fine to do all of them at the same time.
context-switches,cpu-migrations,faults,major-faults,minor-faults
N/A, these are software counters so we have no problems.

Note that those numbers have been computed on an hybrid-cpu architecture. Predicting them is way easier on non-hybrid architectures.

On non-hybrid architectures

On this cpu (Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz), it seems like we only have 2 slots to store counters. Notice how using 4 counters reduces %running to half, and 3 counters to two third.

andrew@poppy ~/d/d/build (main)> perf stat \
                                          -e llc-load-misses,llc-loads \
                                          -o perf-output.csv \
                                          -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@poppy ~/d/d/build (main)> cat perf-output.csv
# started on Fri Sep  4 11:05:18 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

        22,064,123      llc-load-misses                                                       
        85,037,332      llc-loads                                                             

      16.856295657 seconds time elapsed

      65.240831000 seconds user
       0.698790000 seconds sys


andrew@poppy ~/d/d/build (main)> perf stat \
                                          -e llc-load-misses,llc-loads,llc-store-misses \
                                          -o perf-output.csv \
                                          -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@poppy ~/d/d/build (main)> cat perf-output.csv
# started on Fri Sep  4 11:05:49 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

        22,280,763      llc-load-misses                                                         (66.67%)
        86,269,772      llc-loads                                                               (66.67%)
        54,582,036      llc-store-misses                                                        (66.66%)

      17.025249875 seconds time elapsed

      65.750288000 seconds user
       0.701732000 seconds sys


andrew@poppy ~/d/d/build (main)> perf stat \
                                          -e llc-load-misses,llc-loads,llc-store-misses,llc-stores \
                                          -o perf-output.csv \
                                          -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@poppy ~/d/d/build (main)> cat perf-output.csv
# started on Fri Sep  4 11:06:26 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

        22,155,268      llc-load-misses                                                         (49.99%)
        86,168,829      llc-loads                                                               (49.99%)
        54,204,435      llc-store-misses                                                        (50.01%)
        75,783,023      llc-stores                                                              (50.01%)

      16.993879135 seconds time elapsed

      65.549849000 seconds user
       0.700088000 seconds sys

On some machine, you can sometimes run a whole group of counters together, on other it might be less accurate. For example we saw that on poppy I couldn't scheduler more than two counters for cache, but for branch prediction, cycle and instructions it seems fine:

andrew@poppy ~/d/d/build (main)> perf stat \
                                          -e branch-instructions,branch-misses,branch-load-misses,branch-loads,cycles,instructions,ref-cycles \
                                          -o perf-output.csv \
                                          -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@poppy ~/d/d/build (main)> cat perf-output.csv
# started on Fri Sep  4 11:32:17 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

   214,397,785,806      branch-instructions                                                   
       115,759,216      branch-misses                                                         
       115,759,216      branch-load-misses                                                    
   214,397,785,818      branch-loads                                                          
   237,704,805,714      cycles                                                                
   916,301,184,546      instructions                                                          
   225,426,995,668      ref-cycles                                                            

      16.876608090 seconds time elapsed

      65.332454000 seconds user
       0.684730000 seconds sys

However, on Malphite I don't get a perfect %running:

andrew@malphite ~/d/d/build (main) [nix] > perf stat \
                                                    -e branch-instructions,branch-misses,branch-load-misses,branch-loads,cycles,instructions,ref-cycles \
                                                    -o perf-output.csv \
                                                    -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@malphite ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 11:32:18 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

   214,369,684,174      branch-instructions                                                     (85.70%)
        75,235,121      branch-misses                                                           (85.71%)
        75,272,307      branch-load-misses                                                      (85.72%)
   214,369,224,435      branch-loads                                                            (85.72%)
   243,208,079,371      cycles                                                                  (57.16%)
   915,568,862,761      instructions                                                            (71.44%)
   230,871,416,276      ref-cycles                                                              (85.72%)

      15.562529957 seconds time elapsed

      61.648745000 seconds user
       0.709802000 seconds sys

But are still around ~80% which is fine. Simply removing one counter like cycles is enough to obtain 100%.

Hardware dependent counters

Some counters might not be available depending on hardware. Furthermore, they might be available on some cores, but not others.

CPU topology on my Rammus server with hybrid architecture.

As an example on my server Rammus - Intel(R) Core(TM) i7-14700K - I found out that l1-dcache-load-misses is only accessible for P-cores, not E-cores:

# started on Fri Sep  4 11:16:29 2026


 Performance counter stats for 'system wide':

CPU16       7,369,348,475      cpu_atom/l1-dcache-loads/                                             
CPU17       7,427,487,733      cpu_atom/l1-dcache-loads/                                             
CPU18       7,357,191,997      cpu_atom/l1-dcache-loads/                                             
CPU19       7,936,203,849      cpu_atom/l1-dcache-loads/                                             
CPU20       7,401,167,283      cpu_atom/l1-dcache-loads/                                             
CPU21       7,524,298,197      cpu_atom/l1-dcache-loads/                                             
CPU22       7,352,901,924      cpu_atom/l1-dcache-loads/                                             
CPU23       7,531,354,529      cpu_atom/l1-dcache-loads/                                             
CPU24       7,570,947,805      cpu_atom/l1-dcache-loads/                                             
CPU25       6,481,405,530      cpu_atom/l1-dcache-loads/                                             
CPU26       7,660,836,833      cpu_atom/l1-dcache-loads/                                             
CPU27       7,607,228,779      cpu_atom/l1-dcache-loads/                                             
CPU0        8,797,259,917      cpu_core/l1-dcache-loads/                                             
CPU1        8,768,229,510      cpu_core/l1-dcache-loads/                                             
CPU2        8,859,848,700      cpu_core/l1-dcache-loads/                                             
CPU3        8,796,043,566      cpu_core/l1-dcache-loads/                                             
CPU4        8,504,141,413      cpu_core/l1-dcache-loads/                                             
CPU5        8,542,091,672      cpu_core/l1-dcache-loads/                                             
CPU6        7,323,529,780      cpu_core/l1-dcache-loads/                                             
CPU7        7,373,871,153      cpu_core/l1-dcache-loads/                                             
CPU8        7,130,903,277      cpu_core/l1-dcache-loads/                                             
CPU9        7,167,522,727      cpu_core/l1-dcache-loads/                                             
CPU10       7,515,079,847      cpu_core/l1-dcache-loads/                                             
CPU11       7,794,804,919      cpu_core/l1-dcache-loads/                                             
CPU12       8,576,795,744      cpu_core/l1-dcache-loads/                                             
CPU13       8,556,331,444      cpu_core/l1-dcache-loads/                                             
CPU14       8,552,458,452      cpu_core/l1-dcache-loads/                                             
CPU15       8,621,082,982      cpu_core/l1-dcache-loads/                                             
CPU16      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU17      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU18      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU19      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU20      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU21      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU22      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU23      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU24      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU25      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU26      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU27      not supported       cpu_atom/l1-dcache-load-misses/                                       
CPU0           20,254,979      cpu_core/l1-dcache-load-misses/                                       
CPU1           19,526,030      cpu_core/l1-dcache-load-misses/                                       
CPU2           18,942,355      cpu_core/l1-dcache-load-misses/                                       
CPU3           18,479,730      cpu_core/l1-dcache-load-misses/                                       
CPU4           18,756,333      cpu_core/l1-dcache-load-misses/                                       
CPU5           19,068,444      cpu_core/l1-dcache-load-misses/                                       
CPU6           17,457,224      cpu_core/l1-dcache-load-misses/                                       
CPU7           19,975,124      cpu_core/l1-dcache-load-misses/                                       
CPU8           16,307,837      cpu_core/l1-dcache-load-misses/                                       
CPU9           18,127,154      cpu_core/l1-dcache-load-misses/                                       
CPU10          17,312,871      cpu_core/l1-dcache-load-misses/                                       
CPU11          18,617,901      cpu_core/l1-dcache-load-misses/                                       
CPU12          18,021,039      cpu_core/l1-dcache-load-misses/                                       
CPU13          17,992,439      cpu_core/l1-dcache-load-misses/                                       
CPU14          17,859,068      cpu_core/l1-dcache-load-misses/                                       
CPU15          17,795,354      cpu_core/l1-dcache-load-misses/

Looking at the topology, notice how CPU (or PU) 16 to 27 is inside an efficiency core. Weirdly though, they all have their own l1d caches.

I encountered no problem accessing last level cache counters (llc-load, llc-store...).

On both Poppy and Malphite l1-icache-loads is not supported.

andrew@poppy ~/d/d/build (main)> perf stat \
                                          -e l1-dcache-loads,l1-dcache-stores,l1-icache-load-misses,l1-icache-loads \
                                          -o perf-output.csv \
                                          -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@poppy ~/d/d/build (main)> cat perf-output.csv
# started on Fri Sep  4 11:28:29 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

   215,149,776,802      l1-dcache-loads                                                       
    47,638,573,567      l1-dcache-stores                                                      
       252,922,125      l1-icache-load-misses                                                 
    not supported       l1-icache-loads                                                       

      16.897517023 seconds time elapsed

      65.377596000 seconds user
       0.686967000 seconds sys

andrew@malphite ~/d/d/build (main) [nix] > perf stat \
                                                    -e l1-dcache-loads,l1-dcache-stores,l1-icache-load-misses,l1-icache-loads \
                                                    -o perf-output.csv \
                                                    -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-debug (-O0), and is thus not optimized
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-spinlock-check, which slows down a bit
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
cEpoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@malphite ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 11:14:49 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

   215,059,724,258      l1-dcache-loads                                                       
    47,573,570,154      l1-dcache-stores                                                      
       268,487,196      l1-icache-load-misses                                                 
    not supported       l1-icache-loads                                                       

      15.488189731 seconds time elapsed

      61.605655000 seconds user
       0.724745000 seconds sys

More on Floating Point counters

What I've found is that all my servers don't support fp_flops_retired.fp32/64/128/256 compared to my laptop's cpu (Intel(R) Core(TM) Ultra 5 125U). Investigating further I've discovered alternative counters such as:

fp_arith_inst_retired.scalar
[Counts once for most SIMD scalar computational floating-point instructions retired. Counts twice for DPP and FM(N)ADD/SUB instructions retired. Unit: cpu]
fp_arith_inst_retired.scalar_double
[Counts once for most SIMD scalar computational double precision floating-point instructions retired. Counts twice for DPP and FM(N)ADD/SUB instructions retired. Unit: cpu]
fp_arith_inst_retired.scalar_single
[Counts once for most SIMD scalar computational single precision floating-point instructions retired. Counts twice for DPP and FM(N)ADD/SUB instructions retired. Unit: cpu]
fp_arith_inst_retired.vector
[Number of any Vector retired FP arithmetic instructions. Unit: cpu]
fp_assist.any
[Cycles with any input/output SSE or FP assist. Unit: cpu]

Let's compare the FLOPS numbers with my laptop and one of my server:

andrew@malphite ~/d/d/build (main) [nix] > perf stat \
                                                    -e fp_arith_inst_retired.scalar_double \
                                                    -o perf-output.csv \
                                                    -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-debug (-O0), and is thus not optimized
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-spinlock-check, which slows down a bit
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@malphite ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 12:13:21 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

    36,376,683,792      fp_arith_inst_retired.scalar_double                                      

      15.488151098 seconds time elapsed

      61.634755000 seconds user
       0.721711000 seconds sys

Here, only fp_arith_inst_retired.scalar_double is available and seems to match my use case.

andrew@senna ~/l/i/d/d/build (main) [nix] > perf stat \
                                                    -e fp_arith_inst_retired.scalar_double,fp_flops_retired.fp64 \
                                                    -o perf-output.csv \
                                                    -- ./basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-debug (-O0), and is thus not optimized
[starpu][starpu_initialize] Warning: StarPU was configured with --enable-spinlock-check, which slows down a bit
[starpu][_starpu_init_topology] Warning: there are several kinds of CPU on this system. For now StarPU assumes all CPU are equal
[starpu][_starpu_initialize_workers_bindid] Warning: hwloc reported 14 logical CPUs for 12 cores, this is not homogeneous, will assume 1 logical CPUs per core
Epoch: 0, Loss: 0.000000, Accuracy: 0.000000
num samples: 1024, batch size: 64, num epochs: 1
Epoch: 1, Loss: 16.501261, Accuracy: 0.414062
andrew@senna ~/l/i/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 14:13:33 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

    24 965 481 525      cpu_core/fp_arith_inst_retired.scalar_double/                                        (36,26%)
    38 310 749 304      cpu_atom/fp_flops_retired.fp64/                                         (71,53%)

       6,789030214 seconds time elapsed

      83,001870000 seconds user
       1,118885000 seconds sys

In comparison on my laptop I can use both counters fp_arith_inst_retired.scalar_double and fp_flops_retired.fp64.

So we get 36,376,683,792 (scalar_double) on the server, and 24 965 481 525 (E-cores) + 38 310 749 304 (P-cores) flops on the laptop. It seems that the number retrieved on the server matches the number of FLOPS achieved by the P-cores of my laptop. But then the new counter scalar double returns new values specifically for the E-cores. Seems like we found the missing floating point operations that were occuring on the E-cores.

However, this also indicates that the server might miss some operations, as summing E-cores + P-cores FLOPS roughly gives us 60 B operations compared to 36 B on the server. As this is the exact same workload, this seems suspicious.

andrew@malphite ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 12:25:32 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

    36,375,677,050      fp_arith_inst_retired.scalar                                          
    36,376,145,302      fp_arith_inst_retired.scalar_double                                      
           468,312      fp_arith_inst_retired.scalar_single                                      
           468,282      fp_arith_inst_retired.vector                                          

      15.494357105 seconds time elapsed

      61.591024000 seconds user
       0.771582000 seconds sys

Adding more counters does not seem to solve the mystery: fp_arith_inst_retired.scalar seems to regroups both scalar_double and scalar_single.

andrew@malphite ~/d/d/build (main) [nix] > cat perf-output.csv
# started on Fri Sep  4 12:30:08 2026


 Performance counter stats for './basic_cnn generic ../../datasets/big-fashion/234_312 1024 64 1':

    36,357,303,158      fp_arith_inst_retired.scalar_double                                        (66.67%)
           350,832      fp_arith_inst_retired.128b_packed_double                                        (66.66%)
           350,641      fp_arith_inst_retired.128b_packed_single                                        (66.66%)
           355,103      fp_arith_inst_retired.256b_packed_double                                        (66.67%)
           351,535      fp_arith_inst_retired.256b_packed_single                                        (66.67%)
           350,107      fp_arith_inst_retired.4_flops                                           (66.67%)

      15.542216242 seconds time elapsed

      61.595658000 seconds user
       0.701391000 seconds sys

No luck with other counters.

Doing the same on Rammus, we obtain 30,767,791,568 on P-cores again. But as this server has an hybrid architecture, I still don't have the FLOPS for E-cores.

So we have two problems:

On Rammus (Server + Hybrid CPU), we can't get FLOPS on E-cores
On other servers, we it seems that the FLOPS count is off

< Previous post | Next post >