Reading input file....
Matrix is Float
Total number of nonzeros expected across all processors is 99199551
File is 2627728707 bytes
Total number of entries read across all processors is 99199551
As a whole: 5154859 rows and 5154859 columns and 99199551 nonzeros
Reader took 11.4458 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 18.0823 seconds
Input load balance: 1.10405
Input matrix read : time 29.5282
Computing restriction matrix 
As a whole: 5154859 rows and 5154859 columns and 94044692 nonzeros
As a whole, mis2 has: 138403 nonzeros and length 5154859
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 5154859 nonzeros and length 5154859
Rop final (before normalization)... As a whole: 5154859 rows and 138403 columns and 5154859 nonzeros
Load balance (before): 1.05588
Load balance (after): 1.00163
Restriction Op computed : time 15.1016
Matrix split and distributed along layers: time 2.49235 seconds
Matrix split and distributed along layers: time 0.407843 seconds
Matrix split and distributed along layers: time 0.09463 seconds
Matrix split and distributed along layers: time 0.021589 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.154538     0.497682     0.541442   0.531020     0.374685     0.031594     0.400866   2.531827
   3    3     8      6   0.344135     0.559884     0.544298   0.384844     0.383211     0.038084     0.251073   2.505529
Computing RTA
   3    3     8      6   0.227040     0.023807     0.137540   0.011888     0.017208     0.001923     0.046625   0.466031
   3    3     8      6   0.082674     0.019748     0.135528   0.012327     0.015194     0.001456     0.072010   0.338937
Computing RTAR
   3    3     8      6   0.052592     0.007557     0.035942   0.005313     0.005411     0.000283     0.025990   0.133088
   3    3     8      6   0.060852     0.005865     0.035885   0.005303     0.005396     0.000393     0.016444   0.130138
----------------------------
 nnz(A)= 99199551
 nnz(R)= 5154859
 nnz(A^2)= 929023247
 nnz(RTA)= 46996564
 nnz(RTAR)= 17378358
----------------------------
Application 15487715 resources: utime ~4601s, stime ~352s, Rss ~2741960, inblocks ~5384789, outblocks ~514084
Reading input file....
Matrix is Boolean
Matrix is symmetric
Total number of nonzeros expected across all processors is 50331601
File is 839313191 bytes
Total number of entries read across all processors is 50331601
As a whole: 16777216 rows and 16777216 columns and 100663202 nonzeros
Reader took 8.02891 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 15.6843 seconds
Input load balance: 1.00031
Input matrix read : time 23.8136
Computing restriction matrix 
As a whole: 16777216 rows and 16777216 columns and 100663202 nonzeros
As a whole, mis2 has: 1648797 nonzeros and length 16777216
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 16777216 nonzeros and length 16777216
Rop final (before normalization)... As a whole: 16777216 rows and 1648797 columns and 16777216 nonzeros
Load balance (before): 1.19799
Load balance (after): 1.00083
Restriction Op computed : time 24.1999
Matrix split and distributed along layers: time 2.41257 seconds
Matrix split and distributed along layers: time 0.418175 seconds
Matrix split and distributed along layers: time 0.328849 seconds
Matrix split and distributed along layers: time 0.0677769 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.106599     0.226021     0.453010   0.152366     0.136134     0.014628     0.386212   1.474970
   3    3     8      6   0.228574     0.235980     0.474578   0.116510     0.131326     0.018628     0.248976   1.454572
Computing RTA
   3    3     8      6   0.192693     0.025200     0.196110   0.014517     0.015531     0.001292     0.038454   0.483797
   3    3     8      6   0.137585     0.020257     0.186906   0.015563     0.016072     0.002514     0.052737   0.431634
Computing RTAR
   3    3     8      6   0.047806     0.014134     0.081782   0.007692     0.007551     0.000446     0.038556   0.197967
   3    3     8      6   0.044384     0.010459     0.082722   0.007652     0.007512     0.000500     0.047017   0.200246
----------------------------
 nnz(A)= 100663202
 nnz(R)= 16777216
 nnz(A^2)= 347322258
 nnz(RTA)= 41187588
 nnz(RTAR)= 16344371
----------------------------
Application 15487720 resources: utime ~4527s, stime ~203s, Rss ~1685208, inblocks ~1904573, outblocks ~513625
Reading input file....
Matrix is Float
Matrix is symmetric
Total number of nonzeros expected across all processors is 45204422
File is 1581474573 bytes
Total number of entries read across all processors is 45204422
As a whole: 1102824 rows and 1102824 columns and 89306020 nonzeros
Reader took 7.93081 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 8.55082 seconds
Input load balance: 1.02585
Input matrix read : time 16.4898
Computing restriction matrix 
As a whole: 1102824 rows and 1102824 columns and 88203196 nonzeros
As a whole, mis2 has: 6223 nonzeros and length 1102824
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 1102824 nonzeros and length 1102824
Rop final (before normalization)... As a whole: 1102824 rows and 6223 columns and 1102824 nonzeros
Load balance (before): 1.01283
Load balance (after): 1.00956
Restriction Op computed : time 8.86153
Matrix split and distributed along layers: time 2.03788 seconds
Matrix split and distributed along layers: time 0.249981 seconds
Matrix split and distributed along layers: time 0.0188949 seconds
Matrix split and distributed along layers: time 0.00400877 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.179938     0.796585     1.950771   1.353795     0.498976     0.023324     0.741996   5.545385
   3    3     8      6   0.442028     1.038583     1.949493   0.783112     0.475822     0.025923     0.579984   5.294945
Computing RTA
   3    3     8      6   0.295675     0.009649     0.094280   0.005882     0.003106     0.000317     0.042272   0.451181
   3    3     8      6   0.090082     0.005595     0.094344   0.005376     0.002949     0.000339     0.016154   0.214839
Computing RTAR
   3    3     8      6   0.006106     0.000527     0.006921   0.000274     0.000188     0.000014     0.003973   0.018004
   3    3     8      6   0.006555     0.000459     0.006686   0.000224     0.000182     0.000015     0.000785   0.014906
----------------------------
 nnz(A)= 89306020
 nnz(R)= 1102824
 nnz(A^2)= 688649400
 nnz(RTA)= 4321899
 nnz(RTAR)= 100891
----------------------------
Application 15487725 resources: utime ~4651s, stime ~511s, Rss ~4027292, inblocks ~3344270, outblocks ~514585
Reading input file....
Matrix is Float
Matrix is symmetric
Total number of nonzeros expected across all processors is 23737339
File is 663855459 bytes
Total number of entries read across all processors is 23737339
As a whole: 952203 rows and 952203 columns and 46522475 nonzeros
Reader took 5.01395 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 5.94961 seconds
Input load balance: 1.04136
Input matrix read : time 10.9687
Computing restriction matrix 
As a whole: 952203 rows and 952203 columns and 45570272 nonzeros
As a whole, mis2 has: 12435 nonzeros and length 952203
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 952203 nonzeros and length 952203
Rop final (before normalization)... As a whole: 952203 rows and 12435 columns and 952203 nonzeros
Load balance (before): 1.03296
Load balance (after): 1.00326
Restriction Op computed : time 4.79208
Matrix split and distributed along layers: time 1.05584 seconds
Matrix split and distributed along layers: time 0.136999 seconds
Matrix split and distributed along layers: time 0.0156751 seconds
Matrix split and distributed along layers: time 0.00366688 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.083513     0.334178     0.479979   0.303903     0.146320     0.004019     0.223481   1.575393
   3    3     8      6   0.168149     0.280262     0.476013   0.248056     0.127036     0.006780     0.124094   1.430391
Computing RTA
   3    3     8      6   0.096085     0.004308     0.051706   0.003042     0.001983     0.000062     0.017621   0.174806
   3    3     8      6   0.037394     0.003616     0.051010   0.003097     0.002155     0.000116     0.018313   0.115701
Computing RTAR
   3    3     8      6   0.004833     0.000728     0.002943   0.000231     0.000199     0.000018     0.004349   0.013301
   3    3     8      6   0.003002     0.000510     0.002778   0.000243     0.000198     0.000017     0.004177   0.010925
----------------------------
 nnz(A)= 46522475
 nnz(R)= 952203
 nnz(A^2)= 145422935
 nnz(RTA)= 2312261
 nnz(RTAR)= 375058
----------------------------
Application 15487730 resources: utime ~1953s, stime ~165s, Rss ~1308984, inblocks ~1542054, outblocks ~512693
Reading input file....
Matrix is Float
Total number of nonzeros expected across all processors is 283073458
File is 10095089342 bytes
Total number of entries read across all processors is 283073458
As a whole: 2017169 rows and 2017169 columns and 283073458 nonzeros
Reader took 34.1409 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 39.636 seconds
Input load balance: 1.01555
Input matrix read : time 73.8042
Computing restriction matrix 
As a whole: 2017169 rows and 2017169 columns and 324715138 nonzeros
As a whole, mis2 has: 7012 nonzeros and length 2017169
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 2017169 nonzeros and length 2017169
Rop final (before normalization)... As a whole: 2017169 rows and 7012 columns and 2017169 nonzeros
Load balance (before): 1.01914
Load balance (after): 1.01622
Restriction Op computed : time 34.738
Matrix split and distributed along layers: time 6.84686 seconds
Matrix split and distributed along layers: time 0.979838 seconds
Matrix split and distributed along layers: time 0.0321388 seconds
Matrix split and distributed along layers: time 0.0059979 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.811093     2.960169    11.457375   5.714775     1.727956     0.065309     2.029952  24.766629
   3    3     8      6   1.771822     2.904427    11.649361   4.400615     1.523979     0.060512     1.987142  24.297858
Computing RTA
   3    3     8      6   1.090677     0.020776     0.297909   0.016918     0.008040     0.000273     0.079894   1.514487
   3    3     8      6   0.182864     0.024239     0.294740   0.013069     0.007270     0.000407     0.099634   0.622223
Computing RTAR
   3    3     8      6   0.010164     0.001315     0.009405   0.000368     0.000325     0.000036     0.014540   0.036153
   3    3     8      6   0.014092     0.001303     0.008929   0.000318     0.000324     0.000039     0.004478   0.029483
----------------------------
 nnz(A)= 283073458
 nnz(R)= 2017169
 nnz(A^2)= 1768066720
 nnz(RTA)= 10232256
 nnz(RTAR)= 1396638
----------------------------
Application 15487734 resources: utime ~21606s, stime ~1891s, Rss ~15511788, inblocks ~19983657, outblocks ~522220
Reading input file....
Matrix is Float
Matrix is symmetric
Total number of nonzeros expected across all processors is 14506196
File is 447387587 bytes
Total number of entries read across all processors is 14506196
As a whole: 45101 rows and 45101 columns and 28967291 nonzeros
Reader took 2.86716 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Permutation took 2.66985 seconds
Input load balance: 1.00833
Input matrix read : time 5.54398
Computing restriction matrix 
As a whole: 45101 rows and 45101 columns and 28922190 nonzeros
As a whole, mis2 has: 4168 nonzeros and length 45101
Creating a new MPI data type for 10VertexTypeIlE
As a whole, mis2neighUnion has: 45101 nonzeros and length 45101
Rop final (before normalization)... As a whole: 45101 rows and 4168 columns and 45101 nonzeros
Load balance (before): 1.61178
Load balance (after): 1.1604
Restriction Op computed : time 1.75634
Matrix split and distributed along layers: time 0.550482 seconds
Matrix split and distributed along layers: time 0.0701082 seconds
Matrix split and distributed along layers: time 0.00063014 seconds
Matrix split and distributed along layers: time 0.000167131 seconds

 Processor Grid (row x col x layers x threads): 3x3x8x6 
 prow pcol layer thread comm_bcast   comm_scatter comp_summa comp_merge  comp_scatter  comp_result     other      total
Computing A square
   3    3     8      6   0.147921     1.011236    15.125896   0.933515     0.418462     0.017054     1.777651  19.431735
   3    3     8      6   0.398752     0.882370    15.113538   0.835596     0.411898     0.015607     1.652304  19.310065
Computing RTA
   3    3     8      6   0.218502     0.002952     0.026770   0.001461     0.001603     0.000076     0.028974   0.280338
   3    3     8      6   0.016541     0.002258     0.026729   0.001487     0.001559     0.000076     0.008857   0.057506
Computing RTAR
   3    3     8      6   0.005476     0.000576     0.002675   0.000155     0.000217     0.000018     0.001251   0.010368
   3    3     8      6   0.003689     0.000322     0.002529   0.000158     0.000219     0.000017     0.002081   0.009015
----------------------------
 nnz(A)= 28967291
 nnz(R)= 45101
 nnz(A^2)= 482594045
 nnz(RTA)= 2960213
 nnz(RTAR)= 415005
----------------------------
Application 15487756 resources: utime ~14477s, stime ~443s, Rss ~3790824, inblocks ~1124482, outblocks ~513574
Reading input file....
Matrix is Boolean
Total number of nonzeros expected across all processors is 1150725436
File is 20241588318 bytes
Total number of entries read across all processors is 1150725436
As a whole: 41291594 rows and 41291594 columns and 1150725436 nonzeros
Reader took 110.855 seconds
Generating random permutation vector.
Perfoming random permuation of matrix.
Rank 56 [Fri Sep  4 09:36:18 2015] [c7-3c0s8n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack:
MPIR_CRAY_Bcast_Tree(352): message sizes do not match across processes in the collective routine: Received -32766 but expected -1985190112
Rank 8 [Fri Sep  4 09:36:18 2015] [c7-3c0s5n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack:
MPIR_CRAY_Bcast_Tree(352): message sizes do not match across processes in the collective routine: Received -32766 but expected -1985190112
_pmiu_daemon(SIGCHLD): [NID 05984] [c7-3c0s8n0] [Fri Sep  4 09:36:18 2015] PE RANK 56 exit signal Aborted
[NID 05984] 2015-09-04 09:36:18 Apid 15487763: initiated application termination
Application 15487763 exit codes: 134
Application 15487763 exit signals: Killed
Application 15487763 resources: utime ~208s, stime ~15s, Rss ~8852504, inblocks ~4588595, outblocks ~511015

 + --------------------------------------------------------------------------
 +        Job name: Rop_combine_512
 +          Job Id: 3533014.edique02
 +          System: edison
 +     Queued Time: Fri Sep  4 09:24:50 2015
 +      Start Time: Fri Sep  4 09:25:17 2015
 + Completion Time: Fri Sep  4 09:36:21 2015
 +            User: azad
 +        MOM Host: nid02050
 +           Queue: debug
 +  Req. Resources: mppnodect=22,mppnppn=24,mppwidth=512,walltime=00:30:00
 +  Used Resources: cput=00:00:02,energy_used=0,mem=10820kb,vmem=159760kb,walltime=00:11:06
 +     Acct String: m1982
 +   PBS_O_WORKDIR: /global/u1/a/azad/combinatorial-blas-2.0/CombBLAS/3DSpGEMM/edison_exp/combine
 +     Submit Args: batchRop_combine_512
 + --------------------------------------------------------------------------

