KoolReport's Forum

Official Support Area, Q&As, Discussions, Suggestions and Bug reports.
Forum's Guidelines

Sorting in PivotExtract #1815

Closed Venom opened this topic on on Jan 4, 2021 - 10 comments

Venom commented on Jan 4, 2021

I have a set of data retrieved from MySQL and would like to pipe into the Pivot and extract the data set to a table.

In the Pivot Extract Sorting, I have specified the following option

        "row" => array(
            "parent" => array(
            ),
        "sort" => array(
            'confirm_amount - sum' => 'desc',
            ),
        ),

However, the table did not sort by confirm_amount. It just sorted by custname column.

Table outcome ` use \koolreport\processes\ColumnMeta; use \koolreport\pivot\processes\PivotExtract;

class CustomersYears extends koolreport\KoolReport {

function settings()
{
    return array(
        "dataSources"=>array(
            "aimfs"=>array(
                "connectionString"=>"xxx",
                "username" =>"xx",
                "password" =>"xxx",
                "chartset"=>"utf8"
            )
        )
    );
}
function setup()
{
    $this->src('aimfs')
    ->query('select * from quote')
    ->pipe(new \koolreport\pivot\processes\PivotSQL([
        "column" => "signed_month",
        "row" => "custname",
        "aggregates"=>array(
            "sum"=>"confirm_amount",
             "count"=>"confirm_amount"
        ),
    ]))
    ->pipe(new ColumnMeta(array(
        "confirm_amount - sum"=>array(
            'type' => 'number',
            "prefix" => "$",
        ),
    )))
    ->saveTo($node2);
    $node2->pipe($this->dataStore('sales')); 

    $node2->pipe(new PivotExtract(array(
        "row" => array(
            "parent" => array(
            
             
            ),
        "sort" => array(
            'confirm_amount - sum' => 'desc',
            ),
        ),
        "column" => array(
            "parent" => array(
             
            
          
            ),
          "sort" => array(
                'signed_month' => function($a, $b) {
                    return (int)$a < (int)$b;
                },
                
                
            ),
        ),
        "measures"=>array(
            "confirm_amount - sum", 
            "confirm_amount - count", 
        ),
    )))
    ->pipe($this->dataStore('pivotData'));

}

} `

View file

use \koolreport\widgets\koolphp\Table;
use \koolreport\widgets\google\ColumnChart;

@session_start();
$CustomersYears = new CustomersYears;
$CustomersYears->run();


        Table::create(array(
            "dataStore"=>$CustomersYears->dataStore('pivotData'),
        ));
David Winterburn commented on Jan 4, 2021

Please try to print $CustomersYears->dataStore('pivotData')->data() to see its row order. In case the order is still not right please try to apply the Sort process right after the PivotExact:

    $node2->pipe(new PivotExtract(array(
        ...
    )))
    ->pipe(new Sort(array(
        "{{all}} | confirm_amount - sum" => "desc"
    )))
    ...

Let us know the result. Thanks!

Venom commented on Jan 4, 2021

Below is the data() output of dataStore('pivotData'). array (size=51) 0 =>

array (size=9)
  'custname' => string 'A1' (length=41)
  '10 | confirm_amount - sum' => int 0
  '10 | confirm_amount - count' => int 0
  '11 | confirm_amount - sum' => string '6000.00' (length=7)
  '11 | confirm_amount - count' => string '1' (length=1)
  '12 | confirm_amount - sum' => int 0
  '12 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => string '6000.00' (length=7)
  '{{all}} | confirm_amount - count' => string '1' (length=1)

1 =>

array (size=9)
  'custname' => string 'A2' (length=23)
  '10 | confirm_amount - sum' => int 0
  '10 | confirm_amount - count' => int 0
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '12 | confirm_amount - sum' => string '5530.00' (length=7)
  '12 | confirm_amount - count' => string '1' (length=1)
  '{{all}} | confirm_amount - sum' => string '5530.00' (length=7)
  '{{all}} | confirm_amount - count' => string '1' (length=1)

...... . . .

array (size=9)
  'custname' => string '{{all}}' (length=7)
  '10 | confirm_amount - sum' => string '900.00' (length=6)
  '10 | confirm_amount - count' => string '1' (length=1)
  '11 | confirm_amount - sum' => string '480171.33' (length=9)
  '11 | confirm_amount - count' => string '62' (length=2)
  '12 | confirm_amount - sum' => string '620231.80' (length=9)
  '12 | confirm_amount - count' => string '13' (length=2)
  '{{all}} | confirm_amount - sum' => string '1101303.13' (length=10)
  '{{all}} | confirm_amount - count' => string '76' (length=2)

After applying the sort process to the pivotExtract, The Sorting for "{{all}} | confirm_amount - sum" did work, however, is it possible to put the {{all}} row to the bottom?

Sorting Result

David Winterburn commented on Jan 5, 2021

Ok, it seems the PivotExtract's sort works correctly. It just confuses the data type of the sum column as string so the order is not correctly numerically. So if you change the column meta of the sum column to "type" => "number" I think PivotExtract would sort the extracted table fine.

On the second question about "{{all}}" row I think you could apply the Sort process with a custom function on "custName" column:

    ->pipe(new PivotExtract(...))
    ->pipe(new Sort(array(
        "{{all}} | confirm_amount - sum" => "desc"
    )))
    ->pipe(new Sort(array(
        "custName" => function($custName1, $custName2) {
            if ($custName1 === "{{all}}") return -1;
            if ($custName2 === "{{all}}") return 1; //swap these -1 and 1 if {{all}} ends up on top
            return 0; //if neither custName1 nor custName2 are "{{all}}", keep the current order
        }
    )))
Venom commented on Jan 5, 2021

I have specified the ColumnMeta to number for "confirm_amount - sum" as I showed above for output from the PivotSQL. Should I put the ColumnMeta process after pivotExtract?

  $this->src('xxx')
     ->query('select custname, confirm_amount, signed_month,signed_year from quote_header_view')

        ->pipe(new \koolreport\pivot\processes\PivotSQL([
            "column" => "signed_month",
            "row" => "custname",
            "totalName" => 'All',
            "aggregates"=>array(
                "sum"=>"confirm_amount",
                 "count"=>"confirm_amount"
            ),
        ]))
        ->pipe(new ColumnMeta(array(
            "confirm_amount - sum"=>array(
                'type' => 'number'  //Specify the column type to number
              
            ),
        )))
 
    
        ->pipe(new PivotExtract(array(
            "row" => array(
                "parent" => array(
                
                 
                ),
            "sort" => array(
                'confirm_amount - sum' => 'desc',
                ),
            ),
            "column" => array(
                "parent" => array(
                 
                
              
                ),
              "sort" => array(
                    'signed_month' => function($a, $b) {
                        return (int)$a < (int)$b;
                    },
                    
                    
                ),
            ),
            "measures"=>array(
                "confirm_amount - sum", 
                "confirm_amount - count", 
            ),
        )))
       
        ->pipe(new Sort(array(
            "{{all}} | confirm_amount - sum" => "desc"
        )))
         ->pipe(new Sort(array(
             "custname" => function($custName1, $custName2) {
                 if ($custName1 === "{{all}}") return 1;
                 if ($custName2 === "{{all}}") return -1; //swap these -1 and 1 if {{all}} ends up on top
                 return 0; //if neither custName1 nor custName2 are "{{all}}", keep the current order
             }
         )))
        ->pipe(new ColumnRename(array(
            "custname"=>"Client Name",
            "10 | confirm_amount - sum"=>" Oct Total Confirm Amount",
            "10 | confirm_amount - count"=>" Oct Total Confirm Count"
        )))
        ->pipe($this->dataStore('pivotData'));

    }
 

array (size=51) 0 =>

array (size=9)
  'Client Name' => string 'A1' (length=57)
  ' Oct Total Confirm Amount' => int 0
  ' Oct Total Confirm Count' => int 0
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '12 | confirm_amount - sum' => string '480.00' (length=6)
  '12 | confirm_amount - count' => string '1' (length=1)
  '{{all}} | confirm_amount - sum' => string '480.00' (length=6)
  '{{all}} | confirm_amount - count' => string '1' (length=1)

If I use the ArrayDataSource as datasource and pipe into the Pivot for PivotExtract. It looks perfect. I don't need to specific the column type at all.


class ArrayDataStore extends \koolreport\KoolReport
{
    public function settings()
    {
        return array(
            "dataSources"=>array(
                "array_example_datasource"=>array(
                    "class"=>'\koolreport\datasources\ArrayDataSource',
                    "dataFormat"=>"table", //Table data format
                    "data"=>array(
                        array("custname","confirm_amount","signed_month","signed_year"),
                        array("A1","8000","10","2020"),
                        array("A2","2000","10","2020"),
                        array("A3","3000","10","2020"),
                        array("A4","4000","11","2020"),
                        array("A5","5000","10","2020"),
                        array("A6","6000","10","2020"),
                        array("A7","7000","11","2020"),
                        array("A8","9000","10","2020"),
                        array("A9","2000","10","2020"),
                        array("A10","3000","10","2020"),
                        array("A11","4000","10","2020"),
                        array("A12","5000","11","2020"),
                        array("A12","8000","11","2020"),
                    )
                ),
            )
        );
    }

    protected function setup(){
 
        $this->src('array_example_datasource')
      

       ->pipe(new Pivot(array(
        "dimensions" => array(
            "row" => "custname",
            "column" => "signed_month"
        ),
        "aggregates"=>array(
            "sum"=>"confirm_amount",
             "count"=>"confirm_amount"
        )
        )))
        ->pipe(new PivotExtract(array(
            "row" => array(
                "parent" => array(
                
                 
                ),
            "sort" => array(
                'confirm_amount - sum' => 'asc',
                ),
            ),
            "column" => array(
                "parent" => array(
                 
                
              
                ),
              "sort" => array(
                    'signed_month' => function($a, $b) {
                        return (int)$a < (int)$b;
                    },
                    
                    
                ),
            ),
            
            "measures"=>array(
                "confirm_amount - sum", 
                "confirm_amount - count", 
            ),
        )))
     

        ->pipe($this->dataStore('pivotData'));
       
       }
}

array (size=13) 0 =>

array (size=7)
  'custname' => string 'A2' (length=2)
  '10 | confirm_amount - sum' => float 2000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 2000
  '{{all}} | confirm_amount - count' => int 1

1 =>

array (size=7)
  'custname' => string 'A9' (length=2)
  '10 | confirm_amount - sum' => float 2000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 2000
  '{{all}} | confirm_amount - count' => int 1

2 =>

array (size=7)
  'custname' => string 'A3' (length=2)
  '10 | confirm_amount - sum' => float 3000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 3000
  '{{all}} | confirm_amount - count' => int 1

3 =>

array (size=7)
  'custname' => string 'A10' (length=3)
  '10 | confirm_amount - sum' => float 3000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 3000
  '{{all}} | confirm_amount - count' => int 1

4 =>

array (size=7)
  'custname' => string 'A4' (length=2)
  '10 | confirm_amount - sum' => int 0
  '10 | confirm_amount - count' => int 0
  '11 | confirm_amount - sum' => float 4000
  '11 | confirm_amount - count' => int 1
  '{{all}} | confirm_amount - sum' => float 4000
  '{{all}} | confirm_amount - count' => int 1

5 =>

array (size=7)
  'custname' => string 'A11' (length=3)
  '10 | confirm_amount - sum' => float 4000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 4000
  '{{all}} | confirm_amount - count' => int 1

6 =>

array (size=7)
  'custname' => string 'A5' (length=2)
  '10 | confirm_amount - sum' => float 5000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 5000
  '{{all}} | confirm_amount - count' => int 1

7 =>

array (size=7)
  'custname' => string 'A6' (length=2)
  '10 | confirm_amount - sum' => float 6000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 6000
  '{{all}} | confirm_amount - count' => int 1

8 =>

array (size=7)
  'custname' => string 'A7' (length=2)
  '10 | confirm_amount - sum' => int 0
  '10 | confirm_amount - count' => int 0
  '11 | confirm_amount - sum' => float 7000
  '11 | confirm_amount - count' => int 1
  '{{all}} | confirm_amount - sum' => float 7000
  '{{all}} | confirm_amount - count' => int 1

9 =>

array (size=7)
  'custname' => string 'A1' (length=2)
  '10 | confirm_amount - sum' => float 8000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 8000
  '{{all}} | confirm_amount - count' => int 1

10 =>

array (size=7)
  'custname' => string 'A8' (length=2)
  '10 | confirm_amount - sum' => float 9000
  '10 | confirm_amount - count' => int 1
  '11 | confirm_amount - sum' => int 0
  '11 | confirm_amount - count' => int 0
  '{{all}} | confirm_amount - sum' => float 9000
  '{{all}} | confirm_amount - count' => int 1

11 =>

array (size=7)
  'custname' => string 'A12' (length=3)
  '10 | confirm_amount - sum' => int 0
  '10 | confirm_amount - count' => int 0
  '11 | confirm_amount - sum' => float 13000
  '11 | confirm_amount - count' => int 2
  '{{all}} | confirm_amount - sum' => float 13000
  '{{all}} | confirm_amount - count' => int 2

12 =>

array (size=7)
  'custname' => string '{{all}}' (length=7)
  '10 | confirm_amount - sum' => float 42000
  '10 | confirm_amount - count' => int 9
  '11 | confirm_amount - sum' => float 24000
  '11 | confirm_amount - count' => int 4
  '{{all}} | confirm_amount - sum' => float 66000
  '{{all}} | confirm_amount - count' => int 13

ArrayDatasource Pivot

David Winterburn commented on Jan 7, 2021

Please try to place ColumnMeta before the Pivot process, not after it. In case it still doesn't work you still have the Sort process to fall on.

Venom commented on Jan 7, 2021

Could you please help to test out the following case:

CREATE TABLE quote_test ( id int(11) NOT NULL, custname varchar(100) NOT NULL, confirm_amount int(11) NOT NULL, signed_month int(11) NOT NULL, signed_year int(11) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

--

-- Dumping data for table quote_test

INSERT INTO quote_test (id, custname, confirm_amount, signed_month, signed_year) VALUES (1, 'A1', 8000, 10, 2020), (2, 'A2', 2000, 10, 2020), (3, 'A3', 3000, 10, 2020), (4, 'A4', 4000, 11, 2020), (5, 'A5', 5000, 10, 2020), (6, 'A6', 6000, 10, 2020), (7, 'A7', 7000, 11, 2020), (8, 'A8', 9000, 10, 2020), (9, 'A9', 2000, 10, 2020), (10, 'A10', 3000, 10, 2020), (11, 'A11', 4000, 10, 2020), (12, 'A12', 8000, 11, 2020), (13, 'A13', 3000, 10, 2020), (14, 'A14', 4000, 11, 2020), (15, 'A15', 5000, 10, 2020), (16, 'A16', 6000, 10, 2020), (17, 'A17', 7000, 11, 2020), (18, 'A18', 9000, 10, 2020), (19, 'A19', 2000, 10, 2020), (20, 'A20', 3000, 10, 2020), (21, 'A22', 8000, 11, 2020), (22, 'A23', 7000, 11, 2020), (23, 'A24', 9000, 10, 2020), (24, 'A25', 2000, 10, 2020), (25, 'A26', 3000, 10, 2020), (26, 'A27', 8000, 11, 2020);

--

-- Indexes for dumped tables

--

-- Indexes for table quote_test

ALTER TABLE quote_test ADD PRIMARY KEY (id);

I have created the 26 samples for testing the pivotExtract, if I limit the record to 15 rows, the sorting looks fine, but the sorting won't work after limit record set to 16.




use \koolreport\processes\ColumnMeta;
use \koolreport\processes\Sort;
use \koolreport\processes\ColumnRename;
use \koolreport\pivot\processes\PivotExtract;
use \koolreport\pivot\processes\Pivot;
use \koolreport\processes\Map;

class CustomersYears extends koolreport\KoolReport
{
    function settings()
    {


        return array(
            "dataSources"=>array(
                "aimfs"=>array(
                    "connectionString"=>"mysql:host=xxx",
                    "username" =>"xx",
                    "password" =>"xxxx",
                    "chartset"=>"utf8"
                )
            )
        );
    }
    function setup()
    {
     
        
       $this->src('aimfs')
     
     ->query('select custname, confirm_amount, signed_month,signed_year from quote_test limit 16')

        ->pipe(new \koolreport\pivot\processes\PivotSQL([
            "column" => "signed_month",
            "row" => "custname",
            "totalName" => 'All',
            "aggregates"=>array(
                "sum"=>"confirm_amount",
                 "count"=>"confirm_amount"
            ),
        ]))
        ->pipe(new ColumnMeta(array(
            "confirm_amount - sum"=>array(
                'type' => 'number'
              
            ),
        )))
 
    
        ->pipe(new PivotExtract(array(
            "row" => array(
                "parent" => array(
                
                 
                ),
            "sort" => array(
                'confirm_amount - sum' => 'asc',
                ),
            ),
            "column" => array(
                "parent" => array(
                 
                
              
                ),
              "sort" => array(
                    'signed_month' => function($a, $b) {
                        return (int)$a < (int)$b;
                    },
                    
                    
                ),
            ),
            "measures"=>array(
                "confirm_amount - sum", 
                "confirm_amount - count", 
            ),
        )))
       
         ->pipe(new Sort(array(
             "{{all}} | confirm_amount - sum" => "asc"
         )))
         ->pipe(new Sort(array(
             "custname" => function($custName1, $custName2) {
                 if ($custName1 === "{{all}}") return 1;
                 if ($custName2 === "{{all}}") return -1; //swap these -1 and 1 if {{all}} ends up on top
                 return 0; //if neither custName1 nor custName2 are "{{all}}", keep the current order
             }
         )))
        // ->pipe(new ColumnRename(array(
        //     "custname"=>"Client Name",
        //     "10 | confirm_amount - sum"=>" Oct Total Confirm Amount",
        //     "10 | confirm_amount - count"=>" Oct Total Confirm Count"
        // )))
        ->pipe(new Map(array(
            '{value}' => function($row) {
                 foreach ($row as $colname => $value) {
                     $newColName = str_replace("{{all}}", "Total", $colname);
                     $row[$newColName] = $value;
                     if($colname != $newColName){
                       unset($row[$colname]);
                     }
                 }
                $row["custname"] = str_replace("{{all}}", "Total", $row["custname"]);
                return $row;    
            }
        )))  
        ->pipe($this->dataStore('pivotData'));

    }
}


Limit to 15 rows

Limit to 16 rows

David Winterburn commented on Jan 7, 2021

It looks like our second Sort process has a problem. Please try to replace these lines:

         ->pipe(new Sort(array(
             "{{all}} | confirm_amount - sum" => "asc"
         )))
         ->pipe(new Sort(array(
             "custname" => function($custName1, $custName2) {
                 if ($custName1 === "{{all}}") return 1;
                 if ($custName2 === "{{all}}") return -1; //swap these -1 and 1 if {{all}} ends up on top
                 return 0; //if neither custName1 nor custName2 are "{{all}}", keep the current order
             }
         )))

with these ones:

         ->pipe(new Sort(array(
             "custname" => function($custName1, $custName2) {
                 if ($custName1 === "{{all}}") return 1;
                 if ($custName2 === "{{all}}") return -1; //swap these -1 and 1 if {{all}} ends up on top
                 return 0; 
             },
            "{{all}} | confirm_amount - sum" => "asc"
         )))

Let us know the result. Thanks!

Venom commented on Jan 7, 2021

Bravo,It did solve the sorting issue. So the sorting process could not be cascaded or it needs to embed into a single sort process for multiple sorting criteria? Working

David Winterburn commented on Jan 7, 2021

Yes, I think so. The Sort process use PHP's usort function which could be nondeterministic when compare function return 0. So when we use separated Sort processes the last one could undo the previous ones. It's safer to combine them if possible.

Venom commented on Jan 7, 2021

Thank you for your kind support, hope your tips would be added to the document section.

Build Your Excellent Data Report

Let KoolReport help you to make great reports. It's free & open-source released under MIT license.

Download KoolReport View demo
solved

Pivot