KoolReport's Forum

Official Support Area, Q&As, Discussions, Suggestions and Bug reports.
Forum's Guidelines

Pivot Report Slow? [Answer: not really] #1556

Closed Anil Vithani opened this topic on on Aug 10, 2020 - 14 comments

Anil Vithani commented on Aug 10, 2020

There are 4422 rows being returned from database. Problem is that this report have become very very slow and taking around 2 mins to load completely.

My PivotReport.php

<?php
namespace App\Reports;

use \koolreport\processes\Filter;
use \koolreport\processes\ColumnMeta;
use \koolreport\pivot\processes\Pivot;


class PivotReport extends \koolreport\KoolReport
{
    use \koolreport\laravel\Friendship;
    use \koolreport\export\Exportable;
    use \koolreport\excel\ExcelExportable;
	
    // By adding above statement, you have claim the friendship between two frameworks
    // As a result, this report will be able to accessed all databases of Laravel
    // There are no need to define the settings() function anymore
    // while you can do so if you have other datasources rather than those
    // defined in Laravel.
    
    function setup()
    {

		$this->src('mysql')
			->query("SELECT CONCAT(CONCAT('Week Commencing', ' ', WEEK(nr.created_date)), '/', MONTH(nr.created_date)) as wk, ct.team_name, nr.user_name, 
            YEAR(nr.created_date) as year,
			DAY(nr.created_date) as day, QUARTER(nr.created_date)as quarter,
        	SUM(IF(nr.notes_type = 126, 1, 0)) AS candidate_added,
        	SUM(IF(nr.notes_type = 76, 1, 0)) AS dm_added, 
        	SUM(IF(nr.notes_type = 102, 1, 0)) AS account_added, 
        	SUM(IF((nr.notes_type = 139 AND nr.candidate_id> 0), 1, 0)) AS cv_send, 
        	SUM(IF(nr.notes_type = 399, 1, 0)) AS grade_b_jobs, 
        	SUM(IF((nr.notes_type = 71 AND nr.candidate_id> 0), 1, 0)) AS cand_spec_send, 
        	SUM(IF((nr.notes_type = 71 AND nr.dm_id> 0), 1, 0)) AS dm_spec_send, 
        	SUM(IF(nr.notes_type = 207, 1, 0)) AS cv_upload, 
        	SUM(IF(nr.notes_type = 213, 1, 0)) AS add_dm_hiring_profile
        FROM `notes_report`  nr
        LEFT JOIN company_teams ct ON nr.team_id = ct.id
        WHERE nr.notes_type 
        IN(126,76,102,139,399,71,207,213) 
        GROUP BY wk, ct.team_name, nr.user_name, year
        ORDER BY wk, ct.team_name, nr.user_name, year
        
        
            ")
            ->pipe(new Filter(array(
                //array('name', 'startsWith', 'a'),
               array('year', '>', 2003),
            )))
        /* ->params(array(
         * WHERE exo.name LIKE :name AND YEAR(nr.created_date) > :year
            ":name"=>"%Th%",
            ":year"=>"2003"
        )) */
        ->pipe(new ColumnMeta(array(
            "candidate_added"=>array(
                'type' => 'number',
                //"prefix" => "$",
            ),
        )))
        ->pipe(new Pivot(array(
            "dimensions"=>array(
                "column" => "year",
                "row" => "wk, team_name, user_name",
            ),
            "aggregates"=>array(
               "sum"=>"candidate_added",
               "count"=>"candidate_added"
            )
        )))
        ->pipe($this->dataStore('pivot'));
    }
}

My PivotReport.view.php

<?php
use \koolreport\pivot\widgets\PivotTable;
use \koolreport\pivot\widgets\PivotMatrix;
use \koolreport\core\Utility;
use \koolreport\processes\Filter;
use \koolreport\processes\ColumnMeta;
use \koolreport\pivot\processes\Pivot;
use \koolreport\instant\Widget;
use \koolreport\datasources\CSVDataSource;
?>
<div class='report-content'>
  <div class="text-center">
    <h1>Sale Report</h1>
    <p class="lead">
      Summarize amount of sales and number of sales by three dimensions: customers, categories and products
    </p>
  </div>
  <div>
    <?php
    
      /* PivotTable::create(array(
          'dataStore'=>$this->dataStore('pivot'),
        'rowDimension'=>'row',
        'measures'=>array(
          'candidate_added - sum', 
          'candidate_added - count',
        ),
        'rowSort' => array(
          'candidate_added - sum' => 'desc',
        ),
        'rowCollapseLevels' => array(1),
        'totalName' => 'All',
        'width' => '100%',
        'nameMap' => array(
          'candidate_added - sum' => 'Total Candidates Added',
          'candidate_added - count' => 'Number of Candidates',
        ),
      )); */
    ?>
  </div>
  
  <form id='form1' class="form-inline" method="post">

        <?php

        PivotMatrix::create(array(
            "id" => "pivotMatrix1",
            'dataSource' => $this->dataStore('pivot'),
            "measures"=>array(
                "candidate_added - sum", 
            ),
            'rowSort' => array(
                'candidate_added - sum' => 'desc',
                'user_name' => 'desc',
            ),
            'columnSort' => array(
                'month' => function($a, $b) {
                    return (int)$a < (int)$b;
                },
            ),
            'columnCollapseLevels' => array(0),
            'rowCollapseLevels' => array(0),
            'width' => '100%',
            'height' => '500px',
            'headerMap' => function($v, $f) {
                switch ($v) {
                    case 'candidate_added - sum': return 'Total Candiates Added';
                    case 'candidate_added - count': return 'Number of Canidate Added';
                    case 'candidate_added - avg': return 'Average Candiate Added';
                    case 'year': return 'Year';
                    case 'month': return 'Month';
                    case 'day': return 'Day';
                    case 'wk': return 'Week Commecing';
                    case 'team_name': return 'Team';
                    case 'user_name': return 'User';
                    
                }
                $r = $v;
                if ($f === 'year')
                    $r = 'Year ' . $v;
                $map = array(
                    '1' => 'January',
                    '2' => 'February',
                    '3' => 'March',
                    '4' => 'April',
                    '5' => 'May',
                    '6' => 'June',
                    '7' => 'July',
                    '8' => 'August',
                    '9' => 'September',
                    '10' => 'October',
                    '11' => 'November',
                    '12' => 'December',
                );
                if ($f === 'month')
                    $r = $map[$v];
                return $r;
            },
            'totalName' => 'All',
            'waitingFields' => array(
                'candidate_added - count' => 'data', 
                'month' => 'label',
                'day' => 'label',
            ),
            'paging' => array(
                'size' => 5,
                'maxDisplayedPages' => 5,
                'sizeSelect' => array(5, 10, 20, 50, 100)
            )
        ));
        ?>
    </form>
  
</div>
David Winterburn commented on Aug 11, 2020

Please keep the Pivot process in your report's setup and replace your PivotMatrix in your report's view with a text saying "This is a PivotMatrix" and let us know how much time it takes to render the view. Thanks!

Anil Vithani commented on Aug 11, 2020

I replaced the view part with a text, and it took around 8 seconds to render which is fine as data being fetched from a remote computer.

David Winterburn commented on Aug 11, 2020

So the processing time is not a problem but PivotMatrix will cost some time to render many rows because it's more complicated than PivotTable. I would advise you add the following option in the Pivot process so that PivotMatrix only load part of the data for the current page only:

        ->pipe(new Pivot(array(
            "dimensions"=>array(
                "column" => "year",
                "row" => "wk, team_name, user_name",
            ),
            "aggregates"=>array(
               "sum"=>"candidate_added",
               "count"=>"candidate_added"
            ),
            'partialProcessing' => true, //add this line
        )))

Let us know how this works for you. Thanks!

Anil Vithani commented on Aug 12, 2020

Putting that loaded the report initially but it went in endless in progress overlay when try to expand the first row.

David Winterburn commented on Aug 12, 2020

Did you use any framework like Laravel, Symphony, etc?

Anil Vithani commented on Aug 12, 2020

yes, laravel

Anil Vithani commented on Aug 12, 2020

fixed it with this but what you recommend?, it is still significantly slow

PivotMatrix::create(array(

"id" => "pivotMatrix1",
...
"scope" =>  array(
	"_token" => csrf_token(),
            "select1" => $this->params["select1"],
            ...
),
...

));

David Winterburn commented on Aug 12, 2020

How slow are we talking about? If it's possible please send a link of your page to our email support@koolphp.net for us to check it for you. Thanks!

Anil Vithani commented on Aug 12, 2020

sent you email with the link

David Winterburn commented on Aug 13, 2020

How exactly many rows does your query return? In our experiment with a table of 3000 rows PivotMatrix takes less than 1 seconds to process.

Let's measure your processing time by open the file koolreport/processes/pivot.php and add the following line to the beginning of function onInit:

    public function onInit()
    {
        $this->initTime = microtime(true); //add this line

and these lines to the end of function finalize:

    public function finalize()
    {
        ... 
        $pivotProcessedTime = microtime(true) - $this->initTime;
        echo "pivotProcessedTime=$pivotProcessedTime seconds";
    }

Then in each expand please open your browser's dev tool (F12), move to tab Network, click label XHR and check each xhr request's response to see pivotProcessedTime value.

Anil Vithani commented on Aug 13, 2020

pivotProcessedTime=24.248105049133 seconds

Plz see screen shot

David Winterburn commented on Aug 14, 2020

How many rows does your query return?

Anil Vithani commented on Aug 14, 2020

around 4000

David Winterburn commented on Aug 18, 2020

Hi Anil,

After a live remote support session we have come to conclusion that the sole reason for your pivot report's slowness is because of the sql query which takes about 20-30 seconds to return result while Pivot process only takes 0.1-0.2 second to handle thousands of rows. Please try to optimize your sql query (using store procedure, temporary table, cache, etc) for faster report loading. Let us know if you need any help. Thanks!

Build Your Excellent Data Report

Let KoolReport help you to make great reports. It's free & open-source released under MIT license.

Download KoolReport View demo
help needed
solved

None