In my large report, one section calculates the "nexus" and displays it in the report. It works probably 50% of the time. Here is what it looks like when that section of the report doesn't populate.

here is the creation of the datatable:

        $sql="SELECT item as Project, business_component, user_email, employee, w2_wages, dollars_per_pr as qualified_$, taxyear,number_of_employees, number_consultants, num_contracts, qualified_dollars, business_component  
        FROM survey_results_PR_temp srpr
       WHERE campaign='$campaign' AND email='$email' AND taxyear='$taxyear'";
 //       GROUP BY role";

        $this->src('substantiator')
            ->query($sql)
            ->pipe($this->dataStore("survey_project"))  ;

and here is the creation of the grid in that section of the report

Table::create(array(
        "dataStore"=>$this->dataStore("survey_project")->filter("taxyear","=",$_SESSION['last_TY']),
        "showFooter"=>true,



        "grouping"=>array(
        "business_component"=>array(
            "calculate"=>array(
                "{sumAmount}"=>array("sum","qualified_$"),
            ),
            "bottom"=>"<td><b>Total qualified wages for : {business_component}</b></td><td><b>{sumAmount}</b></td><br>",

        ),
    ),

        "showFooter"=>"bottom",
        "columns"=>array(
            "business_component"=>array("label"=>"Business Component"),
            "employee",
            "user_email",
            "w2_wages"=>array(
                "label"=>"Gross wages"
            ),
            "qualified_dollars"=>array(
                "label"=>"QREs"
            ),


            "qualified_$"=>array(
                "label"=>"Wages",
                "cssStyle"=>"text-align:left",
                "prefix"=>"$",
                "footer"=>"sum",
                "footerText"=>"<b>Total Qualified Wages for Business Component:</b> @value")

        ),

        "cssClass"=>array(
            "table"=>"table table-bordered",
            "tr"=>"row-css-class",
            "th"=>"header-css-class",
            "td"=>"cell-css-class",
            "tf"=>"footer-cell-css-class",
        )
    )
);

I am also attaching the json of the file survey_results_PR_temp.

[{"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"accpac project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Applefishy", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "role":"researcher", "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "role":"researcher", "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"xf1xr@imnart.com", "employee":"xf1", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":78000, "role":"researcher", "dollars_per_PR":78000, "number_consultants":0, "num_contracts":0, "w2_wages":78000, "amount":0}]

This is the way it should look and appears sometimes immediately (rarely) and sometimes after I refresh the screen (sometimes). If it doesn't appear I can hit refresh 4 or 5 times and it eventually appears.

This is what I get:

session last_TY = 201708 datastore survey_project data = Array ( )

Does that mean that the datastore is blank? Then I tried it again and now i get:

session last_TY = 201708
datastore survey_project data = Array ( [0] => Array ( [Project] => test project [business_component] => Pear Pie [user_email] => richardbernstein216@yahoo.com [employee] => Richard3 [w2_wages] => 55000 [qualified_$] => 17875 [taxyear] => 201708 [number_of_employees] => 1 [number_consultants] => 0 [num_contracts] => 1 [qualified_dollars] => 35750 [role] => researcher [employee_title] => tester ) [1] => Array ( [Project] => accpac project [business_component] => Applefishy [user_email] => richardbernstein216@yahoo.com [employee] => Richard3 [w2_wages] => 55000 [qualified_$] => 17875 [taxyear] => 201708 [number_of_employees] => 1 [number_consultants] => 0 [num_contracts] => 1 [qualified_dollars] => 35750 [role] => researcher [employee_title] => tester ) [2] => Array ( [Project] => test project [business_component] => Pear Pie [user_email] => xf1xr@imnart.com [employee] => xf1 [w2_wages] => 78000 [qualified_$] => 78000 [taxyear] => 201708 [number_of_employees] => 1 [number_consultants] => 0 [num_contracts] => 1 [qualified_dollars] => 78000 [role] => researcher [employee_title] => tester ) )

So what are the implications when sometimes the datastore doesn't populate? I then tried it six or seven times in a row and it worked. On the eighth try it stopped working. On the ninth try it started working again. Please advise.

I am using kooklreport cache. Could this be related to that? Here are two images; one when it worked and one when it didn't. There have been no code or data changes between each of these.

Here are two images. The first is the sql printout when the table worked and the second is when it didn't work. The runs were only a minute part. To me they look exactly the same.

I ran it again and it didnt work. I copied the table. Here it is when it didn;'t work:

[{"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"accpac project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Applefishy", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"xf1xr@imnart.com", "employee":"xf1", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":78000, "dollars_per_PR":78000, "number_consultants":0, "num_contracts":0, "w2_wages":78000, "amount":0}]

then I took my dog out for a walk and ran it when i got back and it started working and again and I exported the table. Here it is when it worked:

[{"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"accpac project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Applefishy", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"richardbernstein216@yahoo.com", "employee":"Richard3", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":35750, "dollars_per_PR":17875, "number_consultants":0, "num_contracts":0, "w2_wages":55000, "amount":0},
 {"taxyear":"201708", "user_email":"xf1xr@imnart.com", "employee":"xf1", "item":"test project", "campaign":"testco", "email":"richb201@gmail.com", "business_component":"Pear Pie", "role":"researcher", "title":"tester", "qualified":0, "number_of_employees":1, "num_projects":1, "qualified_dollars":78000, "dollars_per_PR":78000, "number_consultants":0, "num_contracts":0, "w2_wages":78000, "amount":0}]

To me they seem the same! Also, I am using Google Charts later, way down in the report, but not at all in this part of the report. My internet is pretty stable here (in US).

do you mean the setup or the view? Can I email it to you? It is probably too large to post. Right now I am having a divide by zero error closer to that area. I'd like to fix that brfore I send it to you (but I don't know how).

OK. Not much to it.

    public function report_generator2()
    {

        $report = new MyReport;
        $report->run()->render();
    }

seems like a race condition. If I wait long enough before rendering it seems to work more often. Is there any way I can test to see if datastore survey_project data is getting populated programmatically? If so I can just delay a few seconds and retry?

Any idea where?

   public function run()
    {
        if ($this->fireEvent("OnBeforeRun")) {
            if ($this->dataSources != null) {
                foreach ($this->dataSources as $dataSource) {
                    if (!$dataSource->isEnded()) {
                        $dataSource->start();
                    }
                }
            }
        }
        $this->fireEvent("OnRunEnd");
        return $this;
    }

If I was to check to see if empty($this->dataStore("survey_project")->data()) !=false), can you think of a way to delay and then to try again? I am currently running on my laptop and the mysql is on an AWS server. So there could be a transmission delay. In reality both will be running on aws servers so it should be much faster.

Where in my code would I put this "if empty"? Before the run()? I am loathe to modify the koolport source code.

Can I put your above code right before the run()

public function report_generator2()
    {
 foreach ($this->dataSources as $dataSource) {
                    echo "foreach dataSource<br>";                    
                    if (!$dataSource->isEnded()) {
                        echo "dataSource is not ended, start to run<br>";
                        $dataSource->start();
                    }
                }
        $report = new MyReport;
        $report->run()->render();
    }

Sebastian, it might be an application issue. I am trying to check that. I'll let you know what I find out.

I think I solved it and it was an application issue.

I thought this was solved but I guess it is not. What is strange is that sometimes the datastore survey_project data populates and sometimes it does not. There is no change in the code or data between attempts.

So I am gong to follow your recommendation to add echo "dataSource is not ended, start to run<br>"; to koolreport/core/src/koolreport.php. I put a breakpoint in it but it doesn't seem to trigger which means, that i am using the wrong koolreport/core/src/koolreport.php. There seem to be a number of koolreport dirs on my machine. How can I tell which location is being used?

Your recommendation of writing a stub to test would not work, IMO. There seems to be some kind of race condition since it occurs occasionally (~50% of the time). Sometimes I can hit the browser refresh and it then works. What does this imply?

ALSO. I have two different sample reports I am running. The one with the small amount of data fails about 50% of the time. When i run the larger report, it fails 100% of the time.

Is there a way to delay the rendering? Will this give AWS RDS time to fully populate the datastore BEFORE koolreport tries to render? Or can I pre-build this part of the report and just "inject it" into the real report at run time?

Well, I created a separate report of only the section that is causing the problem. When i run it alone, I get the same problem. I have some ideas. I'll get back to you.