Adding Data to Datastores from Multiple Datasources, how to do it efficiently?

hsw commented on Sep 21, 2021

Hi,

I meant to output an initial record from "base" datasouce to 2 dataStores "category1" and "category2" first. Then retrieve the respective category's records that are filtered from main, to output to the respective dataStores.

But the following code is not giving the result I want. (The result ended up with both dataStores having the same category1 records.)

I know one way to get it right is to duplicate the part (a) and saveTo $node1 and $node2 separately. Is there a more code efficient way?

    protected function setup()
    {
// (a) Process Base record
        $this->src('base')
        ->pipe(new Limit(array(1)))
        ->pipe(new RemoveColumn(array(
			....
        )))
        ->saveTo($node1)
        ->saveTo($node2);
    }

// (b) Process Category 1 data
        $this->src('main')
        ->pipe(new Filter(array(
			array("Category","=","1"),
        )))
        ->pipe(new Limit(array(6)))
        ->pipe(new RemoveColumn(array(
			....
        )))
        ->pipe($node1);
        $node1->pipe($this->dataStore('category1'));

// (c) Process Category 2 data
        $this->src('main')
        ->pipe(new Filter(array(
			array("Category","=","2"),
        )))
        ->pipe(new Limit(array(6)))
        ->pipe(new RemoveColumn(array(
			....
        )))
        ->pipe($node2);
        $node2->pipe($this->dataStore('category2'));

Thank you.

Sebastian Morales commented on Sep 22, 2021

I think saveTo method passes by reference, so ->saveTo($node1)->saveTo($node2) effectively makes $node1 and $node2 point to the same data. To duplicate the nodes by value separately without duplicating code probably just use a loop like this:

for ($i = 1; $i <= 2; $i++) {
        $this->src('base')
        ->pipe(new Limit(array(1)))
        ->pipe(new RemoveColumn(array(
			....
        )))
        ->saveTo(${"node$i"}); //dynamically create $node1 and $node2 variables
}
//use $node1 and $node2 here

hsw commented on Sep 23, 2021

Thanks for your reply, Sebastian. Will adopt your recommendation.

KoolReport's Forum

Adding Data to Datastores from Multiple Datasources, how to do it efficiently? #2347

Build Your Excellent Data Report