[cascading user] Cascading counter

If in a cascading Pipe as below,

Pipe P1 = new UniqueCount(P1,new Fields(Field1, Field2,Field3), new Fields(COUNT).

From the above pipe, I need to use the COUNT to get assigned to a local variable to display the statistics of how many records are present.

How should i assign this value?

(OR)

What is the other way to get the count value?

You'll have N values for COUNT, one for each unique combination fo values for Field1, Field2, and Field3, right?

So it's not a single value, and it doesn't represent how many records were present, it represents how many groups you get from a key composed of those three fields.

In any case, some options...

1. Attach P1 to a text file sink Tap, then read that when the Flow has completed.

2. Attach P1 to a database sink Tap, and read those records when the Flow has completed.

3. If the number of unique combinations of fields is going to be "small" (say less than 100) then you could use a custom function that sythensizes a counter name from the field values and uses the count as the increment for that counter.

Thanks. But I cannot attach that pipe to tail sink because I have intermediate pipes whose count is also needed. So in this case what should I do?

Split the pipe if you need it both as output and as input to subsequent Flow steps.

Pipe outputCountersPipe = new Pipe("output counters", P1);

Now continue using P1 in the rest of your Flow, and connect the outputCountersPipe to the sink.

Thanks. If we need to count how many records present in a pipe, how to do that?

http://docs.cascading.org/cascading/2.5/javadoc/cascading/operation/state/Counter.html