com.twitter.scalding.typed

TypedPipe

trait TypedPipe[+T] extends Serializable

Think of a TypedPipe as a distributed unordered list that may or may not yet have been materialized in memory or disk.

Represents a phase in a distributed computation on an input data source Wraps a cascading Pipe object, and holds the transformation done up until that point

Linear Supertypes
Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Hide All
  2. Show all
  1. TypedPipe
  2. Serializable
  3. AnyRef
  4. Any
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def cross[U](tiny: TypedPipe[U]): TypedPipe[(T, U)]

  2. abstract def flatMap[U](f: (T) ⇒ TraversableOnce[U]): TypedPipe[U]

  3. abstract def fork: TypedPipe[T]

    If you are going to create two branches or forks, it may be more efficient to call this method first which will create a node in the cascading graph.

    If you are going to create two branches or forks, it may be more efficient to call this method first which will create a node in the cascading graph. Without this, both full branches of the fork will be put into separate cascading pipes, which can, in some cases, be slower.

    Ideally the planner would see this

  4. abstract def limit(count: Int): TypedPipe[T]

    limit the output to at most count items.

    limit the output to at most count items. useful for debugging, but probably that's about it. The number may be less than count, and not sampled particular method

  5. abstract def sample(percent: Double, seed: Long): TypedPipe[T]

  6. abstract def sample(percent: Double): TypedPipe[T]

  7. abstract def sumByLocalKeys[K, V](implicit ev: <:<[T, (K, V)], sg: Semigroup[V]): TypedPipe[(K, V)]

    This does a sum of values WITHOUT triggering a shuffle.

    This does a sum of values WITHOUT triggering a shuffle. the contract is, if followed by a group.sum the result is the same with or without this present, and it never increases the number of items. BUT due to the cost of caching, it might not be faster if there is poor key locality.

    It is only useful for expert tuning, and best avoided unless you are struggling with performance problems. If you are not sure you need this, you probably don't.

    The main use case is to reduce the values down before a key expansion such as is often done in a data cube.

  8. abstract def toPipe[U >: T](fieldNames: Fields)(implicit setter: TupleSetter[U]): Pipe

    Export back to a raw cascading Pipe.

    Export back to a raw cascading Pipe. useful for interop with the scalding Fields API or with Cascading code.

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. def ++[U >: T](other: TypedPipe[U]): TypedPipe[U]

    Merge two TypedPipes (no order is guaranteed) This is only realized when a group (or join) is performed.

  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  7. def aggregate[B, C](agg: Aggregator[T, B, C]): ValuePipe[C]

    Same as groupAll.

    Same as groupAll.aggregate.values

  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  9. def clone(): AnyRef

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  10. def collect[U](fn: PartialFunction[T, U]): TypedPipe[U]

    Filter and map.

    Filter and map. See scala.collection.List.collect. collect { case Some(x) => fn(x) }

  11. def cross[V](p: ValuePipe[V]): TypedPipe[(T, V)]

    Attach a ValuePipe to each element this TypedPipe

  12. def debug: TypedPipe[T]

    prints the current pipe to stdout

  13. def distinct(implicit ord: Ordering[_ >: T]): TypedPipe[T]

    Returns the set of distinct elements in the TypedPipe

    Returns the set of distinct elements in the TypedPipe

    Annotations
    @implicitNotFound( ... )
  14. def distinctBy[U](fn: (T) ⇒ U, numReducers: Option[Int])(implicit ord: Ordering[_ >: U]): TypedPipe[T]

    Returns the set of distinct elements identified by a given lambda extractor in the TypedPipe

    Returns the set of distinct elements identified by a given lambda extractor in the TypedPipe

    Annotations
    @implicitNotFound( ... )
  15. def either[R](that: TypedPipe[R]): TypedPipe[Either[T, R]]

    Merge two TypedPipes of different types by using Either

  16. def eitherValues[K, V, R](that: TypedPipe[(K, R)])(implicit ev: <:<[T, (K, V)]): TypedPipe[(K, Either[V, R])]

    Sometimes useful for implementing custom joins with groupBy + mapValueStream when you know that the value/key can fit in memory.

    Sometimes useful for implementing custom joins with groupBy + mapValueStream when you know that the value/key can fit in memory. Beware.

  17. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  19. def filter(f: (T) ⇒ Boolean): TypedPipe[T]

    Keep only items that satisfy this predicate

  20. def filterKeys[K](fn: (K) ⇒ Boolean)(implicit ev: <:<[T, (K, Any)]): TypedPipe[T]

    If T is a (K, V) for some V, then we can use this function to filter.

    If T is a (K, V) for some V, then we can use this function to filter. This is here to match the function in KeyedListLike, where it is optimized

  21. def filterNot(f: (T) ⇒ Boolean): TypedPipe[T]

    Keep only items that don't satisfy the predicate.

    Keep only items that don't satisfy the predicate. filterNot is the same as filter with a negated predicate.

  22. def filterWithValue[U](value: ValuePipe[U])(f: (T, Option[U]) ⇒ Boolean): TypedPipe[T]

    common pattern of attaching a value and then filter

  23. def finalize(): Unit

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  24. def flatMapWithValue[U, V](value: ValuePipe[U])(f: (T, Option[U]) ⇒ TraversableOnce[V]): TypedPipe[V]

    common pattern of attaching a value and then flatMap

  25. def flatten[U](implicit ev: <:<[T, TraversableOnce[U]]): TypedPipe[U]

    flatten an Iterable

  26. def forceToDisk: TypedPipe[T]

    Force a materialization of this pipe prior to the next operation.

    Force a materialization of this pipe prior to the next operation. This is useful if you filter almost everything before a hashJoin, for instance.

  27. final def getClass(): java.lang.Class[_]

    Definition Classes
    AnyRef → Any
  28. def group[K, V](implicit ev: <:<[T, (K, V)], ord: Ordering[K]): Grouped[K, V]

    This is the default means of grouping all pairs with the same key.

    This is the default means of grouping all pairs with the same key. Generally this triggers 1 Map/Reduce transition

  29. def groupAll: Grouped[Unit, T]

    Send all items to a single reducer

  30. def groupBy[K](g: (T) ⇒ K)(implicit ord: Ordering[K]): Grouped[K, T]

    Given a key function, add the key, then call .

    Given a key function, add the key, then call .group

  31. def groupRandomly(partitions: Int): Grouped[Int, T]

    Forces a shuffle by randomly assigning each item into one of the partitions.

    Forces a shuffle by randomly assigning each item into one of the partitions.

    This is for the case where you mappers take a long time, and it is faster to shuffle them to more reducers and then operate.

    You probably want shard if you are just forcing a shuffle.

  32. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  33. def hashCogroup[K, V, W, R](smaller: HashJoinable[K, W])(joiner: (K, V, Iterable[W]) ⇒ Iterator[R])(implicit ev: <:<[TypedPipe[T], TypedPipe[(K, V)]]): TypedPipe[(K, R)]

    These operations look like joins, but they do not force any communication of the current TypedPipe.

    These operations look like joins, but they do not force any communication of the current TypedPipe. They are mapping operations where this pipe is streamed through one item at a time.

    WARNING These behave semantically very differently than cogroup. This is because we handle (K,V) tuples on the left as we see them. The iterable on the right is over all elements with a matching key K, and it may be empty if there are no values for this key K.

  34. def hashJoin[K, V, W](smaller: HashJoinable[K, W])(implicit ev: <:<[TypedPipe[T], TypedPipe[(K, V)]]): TypedPipe[(K, (V, W))]

    Do an inner-join without shuffling this TypedPipe, but replicating argument to all tasks

  35. def hashLeftJoin[K, V, W](smaller: HashJoinable[K, W])(implicit ev: <:<[TypedPipe[T], TypedPipe[(K, V)]]): TypedPipe[(K, (V, Option[W]))]

    Do an leftjoin without shuffling this TypedPipe, but replicating argument to all tasks

  36. def hashLookup[K >: T, V](grouped: HashJoinable[K, V]): TypedPipe[(K, Option[V])]

    For each element, do a map-side (hash) left join to look up a value

  37. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  38. def keys[K](implicit ev: <:<[T, Tuple2[K, _]]): TypedPipe[K]

    Just keep the keys, or .

    Just keep the keys, or ._1 (if this type is a Tuple2)

  39. def leftCross[V](thatPipe: TypedPipe[V]): TypedPipe[(T, Option[V])]

    uses hashJoin but attaches None if thatPipe is empty

  40. def leftCross[V](p: ValuePipe[V]): TypedPipe[(T, Option[V])]

    ValuePipe may be empty, so, this attaches it as an Option cross is the same as leftCross(p).

    ValuePipe may be empty, so, this attaches it as an Option cross is the same as leftCross(p).collect { case (t, Some(v)) => (t, v) }

  41. def map[U](f: (T) ⇒ U): TypedPipe[U]

    Transform each element via the function f

  42. def mapValues[K, V, U](f: (V) ⇒ U)(implicit ev: <:<[T, (K, V)]): TypedPipe[(K, U)]

    Transform only the values (sometimes requires giving the types due to scala type inference)

  43. def mapWithValue[U, V](value: ValuePipe[U])(f: (T, Option[U]) ⇒ V): TypedPipe[V]

    common pattern of attaching a value and then map

  44. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  45. final def notify(): Unit

    Definition Classes
    AnyRef
  46. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  47. def shard(partitions: Int): TypedPipe[T]

    Used to force a shuffle into a given size of nodes.

    Used to force a shuffle into a given size of nodes. Only use this if your mappers are taking far longer than the time to shuffle.

  48. def sketch[K, V](reducers: Int, eps: Double = 1.0E-5, delta: Double = 0.01, seed: Int = 12345)(implicit ev: <:<[TypedPipe[T], TypedPipe[(K, V)]], serialization: (K) ⇒ Array[Byte], ordering: Ordering[K]): Sketched[K, V]

    Build a sketch of this TypedPipe so that you can do a skew-join with another Grouped

  49. def sum[U >: T](implicit plus: Semigroup[U]): ValuePipe[U]

    Reasonably common shortcut for cases of associative/commutative reduction returns a typed pipe with only one element.

  50. def sumByKey[K, V](implicit ev: <:<[T, (K, V)], ord: Ordering[K], plus: Semigroup[V]): UnsortedGrouped[K, V]

    Reasonably common shortcut for cases of associative/commutative reduction by Key

  51. def swap[K, V](implicit ev: <:<[T, (K, V)]): TypedPipe[(V, K)]

    swap the keys with the values

  52. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  53. def toString(): String

    Definition Classes
    AnyRef → Any
  54. def unpackToPipe[U >: T](fieldNames: Fields)(implicit up: TupleUnpacker[U]): Pipe

    use a TupleUnpacker to flatten U out into a cascading Tuple

  55. def values[V](implicit ev: <:<[T, Tuple2[_, V]]): TypedPipe[V]

    Just keep the values, or .

    Just keep the values, or ._2 (if this type is a Tuple2)

  56. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  57. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  58. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  59. def write(dest: TypedSink[T])(implicit flowDef: FlowDef, mode: Mode): TypedPipe[T]

    Safely write to a TypedSink[T].

    Safely write to a TypedSink[T]. If you want to write to a Source (not a Sink) you need to do something like: toPipe(fieldNames).write(dest)

    returns

    a pipe equivalent to the current pipe.

Inherited from Serializable

Inherited from AnyRef

Inherited from Any


API Documentation hosted by AppDoc.app (https://appdoc.app)