Class ColumnEncoderComposite
- java.lang.Object
-
- org.apache.sysds.runtime.transform.encode.ColumnEncoder
-
- org.apache.sysds.runtime.transform.encode.ColumnEncoderComposite
-
- All Implemented Interfaces:
Externalizable,Serializable,Comparable<ColumnEncoder>,Encoder
public class ColumnEncoderComposite extends ColumnEncoder
Simple composite encoder that applies a list of encoders in specified order. By implementing the default encoder API it can be used as a drop-in replacement for any other encoder.- See Also:
- Serialized Form
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.sysds.runtime.transform.encode.ColumnEncoder
ColumnEncoder.EncoderType
-
-
Field Summary
-
Fields inherited from class org.apache.sysds.runtime.transform.encode.ColumnEncoder
APPLY_ROW_BLOCKS_PER_COLUMN, BUILD_ROW_BLOCKS_PER_COLUMN
-
-
Constructor Summary
Constructors Constructor Description ColumnEncoderComposite()ColumnEncoderComposite(List<ColumnEncoder> columnEncoders)ColumnEncoderComposite(List<ColumnEncoder> columnEncoders, FrameBlock meta)ColumnEncoderComposite(ColumnEncoder columnEncoder)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidaddEncoder(ColumnEncoder other)voidallocateMetaData(FrameBlock meta)Pre-allocate a FrameBlock for metadata collection.MatrixBlockapply(CacheBlock in, MatrixBlock out, int outputCol, int rowStart, int blk)voidbuild(CacheBlock in)Build the transform meta data for the given block input.voidbuild(CacheBlock in, Map<Integer,double[]> equiHeightMaxs)voidbuildPartial(FrameBlock in)Partial build of internal data structures (e.g., in distributed spark operations).voidcomputeRCDMapSizeEstimate(CacheBlock in, int[] sampleIndices)booleanequals(Object o)List<DependencyTask<?>>getApplyTasks(CacheBlock in, MatrixBlock out, int outputCol)List<DependencyTask<?>>getBuildTasks(CacheBlock in)<T extends ColumnEncoder>
TgetEncoder(Class<T> type)List<ColumnEncoder>getEncoders()FrameBlockgetMetaData(FrameBlock out)Construct a frame block out of the transform meta data.Set<Integer>getSparseRowsWZeros()<T extends ColumnEncoder>
booleanhasBuild()<T extends ColumnEncoder>
booleanhasEncoder(Class<T> type)inthashCode()voidinitMetaData(FrameBlock out)Sets up the required meta data for a subsequent call to apply.booleanisEncoder(int colID, Class<?> type)voidmergeAt(ColumnEncoder other)Merges another encoder, of a compatible type, in after a certain position.voidprepareBuildPartial()Allocates internal data structures for partial build.voidreadExternal(ObjectInput in)Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.voidsetNumPartitions(int nBuild, int nApply)voidshiftCol(int columnOffset)StringtoString()voidupdateAllDCEncoders()voidupdateIndexRanges(long[] beginDims, long[] endDims, int colOffset)Update index-ranges to after encoding.voidwriteExternal(ObjectOutput out)Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.-
Methods inherited from class org.apache.sysds.runtime.transform.encode.ColumnEncoder
apply, build, compareTo, getBuildTask, getColID, getColMapping, getEstMetaSize, getEstNumDistincts, getPartialBuildTask, getPartialMergeBuildTask, isApplicable, isApplicable, setColID, setEstMetaSize, setEstNumDistincts
-
-
-
-
Constructor Detail
-
ColumnEncoderComposite
public ColumnEncoderComposite()
-
ColumnEncoderComposite
public ColumnEncoderComposite(List<ColumnEncoder> columnEncoders, FrameBlock meta)
-
ColumnEncoderComposite
public ColumnEncoderComposite(List<ColumnEncoder> columnEncoders)
-
ColumnEncoderComposite
public ColumnEncoderComposite(ColumnEncoder columnEncoder)
-
-
Method Detail
-
getEncoders
public List<ColumnEncoder> getEncoders()
-
getEncoder
public <T extends ColumnEncoder> T getEncoder(Class<T> type)
-
isEncoder
public boolean isEncoder(int colID, Class<?> type)
-
build
public void build(CacheBlock in)
Description copied from interface:EncoderBuild the transform meta data for the given block input. This call modifies and keeps meta data as encoder state.- Parameters:
in- input frame block
-
build
public void build(CacheBlock in, Map<Integer,double[]> equiHeightMaxs)
- Overrides:
buildin classColumnEncoder
-
getApplyTasks
public List<DependencyTask<?>> getApplyTasks(CacheBlock in, MatrixBlock out, int outputCol)
- Overrides:
getApplyTasksin classColumnEncoder
-
getBuildTasks
public List<DependencyTask<?>> getBuildTasks(CacheBlock in)
- Overrides:
getBuildTasksin classColumnEncoder
-
prepareBuildPartial
public void prepareBuildPartial()
Description copied from class:ColumnEncoderAllocates internal data structures for partial build.- Specified by:
prepareBuildPartialin interfaceEncoder- Overrides:
prepareBuildPartialin classColumnEncoder
-
buildPartial
public void buildPartial(FrameBlock in)
Description copied from class:ColumnEncoderPartial build of internal data structures (e.g., in distributed spark operations).- Specified by:
buildPartialin interfaceEncoder- Overrides:
buildPartialin classColumnEncoder- Parameters:
in- input frame block
-
apply
public MatrixBlock apply(CacheBlock in, MatrixBlock out, int outputCol, int rowStart, int blk)
- Overrides:
applyin classColumnEncoder
-
mergeAt
public void mergeAt(ColumnEncoder other)
Description copied from class:ColumnEncoderMerges another encoder, of a compatible type, in after a certain position. Resizes as necessary.ColumnEncodersare compatible with themselves andEncoderCompositeis compatible with every otherColumnEncoders.MultiColumnEncodersare compatible with every encoder- Overrides:
mergeAtin classColumnEncoder- Parameters:
other- the encoder that should be merged in
-
updateAllDCEncoders
public void updateAllDCEncoders()
-
addEncoder
public void addEncoder(ColumnEncoder other)
-
updateIndexRanges
public void updateIndexRanges(long[] beginDims, long[] endDims, int colOffset)Description copied from class:ColumnEncoderUpdate index-ranges to after encoding. Note that only Dummycoding changes the ranges.- Specified by:
updateIndexRangesin interfaceEncoder- Overrides:
updateIndexRangesin classColumnEncoder- Parameters:
beginDims- begin dimensions of rangeendDims- end dimensions of rangecolOffset- is applied to begin and endDims
-
allocateMetaData
public void allocateMetaData(FrameBlock meta)
Description copied from interface:EncoderPre-allocate a FrameBlock for metadata collection.- Parameters:
meta- frame block
-
getMetaData
public FrameBlock getMetaData(FrameBlock out)
Description copied from interface:EncoderConstruct a frame block out of the transform meta data.- Parameters:
out- output frame block- Returns:
- output frame block?
-
initMetaData
public void initMetaData(FrameBlock out)
Description copied from interface:EncoderSets up the required meta data for a subsequent call to apply.- Parameters:
out- frame block
-
writeExternal
public void writeExternal(ObjectOutput out) throws IOException
Description copied from class:ColumnEncoderRedirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.- Specified by:
writeExternalin interfaceExternalizable- Overrides:
writeExternalin classColumnEncoder- Parameters:
out- object output- Throws:
IOException- if IOException occurs
-
readExternal
public void readExternal(ObjectInput in) throws IOException
Description copied from class:ColumnEncoderRedirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.- Specified by:
readExternalin interfaceExternalizable- Overrides:
readExternalin classColumnEncoder- Parameters:
in- object input- Throws:
IOException- if IOException occur
-
hasEncoder
public <T extends ColumnEncoder> boolean hasEncoder(Class<T> type)
-
hasBuild
public <T extends ColumnEncoder> boolean hasBuild()
-
computeRCDMapSizeEstimate
public void computeRCDMapSizeEstimate(CacheBlock in, int[] sampleIndices)
-
setNumPartitions
public void setNumPartitions(int nBuild, int nApply)
-
shiftCol
public void shiftCol(int columnOffset)
- Overrides:
shiftColin classColumnEncoder
-
getSparseRowsWZeros
public Set<Integer> getSparseRowsWZeros()
- Overrides:
getSparseRowsWZerosin classColumnEncoder
-
-