Class OptimizerRuleBased
- java.lang.Object
-
- org.apache.sysds.runtime.controlprogram.parfor.opt.Optimizer
-
- org.apache.sysds.runtime.controlprogram.parfor.opt.OptimizerRuleBased
-
- Direct Known Subclasses:
OptimizerConstrained,OptimizerHeuristic
public class OptimizerRuleBased extends Optimizer
Rule-Based ParFor Optimizer (time: O(n)): Applied rule-based rewrites - 1) rewrite set data partitioner (incl. recompile RIX) - 2) rewrite remove unnecessary compare matrix - 3) rewrite result partitioning (incl. recompile LIX) - 4) rewrite set execution strategy - 5) rewrite set operations exec type (incl. recompile) - 6) rewrite use data colocation - 7) rewrite set partition replication factor - 8) rewrite set export replication factor - 9) rewrite use nested parallelism - 10) rewrite set degree of parallelism - 11) rewrite set task partitioner - 12) rewrite set fused data partitioning and execution - 13) rewrite transpose vector operations (for sparse) - 14) rewrite set in-place result indexing - 15) rewrite disable caching (prevent sparse serialization) - 16) rewrite enable runtime piggybacking - 17) rewrite inject spark loop checkpointing - 18) rewrite inject spark repartition (for zipmm) - 19) rewrite set spark eager rdd caching - 20) rewrite set result merge - 21) rewrite set recompile memory budget - 22) rewrite remove recursive parfor - 23) rewrite remove unnecessary parfor TODO fuse also result merge into fused data partitioning and execute (for writing the result directly from execute we need to partition columns/rows according to blocksize -> rewrite (only applicable if numCols/blocksize>numreducers)+custom MR partitioner) TODO take remote memory into account in data/result partitioning rewrites (smaller/larger) TODO memory estimates with shared reads TODO memory estimates of result merge into plan tree TODO blockwise partitioning
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.sysds.runtime.controlprogram.parfor.opt.Optimizer
Optimizer.CostModelType, Optimizer.PlanInputType
-
-
Field Summary
Fields Modifier and Type Field Description static booleanALLOW_REMOTE_NESTED_PARALLELISMstatic StringFUNCTION_UNFOLD_NAMEPREFIXstatic intMAX_REPLICATION_FACTOR_EXPORTstatic intMAX_REPLICATION_FACTOR_PARTITIONINGstatic doublePAR_K_FACTORstatic doublePAR_K_MR_FACTORstatic doublePROB_SIZE_THRESHOLD_MBstatic doublePROB_SIZE_THRESHOLD_PARTITIONINGstatic doublePROB_SIZE_THRESHOLD_REMOTE
-
Constructor Summary
Constructors Constructor Description OptimizerRuleBased()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static booleanallowsBinaryCellPartitions(MatrixObject mo, ParForProgramBlock.PartitionFormat dpf)Optimizer.CostModelTypegetCostModelType()ParForProgramBlock.POptModegetOptMode()Optimizer.PlanInputTypegetPlanInputType()static booleanisInMemoryResultMerge(long rows, long cols, double memBudget)booleanoptimize(ParForStatementBlock sb, ParForProgramBlock pb, OptTree plan, CostEstimator est, ExecutionContext ec)Main optimization procedure.static StringtoMB(double inB)-
Methods inherited from class org.apache.sysds.runtime.controlprogram.parfor.opt.Optimizer
getNumEvaluatedPlans, getNumTotalPlans
-
-
-
-
Field Detail
-
PROB_SIZE_THRESHOLD_REMOTE
public static final double PROB_SIZE_THRESHOLD_REMOTE
- See Also:
- Constant Field Values
-
PROB_SIZE_THRESHOLD_PARTITIONING
public static final double PROB_SIZE_THRESHOLD_PARTITIONING
- See Also:
- Constant Field Values
-
PROB_SIZE_THRESHOLD_MB
public static final double PROB_SIZE_THRESHOLD_MB
- See Also:
- Constant Field Values
-
MAX_REPLICATION_FACTOR_PARTITIONING
public static final int MAX_REPLICATION_FACTOR_PARTITIONING
- See Also:
- Constant Field Values
-
MAX_REPLICATION_FACTOR_EXPORT
public static final int MAX_REPLICATION_FACTOR_EXPORT
- See Also:
- Constant Field Values
-
ALLOW_REMOTE_NESTED_PARALLELISM
public static final boolean ALLOW_REMOTE_NESTED_PARALLELISM
- See Also:
- Constant Field Values
-
FUNCTION_UNFOLD_NAMEPREFIX
public static final String FUNCTION_UNFOLD_NAMEPREFIX
- See Also:
- Constant Field Values
-
PAR_K_FACTOR
public static final double PAR_K_FACTOR
- See Also:
- Constant Field Values
-
PAR_K_MR_FACTOR
public static final double PAR_K_MR_FACTOR
- See Also:
- Constant Field Values
-
-
Method Detail
-
getCostModelType
public Optimizer.CostModelType getCostModelType()
- Specified by:
getCostModelTypein classOptimizer
-
getPlanInputType
public Optimizer.PlanInputType getPlanInputType()
- Specified by:
getPlanInputTypein classOptimizer
-
getOptMode
public ParForProgramBlock.POptMode getOptMode()
- Specified by:
getOptModein classOptimizer
-
optimize
public boolean optimize(ParForStatementBlock sb, ParForProgramBlock pb, OptTree plan, CostEstimator est, ExecutionContext ec)
Main optimization procedure. Transformation-based heuristic (rule-based) optimization (no use of sb, direct change of pb).
-
allowsBinaryCellPartitions
public static boolean allowsBinaryCellPartitions(MatrixObject mo, ParForProgramBlock.PartitionFormat dpf)
-
isInMemoryResultMerge
public static boolean isInMemoryResultMerge(long rows, long cols, double memBudget)
-
toMB
public static String toMB(double inB)
-
-