a ٴ:eK@s:ddlmZmZmZmZddlZddlZddl Z e j ddej eej eefdddZ eeddd d Ze j ddej eeeeefeeefej d d d Zej eeeefeeefej dddZejedddZej eeeefej dddZej eeej dddZe j ddej eeefeeefeeeej eefdddZej eeefej dddZej eeefeej eefd d!d"Zej eej d#d$d%Zeedd&d'd(Zej eeefej dd)d*Ze j ddej eeefej d+d,d-Ze j ddej eeefej d+d.d/Zej eeefdd d0d1Zeeefeeefeeefeeefeeeefed2d3d4Z eeefeeefeeeeefeeefeeefeeefeeefeej eeeffd5 d6d7Z!e j ddej eeefej d8d9d:Z"dS);)AnyDictListTupleNi,)ttl)dfreturncCs0|jdd}t||dkj}|j|dd|fS)afRemove columns with strictly less than 2 distinct values in input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be checked and potentially removed. Returns ------- pd.DataFrame Dataframe with empty columns removed. list List of columns that have been removed. Fdropnaaxis)nuniquelistindexdrop)rZ count_cols empty_colsriC:\Users\mjcastro\Documents\data-science-function-store\apps\fbco_simulator\app\..\lib\dataprep\format.pyremove_empty_colss r)rrc Cs`t|}|dkr\td|dkr"dndd|dkr4dndd|dkrFd nd d d |d S)zDisplays a message in streamlit dashboard if the input list is not empty. Parameters ---------- empty_cols : list List of columns that have been removed. rThe following columnr s have been removed because z they havezit hasz <= 1 distinct values: , Nlensterrorjoin)rLrrrprint_empty_colss&r$)df_inputdate_col target_colconfig load_optionsrcCs2|}t||||}t|||}t|||}|S)aFormats date and target columns of input dataframe. Parameters ---------- df_input : pd.DataFrame Input dataframe whose columns will be formatted. date_col : str Name of date column in input dataframe. target_col : str Name of target column in input dataframe. config : Dict Lib configuration dictionary. load_options : Dict Loading options selected by user. Returns ------- pd.DataFrame Dataframe with columns formatted. )copy _format_date_format_target _rename_cols)r%r&r'r(r)rrrrformat_date_and_target-s   r.)rr&r)r(rcCszt||}t||dd|dkBrBtj|||dd}|||<||||j}||||j}|dk|dk@t|t|@Brt dt |WSt dt Yn0dS)aFormats date column of input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be formatted. date_col : str Name of date column in input dataframe. load_options : Dict Loading options selected by user. config : Dict Lib config dictionary containing information about default date format. Returns ------- pd.DataFrame Dataframe with date column formatted. dataprep date_format)formatr zNPlease select the correct date column (selected column has a time range < 1s).zQPlease select a valid date format (selected column can't be converted into date).N) pd to_datetime__check_date_formatmaxmindayssecondsnpisnanr r!stop)rr&r)r( date_seriesZ days_rangeZ sec_rangerrrr+Ps($r+)r<rcCsZ|dddk}|dddk}|dddk}||@|@rRdSdSdS) aChecks whether the date column has been correctly converted to datetime. Parameters ---------- date_series : pd.Series Date column that has been converted. Returns ------- bool False if conversion has not worked correctly, True otherwise. cSs|jSN)yearxrrrz%__check_date_format..r cSs|jSr=)monthr?rrrrArBcSs|jSr=)dayr?rrrrArBTFN)mapr)r<Ztest1Ztest2Ztest3rrrr4{s  r4)rr'r(rcCsfzB||d||<|||ddkr>tdt|WStdtYn0dS)a;Formats target column of input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be formatted. target_col : str Name of target column in input dataframe. Returns ------- pd.DataFrame Dataframe with date column formatted. floatvalidityZmin_target_cardinalityzOPlease select the correct target column (should be numerical, not categorical).zIPlease select the correct target column (should be of type int or float).N)astyperr r!r;)rr'r(rrrr,s r,)rr&r'rcCs\|dkr"d|jvr"|jddid}|dkrDd|jvrD|jddid}|j|d|did}|S)aRenames date and target columns of input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be renamed. date_col : str Name of date column in input dataframe. target_col : str Name of target column in input dataframe. Returns ------- pd.DataFrame Dataframe with columns renamed. yZy_2columnsdsZds_2)rKrename)rr&r'rrrr-s r-)r% dimensionsr(r&r'rcCs2|}t||}t||\}}t||}||fS)a{Filters and aggregates input dataframe according to dimensions dictionary specifications. Parameters ---------- df_input : pd.DataFrame Input dataframe that will be filtered and/or aggregated. dimensions : Dict Filtering and aggregation specifications. config : Dict Lib configuration dictionary. date_col : str Name of date column in input dataframe. target_col : str Name of target column in input dataframe. Returns ------- pd.DataFrame Dataframe filtered and/or aggregated. list List of columns removed from input dataframe. )r*_filter_format_regressors _aggregate)r%rNr(r&r'r cols_to_droprrrfilter_and_aggregate_dfs   rS)rrNrcCsFtt|dh}|D]}|j||||}q|j|ddS)aCFilters input dataframe according to dimensions dictionary specifications. Parameters ---------- df : pd.DataFrame Input dataframe that will be filtered and/or aggregated. dimensions : Dict Filtering specifications. Returns ------- pd.DataFrame Filtered dataframe. aggr r )rsetkeyslocisinr)rrNZ filter_colscolrrrrOsrO)rr(rcCsg}t|jddhD]}||jdddkr:||q||jdddkrx||tt||ddg||<q|||dd krt||}qz|| d ||<Wq||Yq0q|j |dd |fS) aXFormat some columns in input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be formatted. config : Dict Lib configuration dictionary. Returns ------- pd.DataFrame Formatted dataframe. list List of columns removed from input dataframe. rLrIFr r rr rGZmax_cat_reg_cardinalityrFr ) rUrKrappendrEdictzipunique__one_hot_encodingrHr)rr(rRrYrrrrPs * rP)rrYrcCs.tj|tj|||dgdd}|j|ddS)a<Applies one-hot encoding to some columns of input dataframe. Parameters ---------- df : pd.DataFrame Input dataframe whose columns will be one-hot encoded. col : list List of columns to one-hot encode. Returns ------- pd.DataFrame One-hot encoded dataframe. )prefixr r )r2concat get_dummiesr)rrYrrrr^"s r^) cols_removedrc Cs`t|}|dkr\td|dkr"dndd|dkr4dndd|dkrFd nd d d |d S)zDisplays a message in streamlit dashboard if the input list is not empty. Parameters ---------- cols_removed : list List of columns that have been removed. rrr rrrrrzthey arezit iszA neither the target, nor a dimension, nor a potential regressor: rNr)rbr#rrrprint_removed_cols5s&rccsHtjddh}fdd|D}|d|d<d|S)aHAggregates input dataframe according to dimensions dictionary specifications. Parameters ---------- df : pd.DataFrame Input dataframe that will be filtered and/or aggregated. dimensions : Dict Filtering specifications. Returns ------- pd.DataFrame Aggregated dataframe. rLrIcs&i|]}||dkrdndqSr meanr5r.0rYrrr VrBz_aggregate..rT)rUrKlowergroupbyrT reset_index)rrN cols_to_aggagg_dictrrirrQFsrQ)r% resamplingrcCsD|}|dddvr@|ddd|d<t|d|d<|S)aFormats date column to datetime in input dataframe. Parameters ---------- df_input : pd.DataFrame Input dataframe whose date column will be formatted to datetime. resampling : Dict Dictionary whose "freq" key contains the frequency of input dataframe. Returns ------- pd.DataFrame Dataframe with date column formatted to datetime. freq)HrrLcSs |dS)Nz%Y-%m-%d %H:%M:%S)strftimer?rrrrAmrBz!format_datetime..)r*rEr2r3)r%rprrrrformat_datetime[s rucsj||drftjddh}fdd|D}|d|d<d|dd|S) a<Resamples input dataframe according to resampling dictionary specifications. Parameters ---------- df_input : pd.DataFrame Input dataframe that will be resampled. resampling : Dict Resampling specifications. Returns ------- pd.DataFrame Resampled dataframe. resamplerLrIcs&i|]}||dkrdndqSrdrfrgrirrrjrBzresample_df..rTrqrr)r*rUrKrk set_indexrvrTrm)r%rprnrorrir resample_dfrs"rxcCsBt||dd|ddkr>tdt|dtdS)aDisplays a message in streamlit dashboard and stops it if the input dataframe has not enough rows. Parameters ---------- df : pd.DataFrame Input dataframe. config : Dict Lib configuration dictionary where the minimum number of rows is given. rGZmin_data_points_trainZmin_data_points_valz(The dataset has not enough data points (zj data points only) to make a forecast. Please resample with a higher frequency or change cleaning options.N)rr r!r;)rr(rrrcheck_dataset_sizes ry)datasetsdatesparamsrpr&rNrc sd}d|vr||djvr:td|dt|d|}ttj|d|d|dd}||krtd |d |d |d d d |d d d|dd tt |d} t |dt | t| krZfdd| D} t| dkrrBz.check_future_regressors_df..r zColumns rNrrz and z; are missing in the dataset provided for future regressors.zColumn rz: is missing in the dataset provided for future regressors.cSsh|]}|dkr|qS)rTrrhdimrrr rBz-check_future_regressors_df..csg|]}|vr|qSrrrrrrrrBzDimension columns zDimension column T) rVrKr r!r;rrr2 date_rangertrU intersectionr") rzr{r|rpr&rNZuse_regressorsZ N_dates_inputZN_dates_expectedZregressors_expectedZmissing_regressorsZ dim_expectedZ missing_dimrrrcheck_future_regressors_dfsn    ""r) rzr{r&r'rNr)r(rpr|rc Csd|vr|d} d| |<tj|dt| j| gdd} t| \} } t| ||||} t| ||||\} } t| |} t | |} | j | d|dk|d<| j dd d} n2tj |dj |d |d d } tj| dgd } t| |} | |fS)a*Applies data preparation to the dataset provided with future regressors. Parameters ---------- datasets : Dict Dictionary storing all dataframes. dates : Dict Dictionary containing future forecasting dates information. date_col : str Name of date column. target_col : str Name of target column. dimensions : Dict Dictionary containing dimensions information. load_options : Dict Loading options selected by user. config : Dict Lib configuration dictionary. resampling : Dict Resampling specifications. params : Dict Dictionary containing all model parameters Returns ------- pd.DataFrame Prepared future dataframe. dict Dictionary storing all dataframes. r}ruploadedr rLr~fullrIr rZ forecast_freqrrJ)rVr2r`rrKrr.rSrurxrWrrrLr6 DataFrameadd_cap_and_floor_cols) rzr{r&r'rNr)r(rpr|future_Z future_datesrrrprepare_future_dfs&)       r)r%r|rcCs<|}|dddkr8|dd|d<|dd|d<|S)aMResamples input dataframe according to resampling dictionary specifications. Parameters ---------- df_input : pd.DataFrame Input dataframe that will be resampled. params : Dict Model parameters. Returns ------- pd.DataFrame Dataframe with cap and floor columns if specified. otherZgrowthlogisticZ saturationcapfloor)r*)r%r|rrrrr8s r)#typingrrrrnumpyr9pandasr2 streamlitr cache_datarrr$strr.r+Seriesboolr4r,r-rSrOrPr^rcrQrurxryrrrrrrrsv "   # +    $*!          Z        ?