^(j1ddlmZddlZddlZddlmZddlZddlZddlZddl m Z ddl Z ddl m Z m Z ddlmZd"dZdZd Zd#d Zed$d Zd Ze j,d ZGddZGdde j2j4ZGdde j2j4Zddde j:fdZdddedde j:fdZ de j:fdZ!ddedde j:fdZ"de j:fdZ#GddejHZ%Gdd Z&ed%d!Z'y)&)contextmanagerN)Path)Image)nnoptim)datacb||Dcgc]}||j|}}||iScc}w)z4Apply passed in transforms for HuggingFace Datasets.)convert)examples transform image_keymodeimageimagess A/Users/danicosta/Desktop/Flux2/ComfyUI/comfy/k_diffusion/utils.pyhf_datasets_augs_helperrs<:B9:M N:Mi d+,:MF N v Os,c||jz }|dkrtd|jd|d|dd|zz}|jjdk(r|j j S|S)zNAppends dimensions to the end of a tensor until it has target_dims dimensions.rz input has z dims but target_dims is z, which is less).Nmps)ndim ValueErrordevicetypedetachclone)x target_dimsdims_to_appendexpandeds r append_dimsr sz 166)N:affX-F{mSbcdd'N223H)1(<(<(E8??  " " $S8ScBtd|jDS)z7Returns the number of trainable parameters in a module.c3<K|]}|jywr)numel).0ps r zn_params.."s6"5Qqwwy"5s)sum parameters)modules rn_paramsr+ s 6&"3"3"56 66r!ct|}|jjdd|jsStj j |5}t|d5}tj||dddddd|Rtjt|djj}||k7rtd|d|d|S#1swYgxYw#1swYkxYw) zLDownloads a file if it does not exist, optionally checking its SHA-256 hash.T)parentsexist_okwbNrbzhash of z (url: z) failed to validate)rparentmkdirexistsurllibrequesturlopenopenshutil copyfileobjhashlibsha256read hexdigestOSError)pathurldigestresponsef file_digests r download_filerE%s :DKKdT2 ;;= ^^ # #C (Hd46F!   x +7G ( nnT$%5%:%:%<=GGI [ HTF'#6JKL L K 7G6F ( (s$ C.%C"<C."C+ 'C..C7c#BK|jDcgc]}|j}} |j|t|jD]\}}|||_ycc}w#t|jD]\}}|||_wxYww)zdA context manager that places a model into training mode and restores the previous mode on exit.N)modulestrainingtrain enumerate)modelrr*modesis r train_moderN3s,1==? ;?V__?E ;'kk$"5==?3IAv#AhFO4 <#5==?3IAv#AhFO4s%BA*BA/1B/-BBct|dS)zfA context manager that places a model into evaluation mode and restores the previous mode on exit.F)rN)rKs r eval_moderP?s eU ##r!c0t|j}t|j}|j|jk(sJ|jD]-\}}||j |j |d|z /t|j }t|j }|j|jk(sJ|jD]\}} ||j| y)zIncorporates updated model parameters into an exponential moving averaged version of a model. It should be called after each optimizer step.)alphaN)dictnamed_parameterskeysitemsmul_add_ named_bufferscopy_) rKaveraged_modeldecay model_paramsaveraged_paramsnameparam model_buffersaveraged_buffersbufs r ema_updatereEs..01L>::<=O    /"6"6"8 88 8#))+ e""5)..uAI.F,,,./MN88:;    #3#8#8#: :: :"((* c$$S)+r!c4eZdZdZ ddZdZdZdZdZy) EMAWarmupaYImplements an EMA warmup using an inverse decay schedule. If inv_gamma=1 and power=1, implements a simple average. inv_gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), inv_gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). Args: inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. power (float): Exponential factor of EMA warmup. Default: 1. min_value (float): The minimum EMA decay rate. Default: 0. max_value (float): The maximum EMA decay rate. Default: 1. start_at (int): The epoch to start averaging at. Default: 0. last_epoch (int): The index of last epoch. Default: 0. cX||_||_||_||_||_||_yr) inv_gammapower min_value max_valuestart_at last_epoch)selfrirjrkrlrmrns r__init__zEMAWarmup.__init__hs," ""  $r!cHt|jjS)z2Returns the state of the class as a :class:`dict`.)rT__dict__rWros r state_dictzEMAWarmup.state_dictqsDMM'')**r!c:|jj|y)zLoads the class's state. Args: state_dict (dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. N)rrupdate)rorts rload_state_dictzEMAWarmup.load_state_dictus Z(r!ctd|j|jz }dd||jz z|j zz }|dkrdSt |j t|j|S)z Gets the current EMA decay rate.rrR)maxrnrmrirjminrlrk)roepochvalues r get_valuezEMAWarmup.get_value}seAt67Q//TZZK??QYrSCDNNE8R$SSr!c.|xjdz c_y)zUpdates the step count.rRN)rnrss rstepzEMAWarmup.steps 1r!N)?rryrrr) __name__ __module__ __qualname____doc__rprtrwr~rr!rrgrgXs+ UV%+)T r!rgc4eZdZdZ dfd ZdZdZxZS) InverseLRaMImplements an inverse decay learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. inv_gamma is the number of steps/epochs required for the learning rate to decay to (1 / 2)**power of its original value. Args: optimizer (Optimizer): Wrapped optimizer. inv_gamma (float): Inverse multiplicative factor of learning rate decay. Default: 1. power (float): Exponential factor of learning rate decay. Default: 1. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. c||_||_d|cxkrdkstdtd||_||_t ||||yNryrRzInvalid value for warmup)rirjrwarmupmin_lrsuperrp) ro optimizerrirjrrrnverbose __class__s rrpzInverseLR.__init__Z" Va78 8 78 8   J8r!cd|jstjd|jSNzTTo get the last learning rate computed by the scheduler, please use `get_last_lr()`._get_lr_called_within_stepwarningswarn_get_closed_form_lrrss rget_lrzInverseLR.get_lr-.. MM8 9''))r!c d|j|jdzzz }d|j|jz z|j z}|jDcgc]}|t |j ||zz c}Scc}wNrR)rrnrirjbase_lrsrzrrorlr_multbase_lrs rrzInverseLR._get_closed_form_lrs~T[[T__q%899t77TZZKG#}}.,GT[['G*;<<,. ..s#A>)rrryryFrrrrrprr __classcell__rs@rrrs! MO(-9*.r!rc4eZdZdZ dfd ZdZdZxZS) ExponentialLRaEImplements an exponential learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. Decays the learning rate continuously by decay (default 0.5) every num_steps steps. Args: optimizer (Optimizer): Wrapped optimizer. num_steps (float): The number of steps to decay the learning rate by decay in. decay (float): The factor by which to decay the learning rate every num_steps steps. Default: 0.5. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. c||_||_d|cxkrdkstdtd||_||_t ||||yr) num_stepsr]rrrrrp) rorrr]rrrnrrs rrpzExponentialLR.__init__rr!cd|jstjd|jSrrrss rrzExponentialLR.get_lrrr!c d|j|jdzzz }|jd|jz z|jz}|jDcgc]}|t |j ||zz c}Scc}wr)rrnr]rrrzrrs rrz!ExponentialLR._get_closed_form_lrs|T[[T__q%899::!dnn"45$//I#}}.,GT[['G*;<<,. ..s#A=)g?ryryrFrrs@rrrs! KM(-9*.r!rryrcpucZtj||||z|zjS)z-Draws samples from an lognormal distribution.rdtype)torchrandnexp)shapelocscalerrs rrand_log_normalrs( KKfE :U BS H M M OOr!infc~tj||tj}tj||tj}|jj |j |j }|jj |j |j }tj||tj||z z|z} | jj|j|jj|S)zEDraws samples from an optionally truncated log-logistic distribution.r) r as_tensorfloat64logsubdivsigmoidrandlogitmuladdrto) rrrrkrlrrmin_cdfmax_cdfus rrand_log_logisticrs & NI & NImmo!!#&**5199;Gmmo!!#&**5199;G 5u}}=7ARSV]]A 779==  # #C ( , , . 1 1% 88r!ctj|}tj|}tj|||||z z|zj S)z/Draws samples from an log-uniform distribution.r)mathrrrr)rrkrlrrs rrand_log_uniformrsJ#I#I JJuV5 9Y=R SV_ _ d d ffr!cLtj||z dztjz }tj||z dztjz }tj|||||z z|z}tj |tjzdz |zS)zJDraws samples from a truncated v-diffusion training timestep distribution.r)ratanpirrtan) r sigma_datarkrlrrrrrs rrand_v_diffusionrsii J./!3dgg=Gii J./!3dgg=G 5u579JKgUA 99Q[1_ % 22r!ctj|||j}tj|||}|| z|z}||z|z} |||zz } tj|| k|| j S)z2Draws samples from a split lognormal distribution.r)rrabsrwherer) rrscale_1scale_2rrnrn_leftn_rightratios rrand_split_log_normalrs| E&6::eZdZdZhdZdfd ZdZdZdZxZ S)FolderOfImageszURecursively finds all images in a directory. It does not support classes/targets.> .bmp.jpg.pgm.png.ppm.tif.jpeg.tiff.webpctt|_|t j n|_tfdjjdD_ y)Nc3pK|]-}|jjjvs*|/ywr)suffixlowerIMG_EXTENSIONS)r%r?ros rr'z*FolderOfImages.__init__.. s/p-ATT[[EVEVEX\`\o\oEoD-As+66*) rrprrootrIdentityr sortedrglobpaths)rorr rs` rrpzFolderOfImages.__init__sL J *3*;pTYY__S-App r!c:d|jdt|dS)NzFolderOfImages(root="z", len: ))rlenrss r__repr__zFolderOfImages.__repr__s&tyyk#d)AFFr!c,t|jSr)rrrss r__len__zFolderOfImages.__len__s4::r!c|j|}t|d5}tj|jd}ddd|j }|fS#1swYxYw)Nr0RGB)rr7rr r )rokeyr?rCrs r __getitem__zFolderOfImages.__getitem__sVzz# $ JJqM))%0Eu%v  s %AA&r) rrrrrrprrrrrs@rrrs&aNq Gr!rceZdZdZdZy) CSVLoggerct||_||_|jjrt |jd|_yt |jd|_|j |jy)Naw)rfilenamecolumnsr3r7filewrite)rorrs rrpzCSVLogger.__init__sZX  ==   !T]]C0DIT]]C0DI DJJ %r!c2t|d|jddy)N,T)seprflush)printr)roargss rrzCSVLogger.write&s t499D9r!N)rrrrprrr!rrrs &:r!rc#Ktjjj}tjjj j} ||tjj_|)|tjjj _d||tjj_|*|tjjj _yy#||tjj_|*|tjjj _wwxYww)zGA context manager that sets whether TF32 is allowed on cuDNN or matmul.N)rbackendscudnn allow_tf32cudamatmul)r r cudnn_old matmul_olds r tf32_moder*s$$//I$$++66J ?  .3ENN +  4:ENN   & & 1  .7ENN +  4>ENN   & & 1   .7ENN +  4>ENN   & & 1 s!AEAC4&AE4AEE)rr)T)NN)( contextlibrr:rpathlibrr8r4rPILrrrr torch.utilsrrr r+rErNrPno_gradrerg lr_scheduler _LRSchedulerrrfloat32rfloatrrrrDatasetrrrrr!rrsQ%    T7 ''$ **$--`&.""//&.R&.E&&33&.R "EP "$2uU|\ainiviv9:?emmg(*R5