python - How to discretize large dataframe by columns with variable bins in Pandas/Dask -

-

i able discretize pandas dataframe columns code:

import numpy np import pandas pd  def discretize(x, n_scale=1):      c in x.columns:         loc = x[c].median()          # median absolute deviation of column         scale = mad(x[c])          bins = [-np.inf, loc - (scale * n_scale),                 loc + (scale * n_scale), np.inf]         x[c] = pd.cut(x[c], bins, labels=[-1, 0, 1])      return x

i want discretize each column using parameters: loc (the median of column) , scale (the median absolute deviation of column).

with small dataframes time required acceptable (since single thread solution).

however, larger dataframes want exploit more threads (or processes) speed computation.

i no expert of dask, should provide solution problem.

however, in case discretization should feasible code:

import dask.dataframe dd import numpy np import pandas pd  def discretize(x, n_scale=1):      # i'm using 2 partitions example     x_dask = dd.from_pandas(x, npartitions=2)      # fixme:      # how can define bins compute loc , scale     # each column?     bins = [-np.inf, loc - (scale * n_scale),             loc + (scale * n_scale), np.inf]      x = x_dask.apply(pd.cut, axis=1, args=(bins,), labels=[-1, 0, 1]).compute()      return x

but problem here loc , scale dependent on column values, should computed each column, either before or during apply.

how can done?

i've never used dask, guess can define new function used in apply.

import dask.dataframe dd import multiprocessing mp import numpy np import pandas pd  def discretize(x, n_scale=1):      x_dask = dd.from_pandas(x.t, npartitions=mp.cpu_count()+1)     x = x_dask.apply(_discretize_series,                      axis=1, args=(n_scale,),                      columns=x.columns).compute().t      return x  def _discretize_series(x, n_scale=1):      loc = x.median()     scale = mad(x)     bins = [-np.inf, loc - (scale * n_scale),             loc + (scale * n_scale), np.inf]     x = pd.cut(x, bins, labels=[-1, 0, 1])      return x

Comments

Post a Comment

Popular posts from this blog

Spring Boot + JPA + Hibernate: Unable to locate persister -

-
i trying work entity inside request method of spring boot application. entity in project , pom.xml has dependencies it. why error message?: org.hibernate.unknownentitytypeexception: unable locate persister: com.ric.bill.model.bs.par package com.ric.web; import java.util.concurrent.atomic.atomiclong; import javax.persistence.entitymanager; import javax.persistence.persistencecontext; import net.sf.ehcache.cachemanager; import net.sf.ehcache.management.cache; import net.sf.ehcache.management.cachestatistics; import org.hibernate.stat.statistics; import org.springframework.beans.factory.annotation.autowired; import org.springframework.boot.autoconfigure.domain.entityscan; import org.springframework.cache.annotation.enablecaching; import org.springframework.transaction.annotation.enabletransactionmanagement; import org.springframework.transaction.annotation.propagation; import org.springframework.transaction.annotation.transactional; import org.springframework.web.bind.annotati...
Read more

go - Golang: panic: runtime error: invalid memory address or nil pointer dereference using bufio.Scanner -

-
i implementing go program uses bufio.scanner , bufio.writer have packaged code follows package main import ( "fmt" "player/command" "strings" ) func main() { //enter code here. read input stdin. print output stdout commands.scanner.scan() { //scan new line , send comand variable check command exist or not input := strings.split(strings.trim(commands.scanner.text(), " "), " ") command := input[0] if command == "" { fmt.printf("$ %s:", commands.pwd) continue } if !commands.commands[command] { commands.throwerror("cannot recognize input.") } else { commands.execute(command, input[1:], nil) } fmt.printf("$ %s:", commands.pwd) } } i using init.go file in main package follows package main import ( "flag" "player/sourc...
Read more

c - double free or corruption (fasttop) -

-
this code reads null terminated string file (file containing null terminated strings & should provide index wanted string). error: *** error in `./main': double free or corruption (fasttop): 0x090a4a80 *** aborted code: char *tmp_realloc=null; char *sstring=null; int i=1; /*set file pointer point string*/ fseek(fh,index,seek_set); sstring=(char*)malloc(sizeof(char)); if(sstring == null) return null; tmp_realloc=sstring; while( (*(sstring+i-1)=(char)fgetc((file*)fh)) != 0x0 ) { /*reallocate more memory*/ tmp_realloc=(char*)realloc((char*)sstring,++i); /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strcpy((char*)tmp_realloc,(char*)sstring); free(sstring); sstring=tmp_realloc; } } could tell me pointer causing issue (& reason) ? remove part /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strc...
Read more