How to join 'n' number of files in ordered way efficiently using paste/join or linux or perl? -

-

thousands of files ends *.tab. first column in each file header. every file have own headers (so different). don't mind have 1 header file.

number of rows equal in files , have order. desired output have same order.

example files in directory

test_1.tab test_2.tab . . . . test_1990.tab test_2000.tab

test_1.tab

pro_01 0 0 0 0 0 1 1 1 0 1 1 0 .....0 pro_02 0 0 0 0 0 1 1 0 0 0 0 0 .....1 pro_03 1 1 1 1 1 0 0 1 0 1 1 0 .....1 . . . pro_200 0 0 0 0 1 1 1 1 1 1 0  .....0

test_2000.tab

pro_1901 1 1 1 1 0 1 1 0 0 0 0 1 .....0 pro_1902 1 1 1 0 0 0 1 0 0 0 0 0 .....1 pro_1903 1 1 0 1 0 1 0 0 0 0 0 1 .....1 . . . pro_2000 1 0 0 0 0 1 1 1 1 1 0  .....0

desired output

pro_01 0 0 0 0 0 1 1 1 0 1 1 0 0 ..... 1 1 1 1 0 1 1 0 0 0 0 1 0 pro_02 0 0 0 0 0 1 1 0 0 0 0 0 1 ..... 1 1 1 0 0 0 1 0 0 0 0 0 1 pro_03 1 1 1 1 1 0 0 1 0 1 1 0 1 ..... 1 1 0 1 0 1 0 0 0 0 0 1 1 . . . pro_200 0 0 0 0 1 1 1 1 1 1 0 0  ..... 1 0 0 0 0 1 1 1 1 1 0 0

my code

for in *.tab/; paste allcol.tab <(cut -f 2- "$i") > itermediate.csv; mv intermediate.csv allcol.tab ; done  paste <(cut -f1 test1.tab) allcol.tab > final.tab rm allcol.tab

it takes quite time 3 hrs. better way? also, there other command cross check output file vs input files? diff or wc?

try this.

#!/bin/bash      tmp=tmp mkdir "$tmp" result=result  #read each file , append contents of each line in them #to new file each line in tmp directory  f in *.tab;     i=1     while read -r l;         echo "$l" >> "$tmp"/"$i"         ((i++))     done < <(cut -f2- "$f") done  #integrate each file in tmp dir single line of $result file exec 1>>$result     f in "$tmp"/*;     while read -r l;         printf '%s\t' "$l"     done < <(cat "$f")     echo done  rm -r "$tmp"

this algorithm can split on number of processors , task done faster.

you can add things checking if $tmp created successfully.


Comments

Post a Comment

Popular posts from this blog

Spring Boot + JPA + Hibernate: Unable to locate persister -

-
i trying work entity inside request method of spring boot application. entity in project , pom.xml has dependencies it. why error message?: org.hibernate.unknownentitytypeexception: unable locate persister: com.ric.bill.model.bs.par package com.ric.web; import java.util.concurrent.atomic.atomiclong; import javax.persistence.entitymanager; import javax.persistence.persistencecontext; import net.sf.ehcache.cachemanager; import net.sf.ehcache.management.cache; import net.sf.ehcache.management.cachestatistics; import org.hibernate.stat.statistics; import org.springframework.beans.factory.annotation.autowired; import org.springframework.boot.autoconfigure.domain.entityscan; import org.springframework.cache.annotation.enablecaching; import org.springframework.transaction.annotation.enabletransactionmanagement; import org.springframework.transaction.annotation.propagation; import org.springframework.transaction.annotation.transactional; import org.springframework.web.bind.annotati
Read more

go - Golang: panic: runtime error: invalid memory address or nil pointer dereference using bufio.Scanner -

-
i implementing go program uses bufio.scanner , bufio.writer have packaged code follows package main import ( "fmt" "player/command" "strings" ) func main() { //enter code here. read input stdin. print output stdout commands.scanner.scan() { //scan new line , send comand variable check command exist or not input := strings.split(strings.trim(commands.scanner.text(), " "), " ") command := input[0] if command == "" { fmt.printf("$ %s:", commands.pwd) continue } if !commands.commands[command] { commands.throwerror("cannot recognize input.") } else { commands.execute(command, input[1:], nil) } fmt.printf("$ %s:", commands.pwd) } } i using init.go file in main package follows package main import ( "flag" "player/sourc
Read more

c - double free or corruption (fasttop) -

-
this code reads null terminated string file (file containing null terminated strings & should provide index wanted string). error: *** error in `./main': double free or corruption (fasttop): 0x090a4a80 *** aborted code: char *tmp_realloc=null; char *sstring=null; int i=1; /*set file pointer point string*/ fseek(fh,index,seek_set); sstring=(char*)malloc(sizeof(char)); if(sstring == null) return null; tmp_realloc=sstring; while( (*(sstring+i-1)=(char)fgetc((file*)fh)) != 0x0 ) { /*reallocate more memory*/ tmp_realloc=(char*)realloc((char*)sstring,++i); /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strcpy((char*)tmp_realloc,(char*)sstring); free(sstring); sstring=tmp_realloc; } } could tell me pointer causing issue (& reason) ? remove part /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strc
Read more