Gathering heterogeneous data with hadoop -

-

we have system, including oracle , microsoft sql dbms, data different sources , in different formats, stores , process it. "different formats" means files: dbf, xls , others, including binary formats (images), imported dbms different tools, , direct access databases. want isolate incoming data , store "forever" , want them later source , creation time. after studies want try hadoop ecosystem, not quite sure, if it's adequate solution goal. , parts of ecosystem should use? hdfs alone, hive, may else? give me piece of advise?

i assume want store files contain data -- searchable file archive.

the files can stored in hdfs ... or may find system amazon's s3 cheaper , more flexible. store files, manage other data data, namely: location, source, , creation time appending file -- simple tab-separated file or several other formats supported hadoop make easy.

you can manage , query file hive or other sql-on-hadoop tools. in effect, you're creating simple file system special attributes, trick make sure each time write file, write metadata. may have handle cases write failures, happens when delete, rename, or move files (i know, "never").

your solution might simpler depending on needs, may find storing data in subdirectories within hdfs (or aws s3) simpler. perhaps if wanted store dbf files source "foo", , xls files "bar" created on december 1, 2015, create directory structure like

/2015/12/01/foo/dbf/myfile.dbf /2015/12/01/bar/xls/myexcel.xls

this solution has advantage of being self-maintaining -- file path stores metadata makes portable , simple, requiring nothing more shell script implement.

i don't think there's reason make solution more complicated necessary. hadoop or s3 both fine long-term, high-durability storage , querying. company has found storing information file in hadoop (which use many other purposes) , storing files on aws s3 far simpler, more secured , cheaper.


Comments

Post a Comment

Popular posts from this blog

Spring Boot + JPA + Hibernate: Unable to locate persister -

-
i trying work entity inside request method of spring boot application. entity in project , pom.xml has dependencies it. why error message?: org.hibernate.unknownentitytypeexception: unable locate persister: com.ric.bill.model.bs.par package com.ric.web; import java.util.concurrent.atomic.atomiclong; import javax.persistence.entitymanager; import javax.persistence.persistencecontext; import net.sf.ehcache.cachemanager; import net.sf.ehcache.management.cache; import net.sf.ehcache.management.cachestatistics; import org.hibernate.stat.statistics; import org.springframework.beans.factory.annotation.autowired; import org.springframework.boot.autoconfigure.domain.entityscan; import org.springframework.cache.annotation.enablecaching; import org.springframework.transaction.annotation.enabletransactionmanagement; import org.springframework.transaction.annotation.propagation; import org.springframework.transaction.annotation.transactional; import org.springframework.web.bind.annotati
Read more

go - Golang: panic: runtime error: invalid memory address or nil pointer dereference using bufio.Scanner -

-
i implementing go program uses bufio.scanner , bufio.writer have packaged code follows package main import ( "fmt" "player/command" "strings" ) func main() { //enter code here. read input stdin. print output stdout commands.scanner.scan() { //scan new line , send comand variable check command exist or not input := strings.split(strings.trim(commands.scanner.text(), " "), " ") command := input[0] if command == "" { fmt.printf("$ %s:", commands.pwd) continue } if !commands.commands[command] { commands.throwerror("cannot recognize input.") } else { commands.execute(command, input[1:], nil) } fmt.printf("$ %s:", commands.pwd) } } i using init.go file in main package follows package main import ( "flag" "player/sourc
Read more

c - double free or corruption (fasttop) -

-
this code reads null terminated string file (file containing null terminated strings & should provide index wanted string). error: *** error in `./main': double free or corruption (fasttop): 0x090a4a80 *** aborted code: char *tmp_realloc=null; char *sstring=null; int i=1; /*set file pointer point string*/ fseek(fh,index,seek_set); sstring=(char*)malloc(sizeof(char)); if(sstring == null) return null; tmp_realloc=sstring; while( (*(sstring+i-1)=(char)fgetc((file*)fh)) != 0x0 ) { /*reallocate more memory*/ tmp_realloc=(char*)realloc((char*)sstring,++i); /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strcpy((char*)tmp_realloc,(char*)sstring); free(sstring); sstring=tmp_realloc; } } could tell me pointer causing issue (& reason) ? remove part /*if not same address copy old new & set old point new*/ if(tmp_realloc != sstring){ strc
Read more